He actually only said slaves.
Not Jewish nor Hebrew slaves.
In addition, the pyramids were not built by slaves, but by native religious believers.
And Josephus is supposed to be writing a history of the Hebrew people.
Adding in myth at all does not lend itself to the validity of the history.