You'd need slaves coming from essentially one source, which would be hard to reach : between Africans, Germanic, Gaelic, Syrians etc. importations and the natural reproduction of the servile poll, slavery was racially as diverse than the overall population.
Not that Roman xenophobia couldn't turn into quasi-racist attitudes, giving whole population definite negative traits regardless of their cultural situation : Benjamin Isaac makes a good argument about how Roman prejudice, if not built on race as we understand it, was really rooted in geo-enironmental determinism from Aristotle, but heavily mixed with political and pragmatic concerns : some sort of social-lamarckism instead of social-darwinism if you will, mixed with an acceptance of slavery as necessary and fair because if they could do it, they did so.
It was as well considered as making these people a favor, bringing them into the service of civilization and its peoples : this is an important departure from Aristotelitian conception of slavery, because Romans did not see "natural slavery" as inherent to people or individuals, but relative to their relations with the Empire, both geographically and politically. As Romans weren't particularly pressed to find a rationalized justification to slavery beyond "we enslave our enemies, because that's cheap manpower, and "they're better off anyway evolving beyond a barbaric station"
So while you could get to some sort of social-lamarckist based-slavery, you'd need an important change in Roman history to go beyond an essentiallisation of Germans, Mauri, etc. as other than peoples without history, living in poor places, and in desperate need of guidance. Not unlike European paternalist justification for colonization of Africa, except Romans couldn't be bothered with conquering these shitty lands.
But race-based slavery? The idea would have been too foreign to Roman concept of the world, too "irredeemable".