The economic impact may well be much less than we imagine, simply because the slave latifundia that the Roman writers depict as typical for agriculture in general were largely a phenomenon of Italy, and of the mid-second century BC to probably second century AD. It is likely the extreme model that Cato describes was already untenable by the early first century AD, though we do not know trhat with any certainty. A larger problem is that the lifestyle and income of the empire's traditional ruling class, the senatorial order, is designed around exactly this mode of agriculture, which may well be what sustained it way beyond its economic usefulness.
The big issue, though, is seeing why Rome might want to do that. It's not that the Romans were invested in slavery per se, or that they had any powerful ideology involved with it, quite the contrary. Very few Romans bought into the Aristotelian concept. Rather, they tended to view slavery as a personal fate, a case of being unlucky. By the second century AD, we see laws and precedent in favour of a more humane treatment of slaves, a movement that has longh roots in Roman Stoicism. But we never see the principle of the thing questioned. Even when Quintilian quotes the jurisconsult Caius in his Institutions sayiong that slavery is contrary to natural law, he merely says that every human being in the natural state is born free. There is no natural law is-ought fallacy in Roman legal philosophy. Almost no Roman ever argues that slavery is good (it is quite possible those in literature that do are satirical characters). But there are also practically none that say it is evil, in the sense the abilitionists do. There is no polarisation. A worlds on this trajectory could see slavery falling into disuse (like early modern northern Europe did), but it is unlikely it would ever muster the energy to change the law so fundamentally. Why? It's not like people would stop being unlucky.