I'm not sure you understood my point (or I may explaining it badly): while the raw material came mainly from colonial production, processing them (as well the capital investment origin) wasn't and where eventually two distinct activities with mainly distinct investors.
One of the principle drivers of innovation, particularly during industrialization, is to lower costs of production. The incentive to lower costs is pretty low when you have slaves. It's certainly a factor as to why industrialization progressed so much slower in antebellum South. As to the textile industry in Great Britain, had there been slave labor, or serfs for that matter, in the textile factories industrialization would have likely been delayed.
The mills of Great Britain and the slave plantations of the American South were all part of the same economy. The fact that they were separated by an ocean, had different backers, and operated under different governments doesn't change that. It was cotton that was produced by slaves and was then worked in factories that was the main engine of industrialization.
In fact, it was the cheapness of the raw material that was of such importance, and we see this in any example of mechanization or industrialization. Steam engines were developed in places where their raw material (coal) was cheap (coal mines). Paper mills were developed when there was an abundance of scrap cloth, and printing was developed when their was an abundance of paper.
This idea that slavery was antithetical to industrialization has some problems with the Roman experience. As their history went on, you saw a decrease in reliance on slavery by virtue of simple demographics. Further, you also saw just the sort of population declines that would make labor more valuable, in theory, due to the shortage of supply. And yet, there was not much in the way of mechanization or industrialization, aside from the apparently steady use of watermills.