The only issue with that it isn’t as economically profitable on a mass scale especially in the long run and likely pisses off a lot of people. You can more cheaply hire poor whites or even indebted labor. Slave labor in more industrial jobs will be there but mostly for low risk labor(it can still be a hard job and suck but chances of them dying or maimed is extremely less in the jobs they get. I remember reading one British writing about being in south. He asked a man who was dealing with unloading ship cargo why the slaves were sitting around while whites unload cargo. He stated something about not wanting to risk breaking the backs of the slave he paid good money for. He then went on about how it was cheaper to have Irish immigrants do it because if they break their back he doesn’t lose much. Blacks might not be considered equal but they are considered more valuable money wise. Think of the coal mines in Appalachia and how they used animals there. They rather send humans to die and get maimed in mines because to owners they are worth less money wise then a animal(how they often saw blacks). If that donkey dies or gets maimed that costs him more money then having its cheap labor die or get maimed. He had to buy and take care of donkey not the people which is similar to how they saw slavery) If rented slaves die or get maimed they have to repay for damages(think how much that could cost them in comparison to cheap wage labor). If some lower class white or Irish immigrant die or gets maimed they don’t have to pay anyone back. That’s actually less people they have to pay now because they are either died or they fire them after getting hurt and can easily replace them with people desperate for work.
You have a half-remembered anecdote. I have cited multiple real sources that show slaveholders liked using slaves for industry, mining, etc and hat it was profitable in the long run.
"Elsewhere in the South bondsmen worked in sawmills, gristmills, quarries, and fisheries. They mined gold in North Carolina, coal and salt in Virginia, iron in Kentucky and Tennessee, and lead in Missouri." -
The Peculiar Institution: Slavery in the Ante-Bellum South by Kenneth M Stampp
"Some Southerners were enthusiastic crusaders for the development of factories which would employ slaves, They were convinced that bondsmen could be trained in all the necessary skills and would provide a cheaper and more manageable form of labor than free whites." -
The Peculiar Institution: Slavery in the Ante-Bellum South by Kenneth M Stampp
"From the earliest beginnings the southern iron industry depended upon skilled and unskilled slaves, Negro iron workers were employed in Bath County, Kentucky and along the Cumberland River in Tennessee. In the Cumberland country the majority of laborers at the iron furnaces were slaves. Montgomery Bell, owner of the Cumberland Iron Works, engaged his own three hundred slaves and many others in every task connected with the operation of forge and furnace. In the Great Valley of Virginia, where the southern industry was centered during the early nineteenth century, slaves constituted the chief labor supply." -
The Peculiar Institution: Slavery in the Ante-Bellum South by Kenneth M Stampp
Stampp also mentions how slaves were used to build roads, railroads, canals, and bridges. Slaves worked as lumberjacks, deckhands and firemen on river boats, dock laborers and stevedores, and mechanics.