They were about 10% of the workforce in Virginia factories before the war. The biggest hinderance eliminating slavery in 1831 and probably still in the northern South after would be this matter.
“But, as it is, we have the wolf by the ear, and we can neither hold him, nor safely let him go...