In OTL the big slave owners had managed to make it impossible to argue publicly against the principal of slavery by the 1850s, perhaps the 1830s
As I understand it many white people in less prosperous areas did not think slavery helped them, they faced unfair competition.
Could there have been a continued, if likely a small minorirty group opposing slavery have existed in most of the Southern states?