It would certainly lead to class conflicts. Especially the businesses not depending on slavery, workers and immigrants. Also could lead to conflict while exploring the western frontiers. Slavers likely want to secure land for expanding slavery. Other people might stand up against slavers economy for economic reasons.
I actually recall from my research that the states' rights crowd were generally anti-standing army, at least initially, so this alliance probably isn't as synergistic as it looks.