I'm no expert but I
think that something like the
Crittenden Compromise being passed, and earlier than when it was proposed OTL (say 1859-60, or even much earlier still), could push it back for a while. (Beyond a few years quite literally anything can happen, so I'm not going to speculate beyond that). Its my understanding that abolitionists still weren't really a majority in the North at the start of the war and much of the Union was more for preserving the Union, at the start, at least. With slavery constitutionalized there isn't really an option for the abolitionists but, well, secession, which I don't think they would have had the support for for a while.
This is sort of the basis for a TL I've been tinkering with, with a later ACW coinciding with an earlier Gilded Age, and the results of that.