Taylor might do something, his being apolitical still caused him to take office, and increased sectional issues I would say would prompt him to either go into politics or stay in the military.
Also, Ganesha, remember that what got Lincoln so outraged was Fillmore signing the Compromise of 1850, Polk may well have taken the Taylor position (threatening to veto it). Fillmore was well-intentioned and was certainly better than near-traitors like Buchanan but he just failed to understand the moral implications of slavery and the fact that the Compromise would put a lid on tensions in the short-run but make them even worse in the long-run.
No Fillmore probably means no Compromise, but Lincoln was just as annoyed by Polk's pro-slavery expansionism as well. An Ostend Manifesto sort of thing that lead to an actual annexation could provoke Lincoln into re-entering politics as well, just remember he had the "spot resolutions" with Polk as well.
Lincoln and the Republicans would still materialize as the fundamental reasons for their founding still exist.
No Compromise could mean same time Civil War or even an earlier one, events go as OTL, still a Fort Sumter stand off, if Polk is at the helm than he sends provisions like Lincoln did, if Abe runs the show than same asOTL as well.
Also, quick note, you are looking for uninterested, not disinterested, the latter means unbiased, as in if you were in court you would want a judge to be disinterested in the case.