I think the exact opposite is the case. The reason that the South seceded in 1860-1 was that a (virtually) all-northern and explicitly anti-slavery party (at least in the sense of opposition to any more slave states) had won the White House. It's really going to be hard to demonize Fillmore the way Lincoln was demonized. He is part of a bisectional party, the Whigs, for which millions of Southerners voted. He had said nothing about putting slavery in the course of "ultimate extinction." Moreover, he was known as a rival of Seward within the New York Whig party, and that by itself was enough to get him considerable southern support at the Whig national convnetion in 1848.
If despite having a reasonably friendly president in the White House, some Deep South hotheads decided to secede, they would be
more isolated from the rest of the South than in 1860-1. And if war developed, Fillmore would be even more emphatic than Lincoln that it was in defense of the Union, not to attack slavery.
(The idea of Delaware seceding in 1850 is particularly bizarre. As recently as 1847 a gradual emanicaption bill had passsed the Delaware state house of represntatives--and was only rejected by the state senate by one vote. John Clayton, a Delawarean, was a notable moderate on slavery; indeed, Michael Holt notes in *The Rise and Fall of the American Whig Party* that "Prior to his inclusion in Taylor's cabinet, Clayton had been the only southern Whig senator ever to cast a vote for the Wilmot Proviso." Calhoun stated in 1850 that it was doubrful that Delaware could be considered a southern state any longer.
https://books.google.com/books?id=8ikKAAAAIAAJ&pg=PA235)