Not nessesarily, just that the Federalist Party isn't destroyed and is able to make a comeback at some point.You mean Madison loses the 1812 election?
What if the period after the War of 1812 where all political divission disappeared never happened?