Let's assume a personal scandal regarding Buchanan's personal life: while serving as US Minister to Great Britain he became besotted with a handsome younger man (black-sheep cadet of a noble family, ex-Guards officer). The chap exploited Buchanan's obvious infatuation; a minor but very pungent scandal. Reports of this reach America just before the November election, with the electoral effect specified by the OP; the election goes to the House (and Senate). By the time Congress meets to carry out that process, the Buchanan scandal has become so notorious that Buchanan can't be elected.
The House elected in 1854-55 will choose. That House was elected during the collapse of the Whigs and the formations of the Republican and "Know-Nothing" parties. Many Representatives changed party designations twice. IMO it would be a huge effort to determine their actual allegiances, requiring a ton of research.
IMO, Fillmore would get the votes of every slave state (15/31), and at least one free state, returning to the Presidency. Breckenridge would be elected VP by the Senate (which must choose between the top two finishers; Fillmore's running mate would be ineligible).