@David T It seems like the 1852 Convention was divided between those candidates who were supportive of the Compromise (chiefly Lewis Cass) and those like Buchanan who were courting southern opinion which was hostile to it; maybe this is how Houston gets the nomination, by rising as a "Pro 1850" alternative to Cass or Douglas? I might be able to see Douglas deciding that his bid is going nowhere earlier on, then deciding to make a deal with one of the only other pro-compromise dark horses for the VP or Sec of State position.
Actually, FWICT, Houston and Cass were the only other candidates explicitly supporting said policy, meaning that if the compromisers were to unite behind a dark horse, it may well be him. And "traitor" or not, Houston is still technically a southerner and a slave holder, so it's not crazy to think he could work as an olive branch to south by the 1850-ers, instead of the "compromise" of just nominating another doughface (Pierce, in the case of OTL). Thoughts?
CONSOLIDATE: Actually got around to looking at the 1852 balloting
in more detail -- seems Houston came in sixth on the first ballot with eight votes, but only managed to get himself to 12 later on, always behind Joseph Lane of Oregon who always got the same 13 votes on every ballot; considering that Stephen Douglas wasn't much higher initially, at 20 votes on the first ballot, I don't think it would be too difficult for the Texan to work himself up to fourth or even third place on momentum alone. The question is what happens when he finds himself in third place behind Cass and Buchanan...