How about the much more likely scenario, actually considered at the time by President Jefferson? The US forms an alliance with Great Britain, enters the Napoleonic Wars and seizes the Louisiana Territory without paying a cent. American privateers have some fun and the US winds up with a larger standing army and navy, as we either send a detachment to Europe or allow volunteers to be recruited by the Brits. No disastrous war of 1812 and the relations between Washington and London are much better, while the US relations with France are much worse.
Alternately, the US is galvanized by the British seizure of New Orleans and the area of modern-day Louisiana. When 1812 begins the US Navy has added several frigates and smaller craft, possibly the first line of battle ships also. More importantly is the size of the US army(20,000?). Given the continued needs in Europe AND the new garrison in New Orleans, the early British victories in Canada vs the US invaders never happen. The US either annexes Canada or keeps substantial border areas while getting New Orleans in return for partial withdrawing from Canada.
No reason immigration to the US would even notice a change, after all, settlement west of Ol' Man River didn't get far until after the Civil War. Any effort by the British to hold the entire purchase would guarantee a confrontation and the worst of all worlds. The US would seize practically everything north of Louisiana(the state), possibly losing bits of Alabama and Mississippi while the British took Florida. Now the US is angry, obliged to field a proper military, while this becomes an issue in every situation with the British. By 1900 the United States in OTL outweighed Great Britain itself in every category, and outweighed Canada by more than 10 to 1. How long before the British wind up making concessions on the grand scale to avoid a permanent foe at their back?
1845: Compromise not possible on Oregon, British join Mexico in war. Mexico is easily beaten as it was in OTL. US navy and coastal fortifications hamper RN actions while privateers ravage Brit merchant marine and English starve without US or Canadian grain.
War ends with border similar to 1848 in south(no Gadsden Purchase?) US obtains British Columbia, most of British territories in southeast.
NOTE: This may seem unlikely but compare the size of the US forces in this war with what appeared in 1861-1865. No reason the US doesn't send Taylor down with 50,000 instead of what he had. And given the alternative of a long and bloody and EXPENSIVE war, the British might find concessions out of Mexico's hide an attractive alternative.
1854: US enters Crimean War, British situation is soon rendered hopeless. US annexes most/all of Canada as well as Louisiana. Any changes in Mexican border are changed in US favor. Permanent peace-time US army of 50,000 destroys native resistance by 1860.
Indian Uprising of 1856? Franco-Prussian War? Sooner or later the British find themselves losing the two-front war.
Civil War? In the US? With much of the south held by London and a constant foreign threat? Don't be silly, never was such a thing.
The fact is that ANY long-term irritation to US/Brit relations would be disastrous for the English. The US as early as 1812 had Canada outclassed more than 10 to 1, guaranteeing a burden on London for any conflict. Given MORE territory to hold in the south and a PERMANENT source of trouble and the Brits would have been VERY unhappy. Transporting and maintaining any kind of military force to North America was extremly costly, and the British were extremly happy to spend the period 1815-1914 on the cheap as military spending went, not that this avoided many complaints from the public. Towards the end of the 19th century the entire British army was less than 140,000 on paper. Imagine requiring a PERMANENT military increase of 20-30% in time of peace, and much more EVERY TIME London has another conflict somewhere.
30,000 BRITS IN NEW ORLEANS!!! AIGH!!!

Are you guys TRYING to hand Napoleon his victory at Waterloo?!?