It wouldn't be the easiest time for the US to invade, but an annexation before the formation of Canada would be the easiest way to make Canadians Americans. Before then there wasn't really Canadian nationalism, just fear of American dominance. PEI and Newfoundland leaned towards joining the US at first but Canada panicked and sucked up to them to get them to join. I remember many in Nova Scotia and New Brunswick were reluctant to be what they thought as being annexed by the Province of Canada (I think Nova Scotia elected some 19 secessionist MPs). And the West was unsettled.
Anyways, since most in PEI and Nova Scotia etc. are proud Canadians today, it shows that an American annexation will probably not lead to lasting hostility as the last British subjects die out.
A pretty ASBish (especially the first part) example:
1837 - The Canadian rebellions are crushed and instead of a more democratic autonomous government, Britain keeps a firm hold on Canada, fearing that if they give it more democracy it will become pro-US and want to be annexed.
1862
The Trent Affair escalates into war between Britain and the US, with war declared on September 17. Russia joins the US and France joins Britain. Napoleon III also intervenes in Mexico, confident that the US will be shortly defeated.
On the same day that war was declared, US Gen. U.S. Grant decisively defeats Confederate Gen. Robert E. Lee at the Battle of Antietnam. Afterwards Lincoln issues the Emancipation Proclamation as well as the Antietnam Address, which cause a backlash in Britain and France of their support of the Confederacy. Grant pursues the Army of Northern Virginia and with it decimated, the crippled Confederacy only lasts a few more months before surrendering.
The Union invades Canada but the Canadians (especially Quebecois) are not enraged due to having democracy suppressed by Britain since 1937 and are horrified to see the mother country allied with a state associated with slavery. Furthermore, Britain declared war on the US, and is seen as the aggressor. The Union quickly steamrolls over Canada since the Confederate front is all but won, and Mexico occupies most of the French military's focus. British PM Palmerston is defeated by a motion of no confidence and Britain cedes what would later make up Canada to the US.
To decrease Canadian animosity towards their conquerors, the US encourages high immigration into Canada, especially the unpopulated West. Eastern Canada possibly put under a light version of Reconstruction. Many die-hard Loyalists go to Australia.