once they were gone Canada didn't stand a chance
...was the common wisdom at the time, of course.
Either way, without the annexation you wouldn't have had the disastrous Second American-Canadian War. Everything seemed to be going so well after the big victory at the Battle of Thorold and the Amherstburg Crossing. The US even announced the annexation. But the Canadian Resistance Army turned out to be pretty potent for a small group, especially with William Otter whipping them into shape, saying nothing of finding common cause with First Nations and Francophones who wanted no part of being American. A lot of people still remembered 1812 and the damage done by the Americans back in those days.
As much as US history textbooks tend to downplay it, the obliteration of the US Army of the North by Otter's much smaller army of Canadian militiamen, ex-Redcoats and First Nations at the Battle of Brant's Ford - not to mention the general strikes against US rule - not only ended the annexation, but forced the US - busy as it was with the rebellion in the Deep South - to sign the Windsor Accord, guaranteeing the independence of Canada. The entire annexation was an immense national humiliation for the US and the national psyche has never truly recovered.
If nothing else, without the annexation, Canada would likely remain a British dominion. English would probably be the first language in Canada, as opposed to French as it is today, and the capital might be somewhere other than Montreal. The country might even still recognize the British monarch as opposed to being a republic as it is today.