I doubt that annexing part of Canada was really in the diplomatic cards. I can see it happening only if (1) Napoleon remains a major threat in Europe and (2) the US is able to take Upper Canada and Quebec. Extraordinarily good luck and good decisions would be necessary for both to occur.
Even if most of Canada were to fall to US forces, the Royal Navy can ravage the American coastline and shipping at will for so long and so savagely that it can force the Americans to give it up. The US government was nearly at fiscal collapse by the signing at Ghent. And although the British government settled for a draw in 1814, it wasn't going to accept a defeat with a loss of territory.
Militarily, the US can take Canadian territory--and vice versa--and both sides did occupy slices of foreign territory by Ghent. But neither was in a diplomatic position to hold them.