It's the "going for" New England part that is ASB. The British do not have some sort of massive conscripted army, and weren't interested in the war enough to embark on a years-long campaign of taking cities to try and force the annexation of some significant territory anyway.
Umm, not even playing on the high unpopularity the War of 1812 had in New England, or even at least co-opting the Federalist Party?
Not to mention that if you're trying to win a war, invading the regions that are the least involved and which hold no special strategic significance doesn't make much sense, unless you want your army bogged down in fighting they could have otherwise avoided entirely.
Like I said - the War was very unpopular in New England, as it was basically leading the region towards economic suicide. Someone could have taken advantage of it - if not outright seceding on its own, then at least bedding with the former enemy, as demonstrated by Block Island, or at least breaking the economic blockade.