1865, after the Confederacy's defeat. The Union had a million men under arms (at least on paper).
And around 40-50,000 of them were Canadians. They may have been British subjects, but the Canadians were undoubtedly on the US side of the Civil War.
Sure, that's not enough to stop them, but it's enough to make it hard.
Myself, I'd say 1914. Either way it's impossible. As soon as one problem begins to disappear, another one comes up. In 1862 it was the lack of transportation in the Canadian Plains. That was more-or-less solved by 1914 with the Trans-Canadian Railroad, but by then you had a wider population base.
Either way, it's not as simple as most people think. You can't just snap your fingers and have an American army group appear at Detroit to start terrorassing up the Niagara Peninsula. They'll be bogged down having to go through every city and town. Even if they don't want to it won't take a genius to funnel the large force needed to subdue a population into those areas.
You have to occupy Winnipeg and make sure that the railline is cut between east and west, but that's a long hard slog which could probably only be really accomplished with the kind of military that has trucks and can move faster than walking speed, since Winnipeg could be reinforced by trains.
You have to capture Halifax, one of the 5 or 6 largest ports in North America and the largest in the British Empire. Heck, War Plan Red actually was amended in the mid-30s to okay the use of poison gas on Halifax because the US Army decided there was simply no way to take it before the British arrived to reinforce the Canadians in the event of a conflict.