Or, the settlers that might have gone to Canada settle instead in the other white dominions.
South Africa wasn't looked on fondly(once reports filtered back), Australia took a while to get of the ground and wasn't exactly a paradise (moving the Canadians to Australia would result in a near 7 fold increase in 1850 or so) and New Zealand would take until the middle of the 19th century to get going as well.
We also need to consider the fat that a large number of those in Canada early on moved from the US or were already there as French people, those people and their descendants are a dead loss straight away.
My guess is that Britain and the US see most of the increase(although it's possible that the US might not be that affected if they no longer have Canada as a way station between Britain and their shores and it might even be a loss).
It's all moot anyway since Britain losing Canada isn't all that likely.