There is also the big fish/small pond attitude at play here as well
Unless you have a very early POD, all of the British settled colonies, as opposed to non white colonies, will have responsible government, which means that the government and associated voters/elites have to be on board with a wider federal project. Now while many people in these colonies will be pro Imperial federation, it will still take some time and effort to work out what this federation will be and do, sort of akin to OTL Canada or Australia. Although the difference being that London in this case will be far richer and powerful than any of the colonies.
Many of the people in the colonies will be quite happy with how things are without federation and may be quite happy with the current status quo in the local parliament, that being that they largely get to do what they want. With a federation this all goes out the window. Now unlike say the Australasia-NZ situation, where the latter would have been an important part of *Australia, roughly equal to NSW or Victoria, in a federation a NZ, Victoria or NSW will very much not be, at least in economic terms.
Those three examples will be like a Kent or Yorkshire, but without the advantage of being a train/horse ride from London. There will also be no established Lords in the former either, so without fundamental and perhaps ASB reform, their influence will be even less, as the House of Lords will continue to reflect the interest of the landed elites based largely in England.
Now most people will know this at the time of federation, because many of these people, especially in say NZ's case will know England, London or the UK well having either grown up there or visitied there later. Certainly at least one of our PM's was a Londoner, although he was pretty pro federation (although I forgot quite what flavour he liked). They will know that a close knit federation with an Imperial parliament will offer more risk than reward to them personally.
I can't imagine that a British settler voter in say Dunedin or Melbourne would long remain happy with such an arrangement if the London based federation became an effective centre of federal government, which invetiably would be a centralising force, just like most other capitals in the 20th century.
Now if it were some sort of much looser federation, more along the lines of an Imperial Council arrangement, which leaves local people to muddle about, that could be attractive.