Yeah, I think it's difficult to overstate the impact that racism(or racial tensions, if you prefer) would have on provoking conflict between old-stock Canadians and the new BWI citizens.
Another thing, the two largest BWI nations, Jamaica and Trinidad and Tobago, together have a poplation about equal to that of Alberta, plus there are lots of other BWI nations besides those that could be annexed as well. And they're ALL English-speaking. One wonders what Quebec, or for that matter non-Quebec francophones, would think about admitting them all into Canada. Especially given the Empire Loyalist-style hoopla that likely would have surrounded annexation in at least the earlier years of Canadian history.
In 1911 the Bahamas only had 65,000 people. It has a low population and actually had a proposal for union. The others didn't really (though there was talk of Antigua as well). If the Bahamas happens in 13-14 then the war happens, Canada is going to be pretty tied up with post-war politics until the mid '20s and probably won't push for another island colony.
I think the others are possible given a 19th century POD, but 1900 is far too late.