It seems strange to compare them to the vast majority of colonial empires. Their geographic presence doesn't explain it because of Jamaica, Surinam etc, and French policy doesn't either because of the Ivory Coast, Senegal etc.
So what were the special factors involved here? Can we repeat those factors with other countries in alternate timelines to maintain colonialism elsewhere?