Enoch Powell would be confined to bashing Americans?
Seriously, though, it probably would have a pretty significant impact on British culture, butterflying away everything from the island influences in punk rock to My Beautiful Laundrette. George Harrison's sitar stylings would be about as "global" as it got for British music. (Though admittedly, foreign influences could come in via recordings as well).
Politically, "post-colonial" communities are obviously less of an electoral force, which probably impacts the Labour Party a bit more than the Tories. Also, you wouldn't have the anti-colonial expat crowd gathered in London and elsewhere, which may or may not effect how things play out in Africa and the Caribbean.