Extrapolating from Gibraltar, I'd think the British would give it back if the residents wanted. What would that take? Well, suppose the Chinese sent thousands of double agents in as "refugees" right before a referendum? Of course, how many thousands of those would truly defect?
Slightly related: What would it take for the British to give it to the Republic of China on Taiwan, rather than the People's Republic of China on the mainland?