Beyond just how the Japanese get to Hainan, there's the matters of being strong enough to actually fend off any Chinese dynasties that want the island and retaining enough trade and contacts with the home islands so as to not become Chinese via assimilation. Hainan's geography makes trade with China too lucrative to pass up, which will mean lots of communication with Chinese merchants and thus fluency in Chinese ends up being extremely important and will see plenty of Chinese merchants visiting or setting up shop. Add to that the previously existing Chinese population of the island and any additional migrations (if any northern nomads invade China, refugees from South China would end up fleeing to Hainan and rebels would organize there, much like what happened on Taiwan/Formosa during the Qing invasion of Ming China) and the prestige of Chinese culture and it'd be hard to actually keep Hainan Japanese. Not to mention the distance makes communication difficult, so the local dialect would end up diverging from the home islands and the local culture will evolve differently from the home islands. At that point, can it really be called ethnically, linguistically, or culturally Japanese?
The best comparison would be the English and German identities, I think. Anglo-Saxons were descended from Germanic peoples who arrived on an island not too close from their homeland and ended up dominating the island, but saw their language and cultures evolve differently (High German consonant shift and the English Great Vowel Shift, for example) because those linguistic and cultural changes didn't travel across the seas and due to foreign military and cultural influence (French for English, due to the Norman invasion and the increased economic and political ties to France it brought). But in this case, the distance is much further and the distance and power discrepancy between China and Hainan Island is skewed much more than the distance and power discrepancy between France and England (closer distance, more power discrepancy). And even before French involvement, within a few centuries of arriving in England, the Anglo-Saxons were distinct from their continental cousins to the point they had a distinct different language that only became more distinct as time went on.
To be specific with an example of cultural divergences, let's take a Warring States Period in Japan. Hainan Island is ill-suited to get involved. Simply communicating with allies will be inconvenient due to the distance, let alone shipping and provisioning troops. But if they don't get involved, that's a huge part of Japanese home isle culture Hainan misses out on. Linguistic changes are inevitable, just based on dialects that form within states in countries that aren't separated by thousands of km of sea. Enough divergences and it'll be hard to call Hainanese culture Japanese (Japonic at best, but that depends on regular contact, which was hard to maintain with pre-1500s naval technology and techniques).
It's not appropriate to compare to colonial nations and their identities/languages to this situation, I think, since European naval technology after the 1500s could facilitate constant communication and population transfers between the colonies and metropoles, which could prevent extreme divergences (and even then, the identities inevitably diverged and the languages are still notably distinct). 283 years (1260 PoD vs 1543, when Portugal initially reached Japan) is a long time for identities and languages to diverge irreversibly. Hell, despite Jeju Island being 82 km from mainland Korea and having been under mainland Korean rule since 1404 (it was officially a possession of the Goryeo dynasty since 1105, though with high autonomy for the next 3 centuries, and a subject nation to the Baekje, Silla, and Goryeo for much longer) and its language is still incomprehensible to a mainland Korean.
