The simplest way and one that's perfectly plausiable from 1500 is to have them fall under french rule.
Worth noting that the english crown had sold jersey to the french onlya few decades earlier in 1461. Albeit the next king didn't recognise the sale and reconquered it in 1468.
But them being given to france in some war or other during the 15-1600s is reasonable enough. And after 400 years of french rule...