Given that James stole Anne Hyde from Henry Jermyn, it would be a little amusing should James then end up marrying Judith Poley whom Jermyn ended up marrying IOTL.
IOTL, Jermyn and Judith failed to have any children so Jermyn failing to marry or dying earlier and a widowed Judith marrying James wouldn't interfere with the timeline significantly outside of the monarchy.
Plus, Jermyn was Catholic so it's possible that Judith was (I can't find a reference) so she's going to tick that box on James' checklist.
Oh that's quite interesting, though would Charles II allow another domestic marriage for his brother, when otl he didn't