He would have got into trouble with some other married woman. He had a penchant for other men's wives and, as many of them were top aristos, politicians and government officials they closed one eye in the hope of advancement. ( Like Brigadier-General Parker-Bowles in the 1980s, whose meteoric rise couldn't have much to do with his abilities).
As they say--no man has greater love than he who lays down his wife for King and Country.
That's the point, though. If he'd met and fallen in love with some other woman, even given that she was married to someone else, he could have lived with her as long as he'd first of all done his duty and produced an heir (and a spare). King Edward VII lived all but openly with a Mrs Keppel when he was King, despite being married to Alexandra.
The English are very relaxed about how their monarchs behave in private, as long as they keep their antics behind closed doors.