Any idea where this story originates? From what I could find, his first wife (Elizabeth Rogers) was born in 1644, he married her in 1659, she died in 1661 (so 15yo at the marriage, 17/18 at her death). His second wife, Margaret Banaster was all of 10 years old when he married her in 1662, 15 when she died in 1667. Third wife, Frances Therese Stewart was born in 1648 and married him when she was 18yo. He died when she was 24yo. So, if he was derided as "impotent" before he wed Frances, I don't think that could be taken seriously.
Thanks for that snippet of info, I didn't know that the girls were that young. A fifteen year old should be able to fall pregnant, but it's not to say it's a given (I think a lot of guys I went to high school with would've been dads before they graduated had that been the case) - look at Charles II's own mom (married in 1625, but her first kid was only in the 1630s). The only wife he doesn't have an excuse for (where the story of impotence becomes credible) is wife no. 3 then. So he might've gone into the Restoration firing on all cylinders then.