Hmm...take a look at this:
Charlemagne had twenty (20) children over the course of his life with eight of his ten known wives or concubines. Nonetheless, he only had four legitimate grandsons, the four sons of his third son, Louis. In addition, he had a grandson (Bernard of Italy, the only son of his third son, Pippin of Italy), who was born illegitimate but included in the line of inheritance. So, despite twenty children, the claimants to his inheritance were few.
If we handwave away all the sons, he doesn't have grandsons either unless he decides to include illegitimate children.