IMHO, the greatest issue facing the Saturn V - and by extension, the Saturn IB - lay in that they were created chiefly to serve a Political Purpose, namely to one-up the USSR which until the mid-60’s, held the lead in the Space Race. And in order to claim it from them, the US committed to the Manned Lunar Mission for which the Saturn V was selected as the Launch Vehicle to make it happen.
In this regard, it succeeded. Wondrously. Achieving the self-imposed goal of both landing a Man on the Moon and returning him safely to the Earth before the decade was over.
But this was also its Achilles Heel.
Having achieved its Political Goal, its Purpose was served. The fact that they had a lot of surplus hardware left and sufficient funding - albeit greatly reduced and still falling - combined with Inertia kept it going for a while longer, even with Apollo 13 happening. But after Skylab, there was no real reason for Saturn V to be retained anymore - especially considering that (a) STS had been approved not that long before, and (b) the Soviet Counterpart, the N1 never managed to complete a Block A Burn before being destroyed.
The only way I can see for Saturn V to have any hope for continuation is a substantially more advanced Soviet Space Effort by about 1968. They don’t need to be ahead, but close enough to the US to make them want to keep pushing to keep the lead they worked that hard to take from the USSR.