There comes a point of diminishing returns on the increasing need to totally redesign the internal spaces for the avionics and engines that became available.
At some point it's simpler to just start from a clean sheet of paper.
Thats right, even something as simple as the HUD had a huge impact on fighter design and required advances in engines to really get the most out of such a design philosophy a HUD allows.
The Phantom was a 'heads down' aircraft, where the crew were to place heavy reliance on their weapons systems to fight. This meant that there wasn't much need for a huge canopy for crew vision, so it was low slung to reduce drag. Unfortunately Vietnam was a 'heads up' A2A environment making the Phantom design a liability.
This problem was solved with the HUD, allowing the crew to do what was formerly heads down fighting heads up. To make the most of this the teens had big, bubble canopys providing fantastic views, however these generate a lot of drag. Turbofan engines dealt with this problem by providing fuel efficiency in dry cruise flight and massive afterburner power to overcome drag in highly supersonic flight. You just can't rig this up in a phantom, lightning or mirage iii.