The thing is, so is a spatha or similar longsword. In a dense formation the infantryman won't have enough room to swing the sword very effectively so he'll be relegated primarily to stabbing motions. To be sure, the spatha is far more wieldy in close quarters than a spear, but a shortsword is wieldier than either. And the Roman's spears would have been most effective in dense formations of exactly the sort that limited the spatha.
A longsword, to avoid confusion, is a two-handed weapon much longer than a spatha and much later in timescale.
What you call a longsword is more properly an arming sword, or, simply, sword.
Really? My understanding was that the soldiers had a thrusting spear, the hasta, which was distinct from the throwing spear.
The hasta is a spear of any kind, just like a scutum is a shield of any kind. It is sometimes used to mean very specific things and sometimes, well, we just don't know. Plus the deliberate archaisms in style of late Classical and early medieval Romans introduce all sorts of confusion into weapon nomenclature. The Rhompaia of 1093 is definitely not the same weapon as the Rhompaia of 43 BCE, and that's just one example.
And now - speculation follows.
First, at what range do you think fighting happens when shieldwalls meet? Not nearly as close as you might think, even if simply to retain cohesion. The spatha is perfect for that range, when thrusting, and you can even fit some cuts into the mix if you measure the cut by your shield edge (similar to how a side-sword works with a buckler).
Secondly, the real point: the spatha was paired with a large round shield, the gladius was paired with a scutum. That's what I would imagine is the reason behind the change.
A spatha matches the round (or oval) shield in size and reach when the shield is held by the grip under the umbon and is so free to rotate around the wrist (so it's used edge-on rather than flat-on to attack the opponent's shield in the shieldwall).
You can put your spatha on either side of a shield that size and still be able to reach the opponent. You may have a real problem doing that with a gladius - your opponent would mechanically reach you first every time even if you had a long enough arm to be threatening.
A large, rectangular, curved shield was not used in the same way. It was there to physically block the enemy's advance and the legionnaires are described as cautiously poking and stabbing with the gladius above the rim or beside the edge.
So the bigger question is why did they change the shield type. Yes, using a round shield edge-on is fairly strong mechanically and maybe stronger than holding the scutum in front of you by one arm, but it also exposes you to other soldiers standing beside the one whose shield you are attacking. The Romans famously had success through teamwork. So why change shields?
I don't see how opposing infantry pinning the front of the Roman infantry is a case for the spatha. If anything it's a case against it, as the gladius would be more effective in that type of combat and the spear would be suited to fending off enemy cavalry.
The gladius isn't more effective
at the spatha's range. The spatha's range is guaranteed by the shield that goes with the spatha. The sword works around the shield.
The gladius is more effective
at the gladius' range, which is in turn provided by the scutum forming a strong enough barrier to prevent the fighter with the round shield/spatha combination from pushing the scutum out of the way and using the opening.
So the answer (and the question) is literally a matter of a few inches, really. Why did people stop fighting at a certain range and started fighting at a range that's a few inches longer instead? I don't know. There's interesting stuff being done on the biomechanics of shield fighting these days, and unfortunately the written sources have nothing new to tell us, so we'll probably need reconstructors to figure it out.