At a basic level VSTOL fighter and attack jets simply don't work.
Not for any useful purpose in a well funded military. I know this seems like a pretty blanket statement but hear me out. Originally the harrier was a designed as a light attack jet with the VSTOL capability allowing it to to operate far closer to the front line than traditional jets allowing a higher sortie rate. That was the theory.
In practice it never worked out like that. The promised sortie rates never materalised, not least becasue rough field capability was on the pipe-dream end of the scale of practicality during sustained operations. This left the Harrier as an aircraft without much of a role, but a lot of money had been spent on it and so it couldn't be scrapped. It ended up being used in much the same way as the far cheaper and more efficient Jaguar.
Had that been the end of it, VSTOL would probably have died in the 80s. However the Harrier found a new role in the Royal Navy's desperation. Deprived of actual carriers, and with no organic air defence for their new anti-submarine warfare 'though deck cruisers'. Harrier was an aircraft that though mostly inferior to conventional types could at least achieve comparable performance, and at the same time operate off of the Invincible class's tiny decks. Which would be more than enough to fight slow moving Bears over the North Sea.
Had the RN possessed proper carriers this role would have been unesscerry.. Having proved the concept it was then adopted by a series of minor navies who desired carrier capability on a budget. Not becasue they needed it, but mostly as a prestige project. The Soviets independently developed a similar arrangement for their Kiev class, again mostly out of cost considerations, and unlike the Harrier they never got it to work quite right. The downside was of course that the aircraft these carrier carried were outclassed agsint pretty much everything, and of limited utility in many roles.
Again had this been the end of it VSTOL might have died at the end of the cold war when the need for budget carriers ended, and the prestige projects wore out.
Two things changed that, the Falklands War, and the United States Marine Corps.
During the Falklands war, harriers engaged conventional multirole aircraft in a set of almost unique conditions that conspired to give them even footing or even an advantage agsint their adversaries. Their performance in this conflict made people believe that VSTOL was a viable concept, ignoring the circumstances under which that performance occurred. (Sadly this is not a new phenomenon, see armoured cruisers at Tsushima).
The USMC are perhaps the greatest driving force behind the VSTOL movement. And ironically they pursued it for much the same reason as the Spanish, Italian and Thai navies. Prestige. The corps had long desired its, own carriers, so that it could be further untethered from requiring the support of the United States Navy. Knowing that realistically they would never be able to swing a full super carrier, or even a medium one operating conventional jets. The plucky little harrier operating from the tiny euro carriers seemed to the perfect solution. One they could easily operate from their massive new helicopter landing platforms.
Armoured in success, and powered by the funding that only a part of the United States military could muster, VSTOL was now here to stay. Despite having no real role that other aircraft and platforms could not fulfill better, and more effectively.
The reason Harrier was not further upgraded, and instead replaced with F-35, was because this was always the plan. As far back as the late 80s, in the era of big joint multinational defence procurement programs. the British and Americans decided to co-operate on a joint replacement for Harrier. At around the same time another much larger joint program was launched to produce the next generation NATO light fighter to supersede the F-16. This might have gone well if not for the USMC's greed.
They were not content with merely replacing the Harrier with an aircraft that was incrementally better. They wanted conventional performance on a VSTOL platform, so that the Gator navy could have true carriers. But that would have been tremendously expensive, and so they had to find someway to get the funds. They did it by fusing their VSTOL attack aircraft with the Light fighter replacement, with the a big dollop of the new hotness, stealth.
And thus the JSF was born.
Upon which a great majority of the Western world's future air power was invested. By the time people began to realize that this wasn't going to be cheap or effective it was far too late. Harrier production had long been abandoned, and there was no alternative.
One day someone will write a book on all this. And it will be titled "How the USMC destroyed Western Airpower".