I never said "and other Europeans", only that the powers that kick out the Portuguese cannot themselves be European.
If other European powers feel strong enough later to take a shot, that's within the OP -- but it will be later, since the Portuguese were the only ones European players until Spain sailed west from the Americas, and then the 17th Century, when the Dutch, England, etc got involved.
The problem is that it was too much at stake for us. Portugal spent the 16th century wrestling for the control of the Indian Ocean against anyone that dared to dispute it's control of the area.
So to kick us out of the Indian Ocean, you need to wank Egypt, hard! We had the tech, the numbers and the ships to control it at the time, and we only accept the best in our ships; The gunners and gunmakers were contracted from Bohemia, Germany and Italy because they had the best, all cannons were melted after 300 shots to be recasted to make sure that the cannons wouldn't explode because of problems in the barrels, the arsenal we built in Goa had a room known as the Room of 10000 Muskets, this are just some example of the power we could project in India. The control of the Indian Ocean was at the time considered a "Holy Crusade" and we were fanatics about it.
Basically either you wank Egypt to giant levels, make the Ottomans ignore the Med to focus only on the Indian Ocean, or have the full Portuguese navy go down during a storm, or we would never stop coming back.
Gama proved that the trade was worth it, and even in the 17th century, a single ship could had a profit of 120% by doing a trip to India, in the 16th it was 350%. With this profits Lisbon would never stop shoving resources into India to ensure the control of the Ocean.