The biggest problem is Swiss resistance to the idea. It would have added a lot of German speakers and Catholics, upsetting the balance that is part of the Swiss political structire. So while you write elements of the Swiss society, it's basically everyone needed to block it on a political level, even if there never was IIRC a official statement from the Swiss government on the topic.
Austria at the time isn't really able to do anything against Switzerland if it goes through aside from sending sternly worded letters.
And the Allies... The only one I can honestly see doing more than objecting is Italy, because strenthening a country they have irredentist claims on isn't in their interest.