May be urban legend, but one of the German generals reputedly spoke to his Swiss counterpart pre-war.
"You can muster a quarter-million men under arms. How would you fare if I arrive on your border with an army of half a million ?"
"Oh, we'd all shoot twice, then go home..."
Perhaps more relevant, the mountains surrounding the passes were tunnelled and mined to the point that any invading army must expect to have megatons of rock fall on their heads and multiply cut the roads. Unable to advance, unable to retreat, the survivors must surrender in place or provide target practice for Swiss snipers and mortars.
Air support ?? Forget it ! When Stukas must dive past tunneled machine gun nests, and changeable mountain weather may suddenly interpose cumulo-dolomite or cumulo-granite, the Luftwaffe's attrition would make their 'Battle of Britain' losses seem trivial...
There's another factor. Apparently AH remembered the grievous losses of WWI's Alpine campaign. There, the 'Mountain Jaegers' on both sides suffered horrendous casualties, proportionally much worse than the Trenches. When one (1) well-placed mortar or pack-howitzer shell could avalanche a snow cornice or scree across an enemy position, when re-supply was at the mercy of fox-holed snipers, when the geology was totally treacherous and hasty excavation nigh impossible, if not suicidal, infantry combat was the stuff of nightmare.
Add cold air, altitude, altitude differences and cross-winds which made precision artillery *difficult*. While Swiss defenders had the relevant local factors to correct their 'shoots', invaders must 'walk' their counter-fire onto targets. That's not going to end well...
An object lesson was the Italian efforts to force their border passes into France. The forts there took one hell of a hammering, but held the passes. The German mountain specialists who came to help fared no better. IIRC, despite the predicted attrition, those forts could have held out for many more months, but reluctantly surrendered after France fell...
The 'Unspeakable Sea Mammal' thing probably had a better chance than trying to storm the Swiss 'interior'...