When Finland and Sweden OTL discussed a state alliance Summer 1940, Sweden demanded that Finland acknowledge the losses of the winter war and drop any claims to regaining Karelia. Finland did not want to do that, and the talks collapsed on this and German and Soviet unwillingness to guarantee that they had no hostile designs on the state alliance. Sweden was willing to assume responsibility for defending Finland, if Finland dropped the claims to Karelia. Finland chose to invite German troops and get involved in the Continuation War instead. At the time, it might have been seen as the more sensible choice, but the fact remains, it cost Finland even more, because of nationalist tendencies. There's your example from OTL.
I was expecting you mention that. It is debatable, though, how much the collapse of the deal was due to Finnish nationalism as such. For one thing, the loss of Karelia was an ongoing national trauma in 1940. It had been grabbed by the USSR in an attack that was universally condemned and led to the USSR's expulsion from the LoN. One must underestimate the deep-seated injustice felt by the Finns, who were at the time trying to resettle over ten percent of their population in very tense international and domestic conditions to chalk Finnish unwillingness to abandon their claims to a territory historically and righfully theirs in what amounts to a heartbeat. What would have Sweden done in a similar situation after it had just lost most of Götaland south of Jönköping to the Germans, say? You just don't shrug that off.
For another thing, the Swedish Foreign Minister Günther made both relinquishing claims and German and Soviet acceptance preconditions to the defensive union. It was the latter point that finally brought the deal down, as Finland in its desperation would have accepted the first one. The demand for the support of the major powers, one of which had just waged a war of aggression against Finland, might in some ways be called a get-away-from-jail card on the Swedish side. Sweden would have wanted a defensive union, sure. One in which it was certain you don't need to actually defend yourself or your ally against hotile powers.
Finland was driven into Germany's waiting arms by nationalism and irredentist dreams, but only for a part. It was much more important that Finland in 1940 was in existential fear over its very survival. That Stalin wuld come and finish what he had started, the annexation of Finland, was accepted as a commonplace, especially as Finland was being kept in a diplomatic vise by the Soviets through 1940. As help from Sweden was not forthcoming, Finland had to turn to the only option left for support and that was Hitler's Germany. Mannerheim and Ryti certainly had no love for the Nazis, but after all the other options had been exhausted, that was it.
I'll return to the other points later.