I believe the Soviets sent a destroyer and two frigates to pick up the sub and bring it home. The PM's orders were to hold the border and the coast defences and air force were ready.
Sweden was quite militarized at the time, and by then much of the armed forces were alerted and ready. Coast and air defences around Karlskrona, which was an important naval base, were very good. (Lots of missiles, coastal artillery, missile boats, rangers stationed on the small islands, and a heavily mined sea).
As mentioned above, would the Soviets had crossed the border they would had been warned, and ultimately fired upon and sunk.
This was exactly the limited skirmish the armed forces was good at, and I believe the Soviets would lose those ships if that happened.
While I weren't there at the time, I know of a (then) lieutenant who was. Special forces were ready to take control of the sub.
I believe preparations were made for an air campaign. A lot of Swedish roads are built to act as emergency airfields, and the JA 37 Viggens (like the Gripens today) are capable on landing on those short strips to be re-armed by a handful of conscripts. At the time that organization was apparently prepared to "go live".
(In the event of a war the airfields were, probably rightly, assumed to be the #1 Soviet target. The fighters were to lift of, fire their missiles against incoming bombers, land at roads, be rearmed and go back up again).
Both the planes and the pilots were of very high quality. (I believe a JA 37 was the first plane to intercept and lock on to a SR-71 for example).
At the time I believe Sweden was to well defended against air or naval attack for the Soviets to try with anything.
Of course the Soviets could strike back and defeat the Swedish armed forces
if they really wanted to, but I doubt they would want the effort, casualties and take the risk. After all, there were US subs in the archipelago, and as recent research has shown, Sweden was (and is) informally a NATO member.
More likely they would lose a ship or two, but they are handed their sub back through diplomacy and the entire thing is covered up.
Several air incidents were handled in a similar manner. Swedish jets often intercepted Soviet flying too close to the border, and often some kind of chasing / dogfighting happened (with no firing), yet sometimes with casualties. I know of at least one MiG that crashed into the sea (with pilot) after trying to follow a Viggen in a dive.
Let me see if I understand the situation here.
Some sort of naval skirmish occurs in the Baltic as a result of the submarine incident, with the Soviets getting the worst of it. At that point, there is a Soviet decision to make about how to follow up. Most here seem to be of the impression that escalation is the only option, but that seems likely to be a very expensive one from the Soviet perspective.
Let's define the freedom of action for the USSR here.
1. The USSR is not going to invade Sweden and occupy the country. NATO would have a cow over something like that, and it would be a costly undertaking at best. One wonders what the diplomatic repercussions would be as well.
2. The USSR cannot totally ignore the incident. It'd probably be too humiliating for domestic factions to stomach, and make the Soviets look weak.
The problem with a reaction closer to #1 than #2 is that the USSR will be unable to control Sweden's actions afterwards, and they are unlikely to be favorable. A harsh Soviet reaction that involves bombing Sweden leaves the door completely open to a tide of NATO support during the conflict, and would seem to guarantee Swedish membership in the alliance later. I doubt that Thatcher and Reagan are going to complain much about the cost of defending Sweden from Soviet aggression either, and Moscow wouldn't likely be winning any friends in Denmark or Norway.
There's no doubt that the USSR could eventually win a conflict with Sweden if it was willing to invest the effort and military resources. It's the overall costs to the long-term Soviet position that makes further escalation a very costly and unrewarding proposition for Moscow, even if it "wins" a limited war.
Very much doubt they would retaliate if the coast defence fired on their salvage operation. As others mentioned before me, the defences would make anything short of an all-out attack a quite costly thing for the Soviets, and they knew it. That was the plan all along, called "the Swedish hedgehog". Too sticky to be worth it.
Their best option at the time was to return eventual fire, while working through diplomacy and threats to get their sub back, cover it up, and forget it.
The Soviets at first probably wanted to sail up to the sub and tow it home. Would the ships they sent be fired upon they would return fire, but they probably knew that would be their end. So they tried to see if the Swedes would let them through peacefully, and they wouldn't. So they didn't try.
We eventually handed them their sub back anyway, with the nuclear torpedo and everything.