As I understand it the Soviets were aware the charges were "only" practice charges the issue was that due the way the American's were deploying them, (sprint in, drop, sprint out) and the way the destroyers were deployed, (sort of a three sided "box" with the Soviet sub somewhere in the middle, which is a very aggressive "hunter-killer" pattern) they were actually getting bracketed, (according to some accounts actually hit more than a couple times) fairly regularly which in general was assumed to be a pointed way of saying "surface now or be destroyed" as in the NEXT time would be the real thing. Hence the nerves on-board were pretty frayed. My understanding was the decision was very much not "shoot-or-don't" but one of "the American's are going to destroy us, so the question is do we take them with us or just let them kill us?" which puts a different context on things.
Launching even a normal torpedo under the circumstances would get them killed no matter what, launching a nuclear warhead they had a high chance of at least taking out the destroyers as revenge.
Randy