Well if they could catch some of the battle-line in harbour, a small group of U-Boats could do some serious damage. The motivation would be different of course, it wouldn't be (primarily) to try to scare the British out of the wars as to knock down so many ships that they couldn't effectively patrol against future surface-ship breakouts.
Even assuming you are in position to exploit the same weakness as the attack on the Royal Oak, how many U boats could you see being able to navigate into Scapa Flow that way?
And since the RN never had the all the major hulls there it would just require juggling to balance out the fleets again. I don't see a German Admiralty betting everything on such a plan.