You could still have that, just actually explain what it was and what it meant. I liked the implication, I just wish it had actually been explained.
But generally I agree, the last season was so rushed
Essentially saving The Man in the High Castle comes down to "Amazon Prime doesn't decide to fund Watchmen but worse in every conceivable way and Jim from the office being Jack Bauer for some reason and actually gives the show an ending."
Well, it's not only that. I know that Amazon wanted to create an analogue to the GoT (a very influential and popular show). I also think that after some time, they became afraid of being accused of "whitewashing" Nazis and whatnot. (Don't misinterpret this; it's almost impossible to do that and it shouldn't happen. But turning them into cliché cartoonish villains is also not a very good idea. People like the Nazis can have exemplary effectiveness and be very competent too (unfortunately for us)).
These however undermined a key aspect of the series : its alternate history setting. I mean, seasons 1 and 2 were very good (at least as far as I am concerned), exactly because they gave importance to this. They also presented an interesting side that we usually don't see (the Resistance as somewhat too bitter and isolated, people who became Nazis in order to protect themselves and their families and not so much out of fanaticism/ideology, normalisation after the war as stability and prosperity buy the regime the tolerance if not the approval of the population). The sci/fi element existed, but it was in the background, and subtle, which was ideal because it created a mystery covering that world and affecting it and was also an interesting idea itself.
In my very humble opinion, they should have kept all these elements and build on them. Keep the sci-fi element somewhat vague in the background. This would add some reality and free some screen time for the show to do things like more world-builng (Tokyo for example, or a scene from Europe), show more the internal political and social situation in the two empires and even, ultimately, build the show towards an ending where Germany marches towards the civil war, as its internal problems and divisions become to big and Japan is in deep trouble due to all the wars it has to wage and thus give the viewer the hope that these two empires will collapse "like sandcastles on the shore" in the end. These would also keep the show more balanced and original and not turn it into more of a sci-fi drama/action show.