Which divisions get wiped out is going to matter. Any battle where the specific units or their officers played an important, or pivotal, role could get butterflied in the other direction due to their absence. I don't think a failed Sea Lion would break the Wehrmacht's back; they're still riding high on the Fall of France in 1940-1941 and Sea Lion would probably have been seen as a risky gamble, not a serious setback.
As one somewhat chilling possibility losing at Sea Lion, especially if Hitler puts it in motion over the objections of his generals, could humble him somewhat and convince him to give the professionals on the OKW more leeway in running the war. In the USSR one of the key turning points in the early days on the Eastern Front was the utter failure of the operations Stalin ordered in the winter of '41-'42 which helped convince him to listen to the generals and stop micromanaging the war. An early, and disastrous, Sea Lion that could be laid at Hitler's feet might have the same effect in Berlin.