IOTL, French and British declaration of war over the invasion of Poland pretty much caught Hitler (and not just him) flat-footed. More so, the order of it, Britain declaring war first and France following later.
You can hardly compare Poland to the Soviet Union, Poland would likely collapse in weeks. The USSR on the other hand, even f the Red Army completely collapsed would take months to defeat and as Stalin is going to much more expecting of a German attack it's very likely the Red Army will actually do much better rather than worse.
In other words attacking the Soviet Union with France and Britain unharmed is even more suicidal than attacking the Soviet Union with Britain handicapped.