Make the 1937-38 recession into a Depression.
There are varying explanations for its causes, but there are two ways to make it much worse.
One, have Roosevelt's tax increases and public works cuts in 1937 be more drastic and lead to a lower consumer confidence. At the same time, have Roosevelt's often demagogic anti-business rhetoric during this period be more strident and have the FBI and the Treasury Department tilt at more windmills of nonexistent conspiracies than they were doing already. This would lead to an even smaller amount of business confidence and investment during the period.
With these two factors strengthened, I think the Recession might end up tipping into a small Depression. The losses in 1938 would be worse, and Roosevelt would lose further popularity as he would have on top of the court packing fiasco, a Depression, and the isolationist vote all acting against him. That might be enough to lose him the Presidency.