From Wikipedia:
Although it was clear by 1941 that the SS hierarchy led by Himmler and Heydrich was determined to embark on a policy of killing all the Jews under German control, there were important centers of opposition to this policy within the Nazi regime. The grounds for the opposition were mainly economic, not humanitarian. Hermann Goring, who had overall control of the German war industry, and the German army's Economics Department, representing the armaments industry, argued that the enormous Jewish labor force assembled in the General Government area (more than a million able-bodied workers) was an asset too valuable to waste while Germany was preparing to invade the Soviet Union.
During this period there were a few conflicts between the Army and the SS over policy in Poland. Ultimately, neither Göring nor the army leadership was willing or able to challenge Himmler's authority, particularly since Himmler made it clear he had Hitler's support.
Unless someone other than Hitler takes hold or we avoid Nazi Germany altogether, the Holocaust happens.