You (and Hitler) said it yourself, 'England is not our enemy' and apparently he didn't want to destroy the Empire (probably bullshit said to try and make folks believe and trust him, which is an impossibility at this point).
Also how are they going to know about the T-34 and KV-1? And know how effective it was (with a good crew anyhow..) So you're assigning them foresight, anything else you want to do in this sloppy titwank? And please don't say 'Ahh but the Abwehr...' it was a useless organisation mostly staffed at its higher levels by people who didn't like the Nazi's and one who had a pretty darn useless time of getting people into other countries to learn things.
And Hitler put it off because he needed to destroy the threat to Germany and the Rhur in the West first. The Anglo/French armies are much closer to Germany than the Russians (further still now thanks to Poland's 'donation' of land) and they needed to be dealt with. And as was pointed out earlier, the Germans could barely supply Rommel's forces, how are they going to supply and ship the forces earmarked OTL for Barbarossa if you move them into Africa? You're still limited by shipping at the end of the day and Italy would probably be sweating cold buckets as well even if there was nothing they could do if the Germans decided to also go "You know what Benny...what's yours is mine right?"
And Hitler also put off anything because the Soviets were helping keep the German econimy going with grain, oil and other materials that they needed, albeit at a significant cost which would start running out once the Germans start running out of places to conquer, loot and pillage. Egypt and the like probably didn't have much in the way of gold reserves the Nazi's could yoink after all, nor is it exactly grain breadbasket territory either. The oil's good yes, but most of that infrastructure was very very poorly developed at the time and there's the issue of building it up, shipping it out etc as you'd have to go all the way into Iran and Iraq to do that. And this all takes time, years to get going. Giving the Soviets, your declared enemy for 10 odd years in news, writing, school and propaganda more and more time to build up, to modernize and fortify. And you're at the short end of the manufacturing and numbers stick.