So, if Speer is in charge of most if not all construction in Germany, he would come to control the economy to a large extent, he would get a chance to show his skills as a manager and in an area where he would have greater experience, he would get the chance to become connected with various prominent figures and, really, build his popularity (construction is an easy way to do that, people usually like it);
Accurate. It could allow him to build a more productive arrangement with Gauleiters than in OTL. Contrary to his post-war myth-making, Speer was the exact opposite of a rank outsider to the Nazi movement. He was an incredibly power-hungry, ambitious man with a web of connections, but his relationship with the Party's regional strongmen was poor during the war, with the exception of Karl Kaufmann and Karl Hanke.
They resented his intrusion in the civilian economy in their local bailiwicks. For one, they were concernd about it upsetting public morale. Moreover, Speer was haughty and heavy-handed. Like, the speech he gave at the Posen conference was interpreted as him threatening to sic the SS on them if they did not comply with his order to shut down enterprises not deemed essential to the war economy. But in this scenario he's the guy who's bringing infrastructure, 'beautifying' their cities and building big buildings. A good way to build popularity, bring in jobs and raise their status - as well as graft, of course. Bormann has his proteges in the Party apparatus, but he doesn't have an absolute lock on it. Never mind the fact that the Old Fighters look down on him.
Moreover, Speer can also still forge a relationship with the SS. It started in the late 30s when Speer needed stones and forced labourers for his building projects, and Himmler needed capital and a way to expand his concetration camp empire at a time when it seemed to have lost many of its functions since there was no organised political opposition. Speer will no doubt be given major construction projects in the east, after all, and that's where the SS wants to build its based and agrarian settlements.
Nazi Germany was not a static regime. It was a dynamic one that was always in motion. No major bigwig could rest on their laurels and trust that a Führer order appointing them to such and such office would always be relevant because there was always someone else trying to win favour and push their agenda by 'working towards the Führer', which inevitably meant elbowing another bigwig simply because it was a mess of overlapping jurisdictions and competences. That was by design because it kept them dependent on Führer and conformed to his Social Darwinism. It's a mess of competing bureaucratic fiefdoms (after the Reich cabinet had its last meeting 1937, meetings between the state secretaries of various ministries effectively became a substitute for cabinet government, until Hitler prohibited that, too, leading to an incredibly cumbersome system where proposals for laws drafted by one ministry had to be sent from one affected agency to the next, leading to constant redrafting until it could eventually be submitted to Hitler - unless it was blocked by Lammers or Bormann - and signed by him, assuming he bothered).
This is what fuelled what Mommsen calls the 'cumulative radicalisation' of the regime. Hitler did show a degree of commitment to Old Fighters, hanging on to many even when they'd proved useless (and the Nazis weren't really in the habit of murdering their own, they were more commonly just forced into retirement or given crap jobs). Bormann still got him to drop some Old Fighters in the '40s. Moreover, if a bigwig was in a bind and his colleagues were intriguing against him, he couldn't count on Hitler bailing him out on account of years of loyalty.
The careers of men like Himmler, Heydrich, Speer, Bormann, Kaltenbrunner and many people on the lower levels such as Stuckart and various RSHA functionaries show that it was the kind of system where even comparatively young people who hadn't played a major role during the 'Kampfzeit' could rise fairly quickly if they had the right combination of ruthlessness, cunning, connections and circumstance (Kaltenbrunner's career is quite interesting since until his appointment as RSHA chief he was a provincial nobody with no connections other than Himmler. But after his promotion he showed he was a player in his own right, even managing to attain direct access to Hitler and bypass his boss).
In the short term, Göring benefits because he's the heir apparent, an Old Fighter with lots of assets and levers and has avoided the humiliations of OTL. He's also actually popular, affable, and can excite a crowd as a public performer, which is an advantage over many competitors - Heydrich hates giving speeches because of his high-pitched voice, Bormann has zero charm and no Volksgenosse even knows about him, Himmler is like a pedantic schoolmaster when he regurgitates the SS word salad. Speer definitely beats the aforementioned on the charm front, he more often than not gave off the image of being cold and haughty.
However, in the long term things look different. Göring was very smart and could be energetic in the short run, but got complacent and fixated on enjoying the trappings of power than exercising it. He had an unhealthy life style that can drag him down and turn the traits that made him appealing into a liability. He accumulated more offices and titles than he could manage, but could not let someone deputise for him since that would create a rival. If Hitler dies early in say the early '50s or so, then I'd say Göring has a good shot and the succession is his to lose. Most bigwigs could live with him for a time because he won't upset the applecart, which does not preclude his authority being challenged further down the line. But if Hitler holds on for longer, things could look very different.
As for the age question, Hitler's top minions were all younger than him. Göring was the oldest of the gang, having been born in 1893. Heydrich was born in 1904, Speer in 1905, Himmler and Bormann in 1900. While a successful outcome of the war will probably leave Hitler in better health, I don't think it will butterfly all his health problems. Especially not with Morell as his doctor.