I find that hard to believe given the size of the debt they had run up and all the Mefo bills circulating that they were using ever more desperate tactics to avoid repaying. Add in a total lack of foreign exchange reserves, the utter dependence on state work of much of the economy ( stop making tanks for instance and what do the factories do ) and a collapse ( or at least a major recession ) as early as 1940 is on the cards.
Evans quotes Hitler in 1936 as stating the Germany economy "must be fit for war within four years" and Hitler again stated in 1937 that Germany needs small wars of plunder to support the struggling German Economy or it would collapse ( as recorded in the Hossbach Memorandum )
What all of this means, functionally, is that rearmament could not continue indefinitely or escalate beyond the very real limitations of the German economy. It also means that living standards were set to deteriorate, as Schacht warned in his rather apocalyptic memorandum after the annexation of the Sudetenland. You also should not judge Hitler's quote uncritically, as his view of economics is inevitably determined by his worldview. To him, an inability to rearm to the point needed for a successful military conquest of the east means the death of Germany, because surely "world Jewry" will then have the time and breathing room to destroy Germany at its leisure. The only option in an inevitable war was to choose the time and place of the battle, so that you could strike first and gain the initiative. But that's how things look like in his embattled mental universe. We on the other hand look at the economic reality, which indeed does tell us that German rearmament is not a long-term sustainable proposition on the scale wanted by Hitler, and that German living standards would suffer as a result of its wind-down. But this does not represent a collapse, and the regime had been through economic hardships before, and survived - the winter of '34/35 was bad, and that was on far shakier political ground too. Moreover, even as Hitler was removing Schacht's last vestiges of influence to place the Reichsbank under "firm National Socialist leadership" rearmament had to compromise with the very factor you cite, the lack of foreign exchange reserves. Precisely at the time when theoretical planning for a gigantic rearmament programme meant to put Germany in a position to fight both Britain and France was reaching a feverish pace, the army got its steel allocation cut from 530,000 tons to 300,000 tons. Why? To bolster exports...
In short, the picture is a lot more complex than it's usually made out to be. That would be even truer in ITTL 1939, since British preoccupation with the Far East likely reduces German frenzy for the last gasp of rearmament expansion required to fight a two-front general European war.
As for OTL - really I should quote the whole chapter "Nothing To Gain By Waiting" but in the interest of brevity, here are two crucial quotes by Tooze in "Wages of Destruction" (bold mine, not the author's):
The German armaments economy had once more reached the impasse that, since 1934, had repeatedly interrupted its expansion. Of course, by extreme measures it would have been possible to raise the share of national income going to the military to above the 20 per cent level already reached in 1938, but only at the expense of abandoning any pretence to a normal peacetime economic policy. Alternatively, Germany could have done what it had done in 1936-8. It could have held off from accelerating rearmament for a period of twelve to eighteen months, accumulating enough foreign exchange for one last burst of military expenditure. But this was hardly an attractive outlook when one considers Thomas's statistics on the global arms race. With close to 20 per cent of its national economy already dedicated to military spending and the Wehrmacht's share of critical raw materials hovering between 20 and 30 per cent, Germany's 'wartime economy at peace' had reached a critical threshold.
As of 28 August, Hitler was driving towards war, fully aware of the likely involvement of the British. Both at the time and after the event there were those in and around the leadership of the Third Reich who refused to believe that Hitler could be deliberately courting such an enormous risk.129 We, however, should not flinch from this enormity. To talk of 'miscalculations' and 'mistakes' in relation to the outbreak of World War II is to underestimate the deliberateness of Hitler's intent.130 As we have argued in this chapter, Hitler was encouraged to pursue this course of quick-fire aggression by interlocking economic and strategic pressures. We have deliberately avoided here any talk of 'crisis'. In 1939 there was no crisis in the Third Reich, either political or economic.131 The means of coercion and control developed since the near-crisis of 1934 were too effective for that. But what could not be obscured by May 1939 was the complete frustration of the medium-term strategic vision that had taken shape in the aftermath of Munich.132