No that is simply not true. While rice was extremely labor intensive, 2-3 harvests could be completed in a given year. This allowed for enormous agricultural surpluses which contributed greatly to the traditional wealth and large populations of the South East Asian states.
Notice how I said in terms of labor needed. With millet and sorghum, all you need to do is to clear and plow the field, scatter the seed, and wait for rain. In paddy style rice farming (the most productive), the fields need to be cleared, plowed and flooded, rice seeds sprouted in a separate germination bed, and then each seedling individually planted by hand. In the time it takes to construct a paddy field and plant seedlings, several fields can be cleared and planted with millet or sorghum. Also, plowing a dry field can be done with any draft animal. Plowing a paddy requires a water buffalo.
Foxtail millet matures in 90 days, which, discounting day length needs, can potentially give you 4 crops a year with a fraction of the labor needed for rice. Millet is also far more drought resistant. It only needs water for the first month or so of growth.
Southeast Asia could afford to grow rice because rainfall is largely consistent in the growing season. Places like northern China needed crops like millet to sustain the population. Even places with more rainfall like Japan took a long time for rice to fully supplant millet in the diet.