When I checked the annual population growth rates within the United States from 1900 I noticed the United States experienced population growth rates of less than 1% from 1930 to 1941, no doubt due to the Great Depression. The birth rate went up after 1941 but it did not spike during the baby boom period except in the years 1947, 1950 and 1957. Population growth plummeted to under 1.5% for most of the years since 1962 and below 1% in 1969, between 1973 and 1976, between 1980 and 1989, between 1994 and 1999 and for most of the 21st Century thus far.
The baby boom wasn't a baby boom at all. It was the population growth rate returning to pre-Great Depression levels. If the Second World War didn't happen the United States population growth would've remained low until the United States had recovered fully from the Great Depression which would've lasted a lot longer than in our time line. It's quite possible that by the time the effects of the Depression had worn off that the contraceptive pill would've entrenched the low population growth rate.
So what would the population be in this scenario? About 250 million.