I think the problem is that both Portugal and Brazil are too far from Portugal to have massive immigration to the U.S. of the scale of the neighbouring Spanish speaking countries. The U.S. has Mexico, Cuba, Puerto Rico and the Dominican Republic all within a short distance. Colombia, Ecuador and Central America are all closer as well.
There are Portuguese in the U.S. but as mentioned earlier the majority are from the Azores. The Azoreans began arriving in New England on the whaling ships that would leave from Massachusetts to the Azores to go whale hunting. Until 1950 the Azoreans accounted for 90% of the Portuguese immigrants to the U.S. Most settled in Fall River, New Bedford and Taunton in Massachusetts as well as around Providence in RI. By the early 1900s Azoreans began settling in the San Joaquin valley of California where they became dairy farmers.
After 1950 there were a smattering of people from the mainland who settled mostly around Newark, NJ in Connecticut and Southern Massachusetts, but most Azoreans still accounted for 70% of the overall total. In Hawaii there were immigrants from Madeira, but this migratory movement ended by the First World War.
One of the problems with attracting large numbers of Portuguese is that Brazil was a very easy country to emigrate too even after its independence from Portugal. Portuguese immigrants were able to easily assimilate in the country and were exempt from quotas that other nationalities were subjected to in the 1930s. Brazil was the destination for 90% of the Portuguese emigrants prior to 1950.
In the 1950s Portuguese emigration starting spreading to new countries. Angola and Mozambique overtook Brazil during this period as the primary destination for Portuguese emigrants, though they were not counted as emigrants since they were technically speaking domestic migrants. By the 1960s with a boom in Western Europe, France became the primary destination as it was easily accessible and would continue until the mid-1970s.
Even if all of the Portuguese who went abroad since 1950 emigrated to the U.S. they still wouldn't equal the sheer numbers of emigrants from Spanish-speaking countries. At most you'd get 4 million Portuguese speakers from Portugal. Below are figures of the leading destinations for Portuguese emigrants since 1950. Keep in mind that Portugal has a high rate of return migration with many of these even from the U.S. having eventually returned home. Incidentally the highest rate of return tends to be amongst Portuguese from the mainland. Also, Western Europe especially Germany have had the highest numbers of returnees with well over half of the emigrants having been temporary.
France 1,055,000
Angola 375,000
Brazil 325,000
Germany 275,000
Mozambique 235,000
USA 200,000
Switzerland 180,000
Venezuela 165,000
Spain 160,000
Canada 150,000
United Kingdom 85,000
South Africa 80,000
Luxembourg 55,000