Was immigration down very significantly during the Civil War. I would think it might have been. That should be factored in as well.
Compared with the decade prior to the war, there were 742,000 fewer immigrants during the 1860s. However, the 1850s were the decade where net migration accounted for 33.2% of decennial population growth in the U.S., this sank to 25.3% in the 1860s and 24.2% in the 1870s, recovering to 30.9% in the 1880s.
It's hard to say what would have happened without the U.S. Civil War, as immigration only declined temporarily. However, the postwar boom attracted large numbers of immigrants, and this boom was followed by the panic of 1873, which would keep immigrant numbers down until 1880. Notice the panic of 1857 had also had an effect on immigrant arrivals to the country, particularly as this hurt the great lakes region and the Northeast, where the majority of immigrants settled. The Civil War brought a stimulus to the economy and by 1865 immigrant numbers were back to their 1857 levels. Economic factors drove immigrants during this period particularly as news travelled faster than before.
U.S. Immigration
1850 369,980
1851 379,466
1852 371,603
1853 368,645
1854 427,833
1855 200,877
1856 200,436
1857 250,882
1858 122,872
1859 121,075
1860 153,418
1861 91,822
1862 91,826
1863 176,214
1864 193,416
1865 248,111
1866 318,491
1867 303,044
1868 288,088
1869 363,074
1870 402,920
1871 342,609
1872 422,978
1873 473,141
1874 327,949
1875 244,632
1876 189,991
1877 165,019
1878 157,776
1879 197,954