They were probably also limited by the very fact that there were relatively few ships travelling back and forth from China to America at this time, at least those taking passengers for scheduled runs.
For instance, it was unlikely that any given Chinese could have known when or where a US ship would show up, how to negotiate passage, where they would be going (it would only be luck if they were headed to California or wherever the Chinese wanted to go), if they could afford passage, know enough about America to want to go there, know how to get back, etc, etc, etc.
Also, trade had been damaged in the depths of the Taiping Rebellion and the reputation of all westerners was low.
The Taiping Rebellion (and northern Nian Rebellion) being more successful is a subplot of my "Quasi-War 4" Timeline ongoing now.