As per usual a lot of people are forgetting the big important defining factor of history- economics.
Pre WW1 Britain was super dominant. It merely had to mutter that it didn't much care for another country to send stop markets tumbling. It takes WW1 for Britain to completely lose its credit rating and the US begin to emerge as the financial centre.
I really think in 1865 the US economy is still too weak to be able to weather the storm British displeasure would bring; just consider the massive mess the Trent Affair caused in the US economy. Really, pre....I don't know, somewhere in the long depression, the US economy was very much ancillary to the British economy.
This may not be such a big deal for many countries but considering capitalism is at the core of what America is....war with Britain wouldn't be pleasant even without any shots being fired.
For a very good analogy just imagine what would happen if the US decided it didn't like the UK today, Britain would surrender without America having to do anything military. Our economies are just so intertwined and the UK is the junior partner. The mid-late 19th century was very much the same but in reverse.
More like the 40s. Even before the war was over, the US already had by far the world's largest navy.
A navy isn't the key thing for invading across the atlantic IMO. More important is the air force to clear the way for the invasion ships to cross. Its not until we get firmly into the cold war that aircraft become quite solidly long distance.