Colonial, and more modern, Australia has a long history of paranoia about foreign invasion. I'm not sure but I think just about every colonial power other than perhaps Denmark was suspected of being a threat at one time or another. Even today there are some nutters in Cowra, central western NSW, who fear invasion by Indonesia. Of Cowra, over 10,000km from Timor!
There's a good scene early in the 1980s movie Gallipoli where the eager hero is off to enlist, dragging on a young but cynical Mel Gibson, and they meet an old bushie and tellis him Australia is at war and they were off to sign up to fight the Germans. The bushie asks "why?" and the hero says "because Kaiser Bill wants to take all this away from us". The old bushie squints through the heat haze at the parched desert around them and says "he's bloody welcome to it".
Colonial Australia built some very impressive/expensive defences when the distance factor alone was a major obstacle for an invader. An even greater deterrent is that invading 1850's Australia meant war with the greatest power of the age (Britain). If France, Russia etc weren't already at war with Britain, why would they want to trigger a major conflict over a minor outpost? If the aggressor was France, surely it would make more sense for them to go for a Carribean island, or maybe a west African colony? Russia would likely aim for India, America would find parts of Canada or Jamaica more valuable etc. I'm not clear how hampering the trade in gold from Australia is going to help any rival to Britain? The Royal Navy was unstoppable in the 1850s and will quickly recontrol the seas around Australia sinking or blockading whatever forces the invader sends. And then the RN does the same to the invader itself, starving them into submission if necessary.