It's not at all impossible that the Japanese could have - had they really wanted to - staged an assault into Darwin, and even secured the port and immediate environs, had they struck early enough in 1942. The problem is, you are going to have to write off any garrison you send there.
Isolating and destroying a Japanese garrison in Australia would have required a massive input of Allied resources to Western Australia, so almost by definition if the IJA force was in jeopardy, the Allies massively overreacted and the diversion worked. Generally speaking, while elimination was possible, given the size of the coast and domination of the Japanese navy in these waters, extraction with or without equipment seems more likely.
Otherwise, they're basically a giant POW camp that the Allies don't have to support. They have no ability to get anywhere else in Australia, and the Kimberley area will struggle to support them indigenously in terms of food.
An invasion can hop down the coast westward towards Perth using light shipping resources. The Australians then freak out and MacArthur pours reinforcments in and the Solomons is indirectly defended....
.