Western Australia naturally complements the Cape and Indies, and could carry on a nice trade with that part of the world. Unfortunately, the OTL Dutch weren't into settler colonialism. But if they were a bit more interested, and found more gold at the Cape and in W. Australia found gold and sandalwood, then you could get a sizable population there which eventually might lead to the Dutch grabbing the rest of Australia. The population of the Netherlands doesn't matter too much, since IIRC it was higher than Portugal (and look what they did with their empire!) and we don't need a ton of emigration to the Cape and Australia to get things rolling. In time, I think Australia would surpass the Cape and South Africa since shipping tech will improve and there's much more open land since the Aboriginal population will collapse like OTL. Even with a Mfecane like event and some rinderpest and slave trade, much of South Africa will still have more people with better organization than Southeast Australia.
That said, I doubt the population would be much more than OTL even here. A lot depends on their immigration policy which might resemble OTL South Africa. A lot of Malays, Chinese, and even Africans will be present from the start, which will affect their attitudes toward non-white immigration, while a trend toward conservative Calvinism like the Boers will likewise affect their stance.