Firstly, they didn't have inferior nautical technology; they had extremely sophisticated nautical technology (both in terms of the physical design of their boats and the navigation methods they used). The rest of their technological package was literally Stone Age, but their boats were excellent.Another question to be answered is why the Austronesians with much inferior nautical technology than the Europeans were able to colonize all the Pacific Island AND Madagascar. What did the Europeans lack compared to them? Did the Austronesian colonize the Americas too ?
It is believed that they at some point had at least some contact with the Americas prior to 1000 AD, due to the presence of New World crops (specifically the sweet potato) as part of their later crop package, but this contact seems to have been extremely limited.
The Austronesians had a couple of advantages. Firstly, unlike the Europeans, they were navigators more or less by necessity. The western Pacific is full of little tiny islands to settle. The Austronesians mostly seem to have avoided colonizing previously inhabited land, but there are plenty of small islands that they can and did settle. The keyword here is small, which means that they eventually have a population surplus, and explorers just have to sail east for a few days and find the next island to settle. Unlike Europe, you have to get used to the tricks of navigating far away from the coast, because "the coast" is a tiny atoll. This process, over millenia, takes you a good chunk of the way across the Pacific. Combine with excellent nautical technology and culture, and you're set. Note that New Zealand, one of the last places they settled, as well as one of the largest, suffered a significant decline in navigational technology by the time the Europeans came, as the traditional small ocean-going craft and tricks were no longer necessary for life on such a comparatively large landmass.
The Europeans, on the other hand, had a massive landmass to inhabit. The Atlantic also has some islands, but nothing like the plethora that Polynesia has. It's much easier in Europe to expand by conquering your neighbor, so focus was generally on that side of things. Even sailing is generally coastal most of time; there's lots of coast, you can stop at ports if you run into any trouble/want to trade/need R&R. The single-minded Austronesian focus on nautical technology/culture is unneeded.