Good ol' Free market. They dominated international system of trade and so were really freaking rich.
Hence the VOC being a monopoly. Free market doesn't describe it well.
Dutch control of the Baltic trade was wrestled from the Hansa through war (and having ships that could make the leg from Poland to the Netherlands in one go, bypassing the Sound/Bight ports), Dutch control of the Indian trade was wrestled from portugal through war.
Now, those two created a rich basis for Dutch merchants to then compete 'fairly' in other European, Asian and American markets, but it was first based on controlling major tradeposts through force.
In my opinion, the critical ingredients are:
1) Urban culture, based on havng good riversystems and good access to the sea, coupled with decent agriculture (see also, Italy)
2) Relative unity, giving a single state able to skim off a bit from all that trade.
3) Government by the merchants for the merchants, so wars were for trade rather than for territorial growth.
4) all the above leading to being way ahead in monetary development, so the Dutch state could outlast tax-based governments financially (the Medway was a result of the Dutch navy outlasting the English, of all people, financially)
Or, in a single sentence, the Dutch Republic was like an Italian merchant city state (esp Venice/Genoa) but bigger and thereby even more powerful.