Japan was never a high priority, not when the EIC was putting all its effort into trying to get China to open. China had silk and porcelain and tea, what did Japan have that was worth messing about with the Dutch and the Shogunate for?
By the time implied by the OP*, the Dutch didn't really have the power for messing with the VOC, as it's charter had expired in 1800.
*BTW, I am runing on the assumption that the OP implies a bakumatsu POD.
well, if it was the dutch who were the big players in japan, what if they had added it to their empire, or a resurgent portugal, or better still Prussia?
Doubtful for the Dutch. They actually weren't that big - for quite a long time they were limited to 2 ships at Dejima. In fact, the declining Dejima trade cotributed to an overall decline in the VOC's fortunes. (Note: That's contributedto, not caused. Highly doubtful for Portugal - they got kicked out for a reason. And utterly ASB for Prussia.
Could Japan at some point have become a Dutch colony??
See above.
The main problems in a colonial Japan scenario are purpose and "crackability".
As already mentioned, there's very little Japan has that isn't more easily available elsewhere. It makes a nice stopover for your China traders and a base of operations for your whalers, especially once you're running on coal. But from the late 1500s onwards, it's a unitary state that has developed or is developing some very modern institutions and technology.
Japan was one of the earliest nations to widely deploy small arms on the battlefield effectively, even if they later gave them up. The merchants of my namesake city were quite sophisticated economically, as were those in Edo.
There are POD's that make for some client states and trade missions, but colony is mostly a no go without something major well before the bakumatsu period.