Poland's main issue is its geographic location, with its only coast being in the Eastern Baltic. This means Polish shipping to America is going to have to sail past the narrow sound between Helsingør in Denmark and Helsingborg in Sweden, making it vulnerable to Swedish attacks, and keep in mind during the period of colonisation Sweden and Poland were often at war. In addition to this, the Danish collected tolls on all ships passing through the sound at Helsingør.
This is coupled with the fact that the Dutch controlled the Baltic trade, and they would more than likely oppose a stronger Polish merchant fleet. Another problem is that the coastal waters of the Baltic are shallower than those of the Atlantic, meaning that the ships would often be unsuited for transatlantic crossings. However, this would be the least of Poland's problems.
Most importantly, colonial expansion required peace, something that Poland lacked throughout this period. Distracted by wars with neighbours, little funds could go towards building a proper navy. France and England for instance were only really able to establish large colonies during periods of peace.
Finally, the largest Polish port was the Free City of Danzig, opposed to the presence of a large fleet. In addition, the Polish nobles were against spending money on a navy as they saw the army as a more important priority.