People have already illustrated the main difficulties of a Swedish colonial empire. Controlling the Baltic is a far preferable option for an ascendant Sweden than far flung colonial possessions, and Sweden does not have the resources to maintain both a brown water and blue water navy. It army would also be all but useless for colonial enterprises.
Regarding the possibilities of taking Norway/Denmark-Norway, it might be possible to do it post Kalmar Union and before 1658. During the regency of Christian IV, the Danish-Swedish rivalry wasn't quite as entrenched and I think I remember reading that several members of his regency council preferred to seek peace with Sweden. Christian IV and Catherine of Sweden, daughter of the future Charles IX, were similar in age. As Charles was just a duke at this time, the marriage wouldn't have been thought to threaten the independence of Sweden.
Such a marriage could prevent the Kalmar War, and Christian IV might choose to look westward more and claim a small colony on the eastern seaboard of America like the French, English, and Dutch did between 1604-1611. There might be butterflies in the 30YW, but it's hard to say. Sweden probably does better without Christian IV working so clearly against Sweden at times, so Sweden might get more north German territories. With Gustav Adolphus still dying and Queen Christine still being her OTL self, the son of Christian and Catherine is set to create a personal union. As there are still bad memories of the Kalmar Union, that no doubt creates some friction. Probably a civil war breaks out between to determines whether Denmark or Sweden is the dominant partner. Sweden wins, starts imposing its policies on Denmark and Norway. Sweden has Riga and Narva already, so has its main Baltic goals, and probably more north German territories than OTL. It suits Sweden to orient Denmark-Norway towards a blue navy, both to prevent it from competing with Sweden's brown water navy and to allow them to strike against the English and Dutch for when they start to work against the Swedish Baltic dominion. This allows them to at least start a colonial empire, whether it grows more than a few minor islands or outposts depending. If this personal union survives long term, meaning Russia doesn't turn its gaze north and Prussia is prevented from rising, this Swedish led union becomes the uncontested power of northern Europe. It probably partitions Poland with Russia and Austria, continues spreading its influence in Protestant Germany, etc.
Although whether this could be called a 'Swedish' Empire is debatable. Gothenburg probably becomes the capital at some point, as it is a more central Scandinavian location and more neutral historically. As for all its other territories, this would basically be a Hasburg-esque collection of territories with different laws, jurisdictions, inheritance laws, etc. It would take luck and skill to create a common identity so nationalism doesn't hit it hard, but if it does it's almost certain that this polity would have the excess population, resources, and prestige to gain at least a nominal colonial empire somewhere.