I'm afraid I'd have to go with economic determinism here. It may be true that back around the year 1000, the population of what is today "Norway" matched or exceeded that of modern Denmark. But things have changed since then and I don't think it reflects an imperialistic Denmark grabbing from Norway, but other way round, Denmark could grab Norway and seize the territories originally colonized by Norse because the balance naturally shifted south. (Then of course the Danish kings did indeed aggrandize what was called "Danish" versus "Norwegian" in favor of Denmark--correct me if wrong but I believe for a long time it was Danish policy to pretty much deny "Norway" existed save as a geographic expression of a region of Denmark--the Norwegian language had no recognition as anything other than some local dialects, etc. Am I wrong in thinking the kingdom of Norway was completely extinguished for a while?)
Part of what has happened is the Little Ice Age; Norway and Iceland and even Greenland used to be warmer around the Viking Age. Part of it is that the more intensive forms of agriculture developed in the Middle Ages and the various phases of Modern times are much more favorable in soils and climates like Denmark's than Norway's.
So, we could play a game where the territory we call "Denmark" OTL gets absorbed into the realm claimed by a strong Norwegian dynasty, instead of the other way round, but we would be bucking the odds. Possession of the sea islands even including Iceland would not be a tremendous asset compared to the agricultural potential of Denmark, not even to mention the fact that control of the straits into the Baltic was a literal gold mine for the Danish kings. Even if the ruling dynasty originally came from the north and aggressively renames the islands and Jutland as being mere provinces of "Nprway," the economic and demographic center of this "Norwegian" kingdom would not be in Norway but in Denmark; I'd think the capital would be moved down to there sooner or later.
I would guess there might have been a fair chance for Norway to avoid coming under Danish control, or for there to be two Norways, with one a province or three of Denmark in the south, and a separate independent nation centered on Trondheim. I also think it would be in the cards for whoever controls Trondheim to have extended control over the mountains to the region of OTL Sweden that is near those passes, to the Gulf of Bothnia, and if that region remains annexed to Norway for a long time it would block Swedish expansion north, so the map might be very different, with Sweden a much smaller state that may or may not also have lost territory to Denmark (or vice versa some act of conquest or dynastic union might produce a unified nation out of both, I suppose) while Norway holds the entire northern part of the Scandinavian peninsula, but perhaps not the southern provinces it has OTL which might be Danish or Swedish.
However there is much to be said for the current borders that do seem to follow a certain natural and geographic logic.
-----
One time line that has been developed here though it seems to have gone into hibernation is Lands of Ice and Mice. DValdron having developed an excellent case for cultivars that can be grown and eaten in the high Arctic lands, his "Thule" (an ATL variant of OTL's Inuit) offer a whole basket of crops and domesticatable or anyway manageable animals that could greatly raise agricultural production in Norway and put Norway, as currently drawn on maps, back in the game. If that is, the Norwegians stoop to growing and eating Thule crops! In the canon TL DValdron actually did provide for that, with the result that a new Norwegian kingdom centered at Trondheim does spilt off from Danish control, and might manage to reclaim Iceland while eventually expanding southward to round out to OTL borders. If one can farm the Arctic, then Norway is in a much better position than if one cannot. Other fans of the TL even wanted the Norwegians, with their infusion of Thule subjects, to sweep east across the entire Arctic seaboard; I personally wanted to see what the north coastal Russians would do with a similar demographic boost from Arctic-farmable crops. But anyway Norway in that ATL could easily wind up extending farther east, perhaps taking the whole Kola peninsula. In such a TL Iceland, and even Greenland, are more desirable possessions. But Greenland is a major center of Thule culture and quite resistant, at last report, to invasion by Europeans; Iceland, it seems to me, is well on the way to becoming a republic again, or anyway an independent kingdom or duchy, and if it does associate with Norway it will be on voluntary and mutual terms, not as a mere colonial possession.
However Svalbard and other Arctic islands are also viable colonies now and IIRC that island does fall in the Norwegian sphere--or maybe it was still Danish last time we looked. However the Danes are definitely facing a challenge from Norway, and if they are going to keep their nominal claims, they are going to have to do some fancy footwork that probably will involve devolving a lot of power to their holdings, lest the colonized jump ship and turn to the Trondheim based kingdom for backing instead. (Or the English, or the French, or Spanish...)
So if you've never read Lands of Ice and Mice, I suggest you do so and perhaps lobby DValdron to resume work on it; it was always one of the most startlingly creative ATL ideas I've ever seen and DValdron can be relied upon to flesh things out interestingly and with good consistency.
------
Sadly though, without some ATL revolution in agriculture or fishery or something, I can't see Norway doing better than maintaining a poor but dignified independence, and even that is a long shot considering the resources Sweden and Denmark can bring to the struggle.