They'd need better boat-building. The sea ice does not extend to Iceland, especially not in the warmer periods in northern Europe. Crossing open seas in the sort of boats the Dorset had is extremely challenging. No point to try and cross that far when there's plenty of food elsewhere. Not impossible, just very difficult to see it happening.
If they did though, they'd find it to be a pretty good environment, since there's plenty of animals and plants to harvest in a relatively temperate environment. Iceland could probably support a sizable population of them. The Celtic monks who lived in Iceland before the Norse would no doubt find the situation interesting.