What if a nation be it Denmark or otherwise chose to use Greenland similarly to how Australia was first settled as a penal colony?
The simple answer is yes, but there was a many good reasons, it didn't up that way.
1: How Denmark saw the Inuit. Denmark saw the Inuit as radical different from how other countries saw the natives of their colonies. Denmark saw them as Danes. This was rather unique even in the Danish colonial empire, but the rason for this was that Denmark had claimed Greenland siunce the union with Norway began. This meant that Denmark saw no reason to Danify Greenland as it already was Danish in the view of Danish state. As for the Inuit their interaction with the Danes was mostly positive.
2: Any penal colony simply depend on the need for unskilled labour. The problem was that the Danish policies for Greenland didn't have a need for unskilled labour, the Inuit was already better hunters than most Danish criminal, and the Danes which dwelled in Greenland was usual skilled.
3: While agriculture was possible in Greenland Denmark saw little reason to establish it, the Danish economic policies on Greenland was based on hunting and whaling in the 18th century, hunting and mining in the 19th century and fishing and industrialisation in the 20th century.
I think no. 1 are unlikely to change, in fact the establishment of Danish control over Greenland more or less build on it. So let's look at the other.
I think we will start with creating a interest in local farming in Greenland. So let's start with creating that interest. In 1742 Anders Olsen a Norwegian arrived in Greenland, Olsen planned to create a European settlement in Greenland, it didn't end that way, instead he became a local administrator. In 1753 he founded a trading post, but he did in a area where agriculture was not really possible. But let's say that Anders Olsen in the 11 years between 1742-53, he end up explore the inland valley's of far southern Greenland. Here he find Qinngua Valley (known as Paradise Valley to the Danes), the valley are home to the only natural forest on Greenland. This inspire Anders Olsen to question whether agriculture would be possible on Greenland, in the kitchen gardens of the local Danes, they already grow their own potatoes. So he contact the crown in Denmark to support a experimental farm (not in Qinngua but in one of the other valleys close to the sea), the crown end up deciding to support Olsen. He tries to hire Norwegians and Icelanders to help setting it up, but few are interested in the hard work in setting up a farm in even more marginal land than their homeland. The Danish crowns decides to send some prisoneers to Olsen's farm. So in 1752 Olsen set up the experimental farm (Skovgaard). Olsen mostly focused on sheep, while the prisoners grew potatoes, hay and experimented with other crop. It wasn't a big success, but it created a small mixed population producing a surplus of food. The farm also planted timber plantages, creating a few artificial forests. But mostly the experiment was forgotten until the 1770ties. In the early 1770ties the Danish state invested in a whaling company on Greenland. The whalers was forced to return to Europe in the winter. But someone became aware of the Skovgaard, and it was suggested that agriculture could be expanded there to feed the whalers through the winter. So Anders Olsen was made responsible for expanding the settlement. Andersgaard now became centre of several farms being set in the same region, while a port (Hvalhavn) was founded for the ships. The new farms was settled with Danish criminals. The result was a increase in agricultural output, but it was limited. The food production was expanded to fishing too. All this lead to a more successful whaling adventure and by 1800 Greenland had become home to around 2000 settlers beside the 5000 natives. There was little conflict between the two groups as the Inuits was hunters, while the Danes was herders, farmers and fishers, and the there was enough work in the whaling fleet for the Inuit to buy food, instead of killing the Danes sheeps.
The Napoleonic Wars changed everything, the colony was isolated for years under the Danish-English Wars. But as Denmark lost Norway, the Danish state decided to develop its oversea possessions further. Some of the agricultural developments was introduce to Iceland, where the more frost resistant Greenlandic potato was a success. Another grop from South America was also introduced; the quinoa. Which would see a expansion of agriculture on Greenland and Iceland. Denmark also began to expand the coal mines on Disko Island. In general it was in this period that large numbers of Danish criminals was send to Greenland. Enough that by 1848 the population was split between 9000 Danes and 9000 Inuits, but with the establishment of Democracy in Denmark the banishments of criminals to Greenland stopped. This lead to several decades of slow development. But as Denmark lost Schleswig-Holstein a new wave of improvements on Greenland, hydroelectric power was introduced and the first iron mines was set up at Godthaab (Good Hope; today's Nuuk) and Greenland saw a influx of Danish miners. As such by 1900 Greenland was home to 25.000 Danes and 11.000 Inuits. Greenland had begun to change in the inland valleys, there was farms and timber plantages. The Inuit already many with Danish ancestry begun to adopt the Danish language and the the difference between Danish and Inuit Greenlander had begun to break down, many of the Danes also had Inuit ancestry. In fact the main difference between the group from Disko Island to the southern tip of Greenland had become mostly a question of lifestyle. There was attempt to set up a agricultural colony on Svalbard, but the climate was to hostile, but the Danes did end up setting up a fishing village and mining villages, the result was that Denmark ended up claiming the island.
Denmark continued to develop Greenland through the 20th century. Greenland saw occupation by the Americans under WWII, and American bases was established in the Post War period, in 1953 Grenland was fully integrated into Denmark, there was little demand for autonomy, as the population felt Danish.
Greenland today
Greenland by 2017 are home to 160.000 people. Around 35.000 identify as Inuit. Danish are the mother language of 140.000 people, while around 20.000 speak the three Greenlandic Inuit languages. The population are 95% Lutheran with the minority being mostly non-religious.
Political Greenland are one of the six Danish regions. East Greenland (Tunu), North Grønland (Avannaa) and North East Greenland (Upernavik) are semi autonome municipalities with the three Greenlandic languages as co-offical languages. The rest of Greenland are split up in several munipalities. The municipality reform which set up Greenland as one region, gave the Greenlandic Region more power versus the municipalities and Danish state than in Denmark. This was necessary because it also needed to control the local infrastructure and education. Greenland have one university in Godthaab. It only offer theology in the native Inuit languages (only east and west Greenlandic). The only other tertiary education which are offered in Greenlandic languages is teaching. there's four native high schools one in Tunu, one in Avannaa, one in Upernavik and at last one in Godthaab using West Greenlandic (the most common spoken dialect).
Godthaab are regional capital and are home to 80.000 people. The vast majority are Danish, but around 10.000 identify as Inuit.
The main green export of Greenland are raw material, fish, wool and aluminium (aluminium are mentioned separate, because it's not mined on Greenland, but instead the large hydro power of Greenland are used to process it from raw material into aluminium).