I actually think this is an interesting idea. Firstly, you'd have to work out why France is deciding to send its criminals overseas than it is in executing them or using them as galley-slaves. Certainly transportation is a very expensive form of punishment.
However, there would be a couple effects:
- the white population of New France might be a little higher than it was OTL due to no need to recruit colonists.
- colonists would come from slightly different sectors of society. You'd have urban criminals and highwaymen/bandits rather than peasants. This might mean a greater diversity of skills amongst the settlers, but less of a willingness to settle down to a life of farming.
- I think you'd see a lot of the colonists fleeing into the woods. Unlike the East coast of Australia (which has mountains and then desert on one side and ocean on the other), the St. Lawrence Valley is located in the middle of a huge forest that's easy to hide in. Where these colonists would end up would be I think the most interesting part of this TL. They might end up just taking on a coureur de bois lifestyle and trade for furs. Or, they might end up as bandits attacking the voyageurs. Or, they might marry into Native cultures, creating an earlier and more numerous Métis ethnicity. Or, they might end up setting up their own towns and villages out of the watchful eye of the colonial government.
- I really like the idea of settlers pushing inland as a way to get away from the colonial government. You'd get an earlier expansion of settlement into the Ohio valley from the North, maybe?
- these early inland settlers would be culturally French, but may be Protestants, and may be very much politically opposed to the French government, so maybe they could end up under English protection.... I'm curious to see what this sort of nation would turn in to....