The issue with early modern Scotland is the massive issue of poor government. You constantly had Kings dying in their prime during wars with England and a succession of child kings, who in turn would grow into their youth and promptly die, being succeeded by another child. This was essentially true from the time of James I to Mary, Queen of Scots (although she didn't die, she was still deposed and succeeded by her infant son, much as she succeeded her own father as an infant), although both James IV succeeded at fifteen, which wasn't as bad as some of the younger ones.
This constant strife gave the Scottish barons great freedom that by the time of Mary was rampant. Royal finances were also in terrible shape, with many royal lands having been leased off into noble hands. Mary's religion wasn't the only reason she was disliked -- it was also the fact that once she turned twenty one, she'd be able to reclaim the lands that had been leased out in her youth, and would definitely help bring the Scottish crown out of penury: in the time of Marie de Guise's Regency, the crown brought in about 40,000 Scottish pounds per annum, compared to the 229,000 pounds per annum that the English crown earned c. 1558-1559.
Antonia Fraser wrote that, "The total royal revenue in 1560 was around £40000 Scots or about £10000 sterling. Compared to this, ... state of the country dwelt vividly on the poverty of the Scottish monarchy, which it ascribed to the lack of a proper royal domain." While
here sheds some light on English finances in the same period, page 24 which covers some info on the English finances in the early period of Elizabeth's reign. The book on a whole is very good for studying the finances of the crown in the early modern period.
Another Scottish issue would be the population. It's population in the period is quite small. England had a small population as well (compared to say, France and Spain), but the enclosure movement created a class of people deprived of land and pushed into the cities. I could see them managing to found a colony somewhere though, although it would have to be somewhere small. After all,
Courland tried (and) failed in colonizing Tobago. If anything, Scotland might be more successful in mercantile colonization, such as trading posts, that sort of thing. Even in the early modern period Scottish trade was important; although it's main partner was England, it traded abroad, too. Surprisingly, Scots could be found in Poland as traders and even mercenaries, and had a quay to themselves in Krakow.
This book has details on the Edinburgh merchant class and their trading abroad in the early modern period. Interesting stuff.
To effectively be able to colonize though, Scotland needs a stable government, peace with England, sounder finances, and above all, the will to explore options abroad.