Actually it is legally debatable -
There is a mixture of tradition - male preference primogeniture, proximity of the blood etc that would all play a role - the entails passed carried legal weight.
Alexander III had passed a law to allow his granddaughter to succeed - she was the only viable candidate at the time (Though he did indeed hope for further issue by his second wife) - her death meant of course a crisis between the rival claims of Balliol (superior in primogeniture) and Bruce (superior by proximity of blood) - Alexander also inserted clauses which mean that seniority of blood took precedence - so if he had any issue by his second wife (even a daughter) they would succeed as nearer in the blood (being his children) in preference to Margaret (who was a generation removed from him).
The best legal advice at the time from Paris was that Scotland had no formal law on the matter and should follow Roman law which considered proximity of blood the higher claim. Which also reflected the situation when William I had tried to persuade the Scots to accept his daughter (it was before his son's birth) they instead insisted his brother and his son came first then Margaret.
Robert I's first parliamentary act of succession of April 1315 which recognised Robert's younger adult brother Edward Bruce and his direct male descendants as heirs presumptive to the throne in the event of Robert I's death without sons, ahead of Robert's daughter, Marjorie Bruce, and her direct descendants to whom the throne would only 'revert' after the passing of Edward and his sons, grandsons etc.
Restated in 1318 after Edward's death without legitimate issue.
"but that when a king dies, the nearest male in the direct line of descent, or if a male was not available the nearest female in the same line, or if that line is entirely lacking then the nearest male in the collateral line, ought to succeed the
king in the kingdom, with concern for the right line by blood by which the right to rule applied to the dead king, the succession to whom will hopefully be achieved without challenge or any kind of obstacle."
In 1326 a fresh entail named Robert I's grandson Robert Stewart as heir to Robert's infant son David.
David himself attempted throughout his reign to remove Robert as his heir - by the children of his full sister (who were closer in the blood to him) and later by an English Prince of his choosing.
He failed and was succeeded by Robert II the first Stewart King - He would pass a new act which effectively dictated that the male heirs took priority over female heirs.
The succession would pass to the "the sons of the king, of his first and second wives, now born, and their heirs male only, shall succeed one after another, in turn."
It did not exclude women explicitly and ended "the aforesaid five brothers and their heirs male descending from them happening finally and wholly to fail (which God forbid), the true and lawful heirs of the royal blood and kin shall thenceforward succeed to the kingdom and the right of reigning."
The Stewarts and the Bruce's couldn't follow the trend of barring female line succession because they themselves relied on it for their claim to the throne - the fact was that the Stewarts were fertile through the late 14 and early 15th century and no further amendments or laws were passed.
Assuming a surviving Albany male line - then in the 1540s they have an exceptionally strong claim to succeed in preference to Mary under the 1373 entail which specifically states the male line first. Mary could of course defend her claim using proximity of the blood and the 1318 entail - which placed a female direct heir higher than a collateral line.
So some choices - James V lives long enough to pass a new entail ensuring in the failure of his male line his female descendants took priority, the Albany Stewarts are loyal and lack ambition and support Mary's claim based on her being the senior heir general, or the Albany Stewarts claim the throne as heir males and plunge the Kingdom into a succession crisis as followed Alexander III's death - or the then Duke has an appropriate male heir who is married to the infant Mary - or the Duke himself arranges to marry her - uniting both lines and ending any dispute.
The Earl of Arran was placed in the succession only after Mary's succession (he was regarded as next in line in default of the Stewart line) - he was not considered a potential alternate male heir at the time of Mary's accession.
The Lennox Stewarts claim was also through James II's daughter (they argued Arran's claim was invalid due to a dubious marriage)
Mary succeeded because she was the eldest child of her father, and she had no brothers. Scottish succession was almost always fraught with issues, and during the 15th centuries they typically dealt with long regencies and numerous child kings—the Kings of the Scots had numerous issues staying alive, though most of them at least managed to beget a son before being killed in battle, or in the case of James I, being assassinated.
Scotland had followed primogeniture from the reign of David I, passing from father to son. In rare cases it might pass from brother to brother, but after that time there was no case of male seniority—any other Stewarts would come behind Mary in the line of succession. Scotland had accepted female succession when Margaret the Maid was briefly hailed as queen, and Robert II was a successor through the female line, that of his mother. After all, the Scots had a plausible male successor in the Earl of Arran (yes, he was a Hamilton, but who's mother had been a Scottish Princess), but he still placed behind Mary in the succession. The Lennox Stewarts are an interesting case in that they are a junior branch of the Stewart dynasty, but descend from Alexander Stewart, the 4th High Stewart of Scotland and not Robert II, the first Stewart King. From the time of Robert II, entailments had been passed regarding the succession: at the time, none of his sons had heirs, so it was necessary to have a system be devised to define the precise circumstances under which his sons could succeed to the throne. None of these entailment's took precedence over primogeniture, though, and Scotland essentially had a semi-salic succession.
James V clearly lamented the birth of his daughter, with his famous saying, but she still succeeded him despite that. Now, is it possible that with a surviving line of Albany that he could be a viable successor to James V? Totally. But given their numerous French connections and their oft exile to that country, some Scots might be suspicious of them—Scotland seemed to swing in between massive love to the old Auld Alliance, and disdain for it—especially in this period. Albany had also served as Regent during James V's minority, but it wasn't a great successor. There's a chance the Scottish nobility might prefer Mary, even if a female, who was born in Scotland, versus Albany's French born and raised son.
I'd personally say Mary still becomes queen. But Albany's son (assuming Albany still dies in 1536) would definitely be her heir until she marries and has children, and he'd have the strongest claim to the Regency in this situation, too. Given Marie de Guise's preference for the French, I could definitely see her inviting Albany's son out of exile. Given the Albany's powerful French connections, they could probably arrange a marriage between Mary and the son or grandson, the Count of Auvergne and Lauragarais to effect a reconciliation between the Senior and Albany Stewart lines.
