Born in Pont-d'Ain, Philibert of Savoy was the son of Philip 'the Landless' of Savoy, a junior member of the ducal family of the Italian Savoy, and his first wife Marguerite de Bourbon. Philibert would have 2 other full-blooded siblings, his elder sister Louis, who married the French Count of Angouleme, and Girolamo, his short lived elder brother. He would also have 6 half-blooded siblings by his stepmother, Claudine de Brosse: Charles of Savoy, Louis of Savoy, Philip of Savoy, Assolone of Savoy, Giovanni Amedeo of Savoy and Philiberta of Savoy.
In 1496, Philibert's father succeeded as Duke, when his young grandnephew Duke Charles II of Savoy died. This death left Philip the Landless as the first male line heir. In this same year, Philip had his son Philibert, who was 16-year-old at the time, marry the 9-year-old Yolande Louise of Savoy, his cousin and the only sister of the deceased young duke. She was daughter of Duke Charles I 'the Warrior' of Savoy and Blanche of the Montferrat. Yolande Louise was also, after the death of her brother, was the succession of the kingdoms of Cyprus, Jerusalem an Armenia, although Philibert's father took those titles. Philip II of Savoy would die shortly after ascending to the Dukedom. Philibert then became Philibert II of Savoy.
In 1499, 3 years after ascending to the throne, Philibert's bride Yolande Louise died at the age of 12, leaving her cousin Charlotte of Naples as her heir. Philibert would continue to use her titles. Due to his widower state, Philibert searched for a new spouse and eventually settled on Margaret of Austria in 1500, sister to the Duke of Burgundy and daughter to the Holy Roman Emperor. With this connection Philibert was able to protect himself against the French, who continued to war through Italy and thus were always on the cusp of attacking Savoy.
The couple would be childless for the first few years of marriage and in 1504 Philibert took ill, leaving the possibility Margaret would again be a widow. However, he pulled through and by the end of 1504 was healthier than ever.