So the
abdication system becomes the core political structure in China? It was apparently an influential enough idea in the Warring States period that some surviving texts sought to critique it, or criticise King Kuai of Yan who used the political structure of that legendary era of China to abdicate to his chancellor (and not long after was defeated through a conspiracy by his son and the nearby state of Qi). The system was criticised as one not suited for the current era, therefore an inevitable failure--that might have some fact to it since it is hard to imagine a country totally dominated by scholars in such a violent era.
The texts that actually supported this were mostly destroyed, but one was discovered and laid out the example "a man must study from 20 to 30 under a scholar, must serve as a scholar/sage from 30 to 50, and then may only rule as king from 50 to 70 before he retires." That seems like a system where there would be checks on the ruler's power.
It seems part of the reason this sort of revival never caught on was because King Kuai and his chancellor were not very good rulers hence were ripe for criticism that the idea was ill-suited for the era. But had a successful state done so, like maybe a ruler whose heirs are either dead or thoroughly incompetent, maybe the framework for the idea could have been laid. And then finally this successful sage-republic conquers the other Warring States.
So maybe it would be governed by some network of academies to organise and train scholars, and these scholars would be recommended for advancement to higher positions, and then finally a central academy. The ruler probably appoints men to the central academy, and only these men are allowed to be his successor. It might be that the successor serves as chancellor like some of historic examples of abdication. That would solve the question of "what if the ruler dies without appointing a successor."
My guess is that if we imagine an alt-Yan reunites China and Kuai goes down in history as a wise genius, the system will collapse within a few centuries. Conflicts with nomads will keep the "Yan Republic" militarised and not totally amenable to scholars. Maybe actual power ends up residing in a military dictatorship which would likely be hereditary. When the "Yan Republic" falls, it will likely be replaced by a host of barbarian states as the Sixteen Kingdoms and these barbarian states will have hereditary rule. Now there were systems like the Mongol
kurultai among these northern barbarians (Mongolic tribes were among the invaders)--it would be interesting if the system that re-emerges ends up being something akin to some Northern European monarchies in the Middle Ages where the king is elected and rules for life but has a strong council of nobles who can theoretically elect whomever they choose. Hence a council of elite scholars would help steer the government and ensure the right heir is appointed.
What kind of republic this would be? I think that most plausible would be something like oligarchic merchant republic. Some European like city state model wouldn't work at all since it is meant to small city states not Europe sized country.
No, just an oligarchic republic of scholars ruling for life and yielding the throne to the best suited as they grow old. That is the example in Chinese legend when Emperor Yao abdicated the throne to Emperor Shun, and occurred in actual practice in the Warring States period when the aforementioned King of Yan abdicated to his chancellor.