The nine fortresses established in 1108 after the conflict with the Jurchen were located along the northeastern coast of the peninsula, along with a few in southeast Manchuria. The expansion into Liaodong occurred around 1360-70, and although a second attempt occurred in 1388, the general who was in charge decided to turn back, and established a new dynasty, which caused the claims to be put on hold. The temporary expansions were eventually pushed back due to political strife within Goryeo, and while I certainly understand that they were limited in scope, different circumstances could potentially lead to different results.
The rapid decline of the Mongol Empire allowed for such an expansion, said factors wouldn't exist in this scenario if the Liao Dynasty remains strong and stable in the north. Its very existence most likely butterflies away the Mongols, and if that doesn't the precondition of it remaining most certainly deflects most of the damage.
In terms of the size, just taking the coastal regions of Liaodong doesn't seem to be that much a stretch, and at most, the smaller peninsula only seems to be 1/5 or so of the larger one. In comparison, you have to realize that from about 1258-1356, the northern border was initially located somewhere between what is now Pyongyang and Seoul, as the Mongols temporarily occupied the northern regions. However, after Gongmin came to power, he ordered his generals to push the border further north and expel the Mongols, effectively doubling the size of Goryeo after taking over Liaodong and the areas around the Tumen River. Although Liaodong was lost after 1380, the fact that Goryeo managed to retain the northeastern borders during the chaos, with incidents such as the Red Turban Rebellion, which actually temporarily seized Gaegyong, the capital, suggests that with adequate preparations, it would be possible for the state to expand further into Manchuria and Liaodong with different conditions.
The Tyumen River was basically the border of Korea, and this was a rapid reclamation of older territories in the wake of the Mongols. Liaodong was only held as long as they could keep it before the Chinese marched in to take it.
While it does suggest the possibility of further expansion one has to take into consideration the sheer mess the Mongols made of things in their decline, basically giving birth to the entire geo-political situation in Asia. Assuming a strong Liao Dynasty and a butterflied Mongol period we just don't see this kind of scenario.
See above. The northern areas of the Korean peninsula and the southern parts of Manchuria were generally sparsely populated, and if Goryeo thinks that there will be a security threat if the borders were not extended further north, as subjugating the hostile tribes could quell the constant raids, it's possible that they could do so. "Unified" Silla's northern borders were around Pyongyang, but soon after Goryeo reunified the peninsula, it pushed it further north, until it eventually reached the Yalu by 1000. Although the border shifted both north and south for about three centuries, and was temporarily pushed south of Pyongyang under Mongol influence, the border was soon moved to the Yalu and Tumen Rivers around 1370-80, and was finally consolidated after Sejong sent some generals to the north in 1433.
The Yalu and Tumen rivers marks the boundary of Korean expansionism after the fall of Gorguryeo and Balhae, saying the border was around there doesn't convince me (and probably wouldn't convince anyone else) that this meant they were likely to expand.
That, and I think you are sidestepping the issue of a collective military farming colonization being an impractical belief and expectation.
Again, see above for what occurred during Gongmin's rule. The Red Turban Rebellion, which devastated Goryeo, certainly did not deter it from pushing the border north to the vicinity of the Tumen River, and there is no reason to assume that Goryeo would refuse to push the border north if the Liao started disintegrating. Also, see below:
Three things worth nothing:
First off, for the rivers, here's where they are for someone who is unversed in the geography of the region:
Yalu River
Tumen River
Basically, those rivers establish the northern frontier of Korea. Not only that, but they are useful borders being good enough to ward off most minor nomadic threats.
Second, your main argument that I have gleamed is a continual reference towards how the Goryeo expanded in the wake of the collapse of the Mongol Empire, or rather the Yuan, which is something that's not analogous to anything that could come from the wake of the Liao Dynasty coming to a violent end. In particular, much of that expansion was reclamation as well as expansion into areas glad to be free from the Mongols. For an apparent analogy, the vast death and starvation that was occurring behind Soviet Lines that devastated the country didn't stop them from marching all the way to Berlin.
Third, this is based on the assertion that the Liao will collapse and Manchuria will be in a volatile enough situation to allow for the Goryeo to expand into it provided they prepared for it in advance. (as well as seeing a threat that helps catapult this decision) This one has some problems, notably that the Liao won't be collapsing and there's little evidence to support that Manchuria will collapse if the Liao doesn't.
---
As for the Balhae revolts, which I would more call heavily Korean influenced natives more so than actual Koreans since modern revisionism in this regard has turned me off to much of it, I have to beg the question: Why didn't Goryeo help in OTL? If they did what prevented them from being successful? You yourself claimed that this was a lightly populated region that were filled with the decendents of an older Korean state that revolted against their rulers, and that since Goryeo considered itself the successor to Goguryeo that they were looking to reclaim Goguryeo lands, so it would seem a shoe-in for a successful expansion. Something that didn't happen as far as I'm aware. Because of this I have to know more before I can make a judgment call because it seems as though Manchurian Expansion was handed to Korea on a silver platter, one they promptly refused.
Which brings up that cultural resistance to expansion that I brought up, but for now I just want to know why something didn't happen when all the factors presented makes it seem more likely to happen and be successful than over 90% of military campaigns.