It's a little more complicated than it seems.
One of the reasons that the Jin was able to conquer the Liao was because it negotiated with Goryeo first. Although Goryeo was able to retain its northwest border at the Amrok/Yalu river because it repelled initial invasions, then convinced the Liao, it found itself on the other side when they confronted the Jurchen.
After building a wall in order to prevent further invasions, Goryeo initially managed to seize territory north of the wall and set up nine fortresses in Jurchen territory. However, Goryeo aristocrats argued that they would be costly to maintain, so the Korean ruler offered the Jurchens an option to kneel and pray if they wanted their territory back. A significant amount did this, causing the recently conquered territory to be reverted. Only eight years later, they conquered the Liao, pushed the Song south of the Huai river, and founded the Jin dynasty.
Had Goryeo been more adamant in retaining its fortresses, it might have later found a reason to push north of the Amrok/Yalu and Duman/Tumen rivers, and eventually claim a significant portion of Manchuria. What happens after is open to speculation, because Goryeo, instead of the Jin, could have overthrown the Liao in optimal circumstances, which could have led to tense relations with the Song. Meanwhile, any POD after the Jin took over North China would be problematic, as the Jin rulers viewed themselves to be direct descendants of a Silla or Goryeo person who fled because of political strife. This meant that the Jin would have been less willing to invade Goryeo, and Goryeo would only have taken defensive actions as it did with the Liao. Some records do suggest that Goryeo probably had territory and a few fortresses in southern Manchuria, but it was most likely temporary and was in response to or to prevent an invasion.
On the other hand, the Liao was able to maintain a border with Goryeo after it conquered Balhae. Had Goryeo managed to convince the Liao that the former Balhae territory should be theirs by emphasizing cultural and linguistic relationships, they might have taken over a significant amount of southern Manchuria, but it would be hard to say how much, because Goryeo might not be able to pacify the population, and the Khitan would not have been happy with losing so much territory. Another possibility is for Balhae to repulse Khitan invasions, although because we have very limited information about the former state, it is hard to assume what they could have done in order to stay independent until it negotiated with Goryeo to form an alliance.
I don't see how Goryeo would have been able to take Liao territory: historically, the Liao probably suffered more setbacks from Goryeo than Song China, and Goryeo still agreed to become a Liao vassal after the Liao invasions were defeated. The Liao would probably be too strong.
I'm pretty sure the Liao conquered Bohai at the peak of Khitan strength, so that would also make it difficult for Bohai to survive.
EDIT: I've never heard that part about Goryeo and the Jin during the Liao conquest. Do you have a source? I'm pretty sure the reason the Jin didn't ever invade Goryeo was actually because Goryeo was a lot quicker to capitulate than they did with the Liao. I'm pretty sure this is also the reason Goryeo never initiated hostilities against the Jin, because they remembered how bad things got when the Khitans invaded.
Last edited: