So I was wondering, why didn't Russian Empire annex the Korean peninsula? And is there any way it could do so in an ATL (preferably in the early parts of the mid 19th century)?
Extremely unlikely. In order for Russia to exert a significant amount of influence over Korea, China would need to be fragmented long beforehand, essentially requiring a PoD long before 1000. However, this alternate scenario could also theoretically give Korea more diplomatic freedom in regards to China, such as trading concessions and territory, making it much more difficult for Russia to expand into Northeast Asia much later on. It would also have been extremely unlikely for the Taiping Rebellion to break China up into numerous autonomous entities in the long run for reasons that I will state below. In addition, it took over three decades for Japan to annex Korea through deception, bribery, and coercion, not to mention that the Donghak Rebellion (1894-5), which pitted the peasants against the Korean and Japanese troops, suggested that the general population was firmly opposed against the court siding with a foreign power, while the Imo Incident (1882) and the Gapsin coup (1884) illustrated dissatisfaction from other parties as well.
The Goryeo and Chosen Dynasties had been China's most loyal vassal/tributary state (to the extent it proudly called itself "Little China"). During the late 16th century, Ming Emperor Wanli threw all his troops defending Korea against Hideyoshi's invasion (which he launched to eliminate ambitious shoguns).
Joseon, yes, given the fact that one of the main conditions for its establishment was to stabilize relations with the Ming, as Goryeo and the Ming almost went to war over Ssangseong and Liaodong due to the fact that both were simultaneously autonomous areas under the Yuan and under Goryeo's influence, making them subject to dispute when the Yuan collapsed. Goryeo, however, was much more ambiguous, as the Song failed to aid it when the latter came under attack due to geographic separation, so the two were generally limited to an unspoken military alliance. However, Goryeo essentially managed to double its territory within several centuries, briefly holding portions under Southern Manchuria, and even managed to indirectly control Liaodong, Liaoyang, and areas further east under the Yuan's influence. However, Goryeo and Joseon continued to send tribute to the Song and Ming/Qing, respectively, in order to maintain stable diplomatic relations while obtaining favorable trading concessions. On the other hand, Joseon continued to unofficially refer to the Qing as a "barbarian" dynasty, which is when the concept of a "Little China" emerged, as it considered itself to be the only remaining "civilized" state.
In addition, this is a minor nitpick, but your romanization isn't particularly consistent. The former should be Goryeo (RR) or Koryŏ (MR), while the successor should be Joseon (RR) or Chosŏn (MR), along with other unofficial variants. "Chosen," in fact, is entirely incorrect, as it is a rendering of the Japanese pronunciation, specifically when the peninsula was under Japanese rule, suggesting that Joseon was a Japanese colony long before 1910, which is a strange assumption, although it's obviously not what you intended.
The 1885 Tientsin Convention turned Korea into a joint Sino-Japanese protectorate after a failed Japanese-backed coup. The First Sino-Japanese War ended with Korea as a fully independent state. And of course the Russo-Japanese War ended with Korea as a Japanese protectorate.
Have the Qing Dynasty collapse during the Taiping Rebellion, with the Taipings unable to establish control over all of China, and China enters a prolonged Warlord Period. *Then* a succession of ambitious and clever Tsars who dominate northern China *might* do the trick.
The First Sino-Japanese War only made Korea independent in name by severing ties with China, as Japanese influence had continued to grow for more than a decade beforehand.
It would be virtually impossible for the Taiping Rebellion to succeed in the long term, partly due to the fact that it was directed by ethnic minorities, not to mention that European powers eventually intervened in order to ensure that the Qing did not collapse. The Qing continued to exist until 1912 partly due to the fact that the Europeans wanted to influence China under a consolidated government, which would result in favorable economic concessions, so division within China would have been unthinkable, as the colonial powers would have struggled to contain such a large hostile population consistently at war with each other. This was also why the Opium War largely left China politically intact.
That was later though. I'm talking about the 19th century here. Also, I don't recall hearing/reading about any annexation attempts of Korea by Russia. Pacific warm-water port(s).
Also, just read on wiki that "In January 1866, Russian ships appeared on the east coast of Korea demanding trading and residency rights in what seemed an echo of the demands made on China by other western powers". I wonder if this could have lead to a Russian-dominated Korea if someone else was leader of Korea other than the young King Gojong...
Distance was the biggest factor, although Russia also had its hands full with major conflicts within Europe, while Korea only saw Russia as a counterbalance to China and Japan, so it would have been difficult to establish closer relations without antagonizing other powers. It's also extremely unlikely that a more reform-minded monarch could have taken power, as the Heungseon Daewongun was essentially the only royal family member who could have stood up the Andong Kims, and both firmly opposed opening up to foreigners.
As said before, you need a larger border, and to do that you need to annex Manchuria first.
Bit hard to accomplish when the only options for deploying troops are to sled through Siberia or sail across half the world. Or to literally breed soldiers there (which they did). Britain beat China bloody and only took some economic concessions.
This too.
Let's concede that Russia cannot annex all of Korea mid 19th Century. But Count Muraviev is a busy diplomat and all that Russia really needs is Northeast Korea up to Hungnam Bay. So that's what Rissia does. Annex Northeast Korea (the mouthpiece on the Korean Horn), even leaving the Tumen River as an international waterway with China or foreign nations able to develop a port for China on the Tumen above it's 1.8 mile frontage with Russia..
This ignores the fact that Northeast Korea was (and is) one of the most sparsely inhabited and inhospitable regions within the peninsula, making it much more reasonable to supply resources from Vladivostok. In addition, Russia would have to firmly convince Korea that it would not be undermined by further negotiations, which would take years, if not decades, to resolve, given that it took 34 years for Japan to do the same, at which point other countries could attempt to directly counter the Russians' influence by that time.
I can see the Korean Empire as a Cuba-analogue; a protectorate under Russian economic domination with a number of military bases under Russian jurisdiction.
This too, given that Russia would be unable to directly control Korea, although this would require numerous butterflies affecting China and Japan long beforehand.
It is quite absurd to believe Russia would really care about Korea when it has bigger problems closer at hand. Russia's foreign policy was always to solve problems closer first (i.e. Great Game, Pan-Slavism, Polish independence) and then solve problems farther (i.e. Manchuria, Alaska, Korea). Russia's not going to care for warm water ports in the Far East when it has the Brits going for its ass. Russia would, for example, choose full domination of Persia over any sort of annexation of a small country in the Far East.
This is true as well.
They cared enough OTL to risk (and lose) their entire battlefleet OTL...
Yes, but this only occurred in 1905, despite the fact that Russia could have directly intervened in Korea since the Treaty of Aigun (1858), when the Qing ceded Outer Manchuria to the Russians. For comparison, in 1863, Finland was given greater autonomy, while Poland revolted for the second time, and Russia participated in the Caucasian War (1817-64), Crimean War (1953-6) and the Russo-Turkish War (1877-8), while the Great Game lasted from 1813-1907, not to mention internal political issues, such as Alexander II's assassination (1881). Although conflicts died down after 1878 or so, Russia continued to focus on European affairs, including the League of Three Emperors (1873-87) and the Reinsurance Treaty (1887-90), partly due to the fact that the Balkans were a hotbed of conflict due to the Ottoman empire's disintegration. In any case, the fact that Russia limited itself to trading concessions in Northeast Asia, despite the fact that it had a growing sphere of influence in Mongolia and North China, suggests that it was unwilling to risk war with either China or Japan before 1900 or so, and was certainly in a better position to do so after China was removed from the equation in 1895 after its defeat to Japan.
In other words, given the fact that the Russians were concerned with other major issues in the west, while continuously suppling troops in the east could have turned into a nightmare if it had attempted to do so before 1900, Russia would most likely have been unwilling to overextend its capabilities in order to carry out such a risky endeavor.