I guess it can easily happen if Stalin decides on a whim that "pleasing Poland" should be an even higher priority. Kaliningrad was useful as a warm-water port to the USSR, but it was not essential, as the Baltic republics already had several of those and I doubt Stalin accounted for the hypothetical scenario of the USSR dissolving and thus losing access to these ports.