While still debated (giving we're talking about proto-historic peoples, it's only fair, especially giving the lack of sources), its seems that Bronze Age cultures of the region are part of the Corded Ware (largely accepted as at least largely IEzed) horizon or legacy such as Baltic Battle Axe or Middle Dniepr, even if the Hunger-Gatherer resurgence may have been important in the former which may have lead to a more porous contact with non-IE peoples in northern Baltic (especially Proto-Finnish peoples that gave birth later to Livonians, Estonians, etc.) which we know have shared some features with their southern neighbours.
From there, there is little interruption for what matter obvious cultural changes.
The names that Ptolemy gives to Baltic peoples are to be taken cautiously, but seem to have Proto-Baltic-Slavic roots.
So I'm not sure about your affirmation about them never having existed : is it a matter of classification, do you think Baltic and Slavic groups while related never formed any closer ensemble, or else?
Now, I think we can agree that Balts or Slavs as such didn't existed yet at this point : hence why I said "Proto-Balto-Slav" in order to point a period where they were still largely undifferenciated in my opinion (I hypotheised, in another thread, that Sarmatian presence may be one of the factors of said differenciation).
But I digress.
The coasts of the Eastern Baltic weren't super undeveloped, though. They were still the starting point of the Amber Road, and archeological evidence points to the region being at least moderately wealthy in comparison to it's surroundings.
I mostly agree : my point wasn't to say the region waspiss poor, but
structurally undeveloped to Romans, even compared to Germania Magna.
I'm not specialist on the trade network of Ancient Baltic tough : but my understanding was that this trade was essentially made by land, which mean avoiding most of Baltic region itself safe for Vistula mouth.
As you asked for a Roman presence in the land of Aesti (which could admittedly means anyone living along the Baltic as much Tacitus was concerned) and especially among peoples that would eventually give birth to Lithuanians, I assumed (maybe wrongly) you didn't meant a region with a more or less important Germanic presence. That said, I gladly concede you may have been considering Proto-Prusians peoples as well.
Still, you don't build a nation out of quasi-nothing : even Greek colonists in western Mediterranean reached shores were urban society existed (if a really early and simplified way) on which their trade and political network could develop (and never hugely successfully so).
These Romans could probably pull out an emporia thanks to Amber Road (and possibly salt too), but would be too far from Rome and Mediterranea to hope for economical/cultural/political reinforcement in face of a more numerous people who, as you said justly, were fairly wealthy and with the resources to prevent an handful of isoled newcomers to take the lead.
Which, along with early contact with the Romans, was the reason why the Western Balts split off and formed separate tribes earlier than the East.
It's a possibility, but I don't think it was the main factor, mostly because you have too few (if existing) exemples and clues to Roman influence in the region contrary to other parts of the trade road : remember this was a trade by proxy, meaning the main direct influence Proto-Balto-Slavs recieved at this point may have been Germanic (Celtized then Romanized to an extend, I'd agree).
I'd tend to think that the climatic then social changes of the IIIrd century did more directly for that (as Sarmatian presence may have dealth with the growing differenciation between Slavs and Balts, IMO).
I don't think that the existing, but quite weak, Roman influence along the Baltic would have been nearly enough to be a support for these exiled Romans. At very best, I'd go with something similar to Hiberno-Normans emporioi in Ireland, going native in a matter of generations.