If Louis couldn't have married Elizabeth he would have married someone else (it was expected from kings) so there is a possibility he would have had heirs after all.
But let's assume he didn't or he was childless. If Louis of Hungary dies before Casimir III, it is possible Casimir IV is never adopted by the Polish king, since it was most probably done to keep Louis from Polish throne - however, Casimir III's last will was annulled and Louis was crowned king of Poland. Anyway, Casimir III is too good a politician to leave such a question unsolved, so he would have some solution - maybe ha adopts Casimir IV after all, maybe he makes a deal with Siemowit III or something else. A lot depends of the way Hungarian succession is solved. Or perhaps he looks for a Silesian Piast? Or some other Pomeranian pronce?
If Casimir IV takes the throne, weel, a lot depend of the way he rules and how long he lives. But if he is indeed childless, Poland will look for a new king. But why does it have to be Sigisimund I?