The Teutonic Knights could, and did, do some damage to Lithuania, but they didn't have the resources to push all that deep into central Europe from the Baltic.
The real, long-term military threat to Lithuania was from the East. Moscow was not strong enough to bring Lithuania down at the time it converted to Christianity in OTL (early 1400s), but it grew more powerful thereafter. If Lithuania was still pagan by the time of Ivan the Terrible, it's highly likely that he would try to conquer it. If he failed, the threat would renew under Peter the Great. You can butterfly both rulers away, but Lithuania would be an obstacle to Russian ambitions by the 1600s, and Russia had plenty of resources.
The intellectual threat is that the polytheistic Lithuanian religion was looking increasingly anachronistic as time went on. At some point, maybe the Enlightenment, even if the state had withstood all foreign attack, the religion just wouldn't have seemed credible any more.