As Wannis said, the claim for roman papacy is certainly not the demographical importance of the city, but the apostolic foundation of the roman church, affirmed and acknowledged since the roman synod about the Council of 381 in Constantinople : while Constantinople bishop claimed the first rank after Rome before Alexandria and Antioch, the bishop of Rome claimed (and received) the acknowledgment of his primacy due to Rome "being the first bishopric seat of Peter the apostle" and to powers given by him.
Now, it was the first mention of such, and not widely accepted on its time. If you manage to butterfly the pretension of Constantinople to rule over eastern Christianity, you may delay the roman pretension long enough to weaken it.
One of the big issues is that the apostolic foundation of a church was a powerful tool in the ecclesiastical power.
Others churches than patriarchates have claimed being founded by apostles, but they were too small in this period to have a real voice. You would certainly have to boost churches of North Italy (as Milan or Aquilea) and to weaken Roman church (probably more persecution on it) to weaken roman pretensions.
Of course, it wouldn't make the papacy moving, and could actually quite crush papacy concept if over-done, but it would give slightly less obstacle to pope moving.
Another big problem is that bishops are supposed to live within their churches. While the rule was quite ignored in later times, it was still really important then and the bishop of Rome can't do that without a real reason (destruction of Roman church entirely, something quite hard to achieve after first wave of Christianization).
Even a conquest by Lombards seems unlikely to chase pope off (and Saracenic conquest proposed by Wannis is definitly ASB without a PoD that could butterfly as well North African conquests. When they took Rome OTL, it wasn't enough to chase the pope from Rome anyway).
What you need is a general crisis of the western Church (Great Schism, French takeover of papacy, etc.), and when the crisis is over...Well, they simply turned back to the city.
A successful Byzantine takeover of the papacy, forcing the pope in exile to Constantinople (as it happened OTL) and forcing the successors to do the same is really implausible, would it be only for the rivality between Constantinople patriarch and the pope that would make the papal exile not really worthwhile.
What you ask is quite hard to do actually : crushing the legitimacy of roman maintain, the apostolic foundation, is as well crushing papacy. And once you have it, the pope began to be quite inamovible.
And remember, "Dark Ages", outside Britain of VII/VIII centuries is meaningless. It's for naming a period where, not having contemporary written sources, it's historically "dark".
Others uses of the word vanished away in the same time Charleston ceased to be trendy.