Well, you have to remember that the Empire was not constantly on the up and up-a lot of what we think today is just long-held propaganda. It had several very close calls, particularly back during the early period, when the Emperor had absolute power. It meant that one bad emperor could undo the work of five good ones (much like Rome-in-the-West). Thankfully, during the really tight spots (the Sassanid wars, the Arab conquests, the Mongol wars, etc), they had brilliant, or at least competent Emperors.
The only time I could see this being feasible is when the Europeans were trying to break the Silk Road monopoly (and succeeded in some cases). The whole Mediterranean economy went downhill, and the Empire itself was somewhat impoverished. Military cuts were inevitable. If a strong enough enemy, perhaps one of the turkic groups, had managed to unite, they could have blown through the Eastern defenses and stormed across Anatolia.
Whether they could have taken Constantinople itself, though, is another matter. Pre-explosives, the city was nigh-invincible.