Talk of the Ottoman ascendancy necessarily involves the Byzantine death-spiral, so this is good discussion. However, I'll ward against fatalist thinking with regard to either 'empire'. Neither was 'doomed' to succeed or fail.
Byzantium post-1204 was
not nearly as mortally wounded as a superficial understanding of their history often conveys. The lands of the imposter [Latin] Empire were retaken by the Palaiologid Nicaean Empire, Constantinople restored as capital, and the other despotates subdued and, if not re-conquered outright, at least made subject to the Emperor in Constantinople. For a while---indeed, until the rise of the house of Osman---it seemed as though the 'Byzantines' would regain much of their former glory and reconstitute into a modern feudal state.
I'm not sure what the conversion of Osman I to either Christianity or a different Muslim sect would do to the Byzantine decline. Probably, not much. But that, I think, is where the conversation should focus---because the main effect of the rise of the 'Ottomans' was the downfall and supplanting of the defunct Empire with their own. The region (Balkans + Anatolia) would have had a dominant 'empire' either way. The geography and demographics of the time created a gravity well toward which the various polities spiraled; a race to the bottom, so to speak, toward Empire.
I'm interested to hear what suggestions others have as to which existing polity at the time could have filled the role played historically by the house of Osman, if the latter is taken out of the equation. Would the Palaiologids have ever gotten their shit completely together, as they started to in the 1260s? Was the existence of a strong, self-sustaining Christian Empire in this region a nonstarter after the 13th century? As a follow-up, was a Muslim Empire in the region an inevitability?