Geopolitical succession is entirely based on international recognition at the time of transition. Mehmed held Roman lands, ruled over Roman people, and most importantly publicly claimed succession to the title of the Roman Emperor. He and his successors were acknowledged as such too by their contemporaries.
This issue of culture and religious descent has unnecessarily been conflated with legal succession (probably by Western European scholars wishing to promote their own claim to the roman legacy).
The legal transition is simple-
Upon receiving the Western imperial Regalia, the Eastern emperor succeeded to the office and the resultant responsibilities of the unified Roman imperium. The fact that the emperor no longer exerts influence over the western half means little to his right to this office; he is the emperor over the whole empire however limited its territory may be. To look at it from another angle, whatever the eastern empire claims/holds is the whole Roman Empire.
Cut to 1453, where we see this office abolished, its administrative apparatus absorbed and its title, rights and claims assumed by a new office. These claims and rights are acknowledged by the international community, and so are valid legally, as is evidenced by the use of the title Caeser of Rome on binding international treaties.
Also, the fact that the later sultans didn't really press this doesn't matter. For example, Russia doesn't need to keep saying it's the successor of the USSR to hold its Security council seat. It only needed to do so at the time of transition.
Legal succession, in general, is a matter of international and internal recognition. The factors that this recognition is contingent upon can differ from case to case, but the supremacy of the recognition itself as the decisive criteria remains paramount.
Cultural-religious descent can be debated of course, but regardless of who claims it to what degree, that descent would be from the Roman people, not the Roman state.