The Nicaeans were most legitimate due solely to their championing of and respect for the traditions and culture of the Orthodox church, their success in arms against the Latins (and to a lesser degree, the Muslims in Asia Minor), their support and maintenance of the Patriarchate, and very obviously, their eventual success in recapturing Constantinople.
In my opinion, by this metric the Ottomans had far greater legitimacy and claim over the Byzantine legacy than any 'Latin' state could have by their respect and maintenance of Byzantine customs and religious practices, whereas Latin Christians trampled over Byzantine customs in a very (dare I say) barbarian manner.