The subsequent history of Shiism kinda shows that even when there are more or less clearly appointed successors, splits happen.
In terms of sacred history, Muhammad is the last prophet, so no one could be his "successor" in that role, a role that was not his to give to begin with anyway. Prophecy is not something that can be passed down. Had he had male direct heirs, it is likely that the leadership of the Islamic Umma would have gone to them, but even this was likely to be challenged at some point.
Now, there's indication that early Caliphs had a much larger religious role than subsequent Sunni views would have been comfortable with, and certainly they were regarded as able to appoint successors, even though consultation with prominent people in the community was the way to go.
In general, even if Muhammad had clearly indicated a successor (which the Shiites argue he did), disputes would have occurred down the line. Even IOTL, it seems that the would-be Ali supporters were initially more or less fine with Abu Bakr and Umar anyway, but things went downhill under Uthman. The legitimacy of Abu Bakr was contested IOTL, but not primarily by Ali supporters. A Caliph supported by a generally recognized indication by Muhammad would have stronger legitimacy but would still face some trouble asserting power since he would not be a Prophet. Sooner or later along the succession line, different views of what the Caliphate is would coalesce around different claimants along lines probably not very different from OTL as the secular and religious tasks of the Caliph diverge.