Who is to say that he would not have been assassinated later in his second term.
If he serves a full second term:
He would of had a fight with the Radical Republicans on reconstruction of the South. Such a fight would have dimmed his legacy some. Would he prevail in his more moderate view??????? Reconstruction would more likely be in the middle from his moderate view to the harsher view of the Radical Republicans.
After his second term, Mary and him would take a tour of Europe.
He would come back and resume his law practice and disengage from politics.