That's the textbook answer, but I don't like the textbook answer.
Although the failure of Reconstruction cannot totally be laid at the feet of Andrew Johnson, the failure of Radical Republicans in Congress to be able to carry through a real Reconstruction led to a century of a racial injustice. Too high a cost. And if we want to be less idealistic and more pragmatic, the failure also led to a century of economic stagnation for the south. Also, too high a cost.
How would removing Johnson have changed any of the above? Reconstruction "failed" not because of anything this or that President did, but because most Northerners had little interest in protecting Black rights, while Southern whites were very deeply committed to rejecting them. One side cared, while the other mostly didn't, hence the outcome. As you yourself note, this situation continued for the better part of a century after AJ has passed from the scene. There were literally generations in which nobody thought the matter worth pursuing.
Incidentally, why would the Southern economy be any less stagnant just because Black sharecroppers went on being able to vote? Both they and huge numbers of whites would still be just as dirt-poor.
And there's a scarier possibility. Allowing minority presidents in critical times, opens the door to military coup, as has happened in at least several countries. And perhaps a contributor more familiar with world history than I might be able to tell us if it's more than just a few. Way too high a cost.
Not sure what you're driving at. "Minority" Presidents have been elected four times (five if we count1824) without any hint of a military coup. Afaik, the only POTUS against whom such a coup was even slightly seriously contemplated was FDR, who had just won thumping majorities of both popular and electoral votes.
In any case, how does making impeachment easier reduce the chances of such a coup?
We would have been far better off with merely a tradition of not impeaching and removing from office except for really important reasons, and letting it stay vague as traditions often do. Hardwiring in a two-thirds requirement is one of the bum parts of our U.S. Constitution. And if a change led to more of a Parliamentary system, that's a far more acceptable cost than any of the above.
If a POTUS could be removed by simple majorities in Congress, you'd need a revolving door on the White House. Trdu=itions are respected just as long as people feel like respecting them.