This is the problem with judging historical figures. All too often, we try to hold them to today's morals and standards without regard for the time they lived in. For example, by today's standards, Abraham Lincoln would be considered a huge racist and likely card carrying member of the KKK. By the standards of 1860, he was radically liberal on civil rights. Theodore Roosevelt too would be called a racist and white supremacist today. But in 1900 he was, at worst, moderate in his views and even invited a black man into the White House for dinner, something no President had ever done before. Advancing the cause of civil rights and racial equality is a process that moves step by step. We can't expect people who lived over 100 years ago to hold to the same moral code that we aspire to today. Even when people knew that what they were proposing didn't go far enough to advance that cause, they also knew what they could and could not do based on the attitudes of the times.
You beat me to mentioning TR's dinner with Booker T Washington. Unfortunately that set off a wave of race riots and lynchings.
I mean the act that convinced John Wilkes Booth to murder Lincoln was hearing about a speech Lincoln had given about why giving voting rights to educated blacks and black ACW veterans might possibly be a good idea.
The past was a horrible horrible place.