Indeed, and paradoxically, a harder Reconstruction might have resulted in Blacks doing worse.
Under a tougher postwar regime, the Southern states might not have dared to reject the 14th Amendment. If they ratify it, they could well get readmitted without giving Freedmen the vote.
I have often thought that white politics would have been very different if the South (theoretically), the Midwest and the West (theoretically again until in some cases
later than in the South, especially re Native Americans in the Southwest) had not have to extend the vote to non-whites. If the South (and West) had not had to give the vote in theory to non-white populations, they might have been less obsessed with controlling them because their landowning and mining interests would have not had to fear their political power. In the Plains and Southwest, large mineowners were able to exclude Native Americans – whose interests in land control were irreconcilable with their own – from voting by law even
after the Indian Citizenship Act of 1924. In fact, in Arizona almost no Native Americans voted in any elections before
1972. The effect of this was that those white voters nearest to large Native concentrations were able to simply acquiesce to the demands of the mining capitalists and vote on issues
other than Native American political and civil rights, to an extent which Black Belt whites never could do.
Under a scenario where the Confederate States were readmitted (and Western states continued to be admitted) without giving non-whites the right to vote, the persistent fear of black political power might have been absent and the sectional politics of the “System of 1896” with its massive downturns in white political participation (not to neglect larger declines in black voting even in loyal states), might have been avoided.
The only problem is how would the United States have presented a system that forbids non-whites from voting (except perhaps in the Northeast) to the rest of the world as its power grew?
Or what would the response of non-whites have been if their economic fortunes declined or poor whites found that policies intended to help (and actually helping) them also inadvertently helped non-whites and wanted to stop them for this reason?? I can imagine much more violent Civil Rights struggles, unless blacks could be more tightly united by churches.