Well, by making the reconstruction even more radical, you could trigger various destructive affairs: decades of occupation in the south, white guerillas, egalitarian land reform for freedmen*, black governors appointed by Washington, military dictatorship under Grant, economic stagnation, decline of immigration to the US... that would be a hell of a TL and you would wish the reconstruction was more moderate.
*This isn't bad per se, but in the context it just adds fuel to the fire.