I think it's the other things regarding it that made it so harsh.
If those "other things" are things like black suffrage and expansion of civil rights, then I've got to disagree with you. It is true that in 1863, Lincoln butted heads with Congress on the question of Reconstruction, with the President's proposal being fairly mild (10% solution, guiding incoming states to provide vote for literate blacks and Union vets, etc). However, by early 1865, they were working through their differences, with a compromise plan going forward, wherein those states which had begun Reconstruction would be readmitted, while those yet to begin would do so along the lines of the Wade-Davis Proposal; in addition, I see Lincoln and the Radicals playing a sort of "good cop, bad cop" routine with the Southern states in getting them to, at minimum, offer enlarged expansion along Lincoln's proposed lines. And nothing like what Andrew Johnson countenanced OTL (black codes imitating slavery, etc) would pass by a surviving Lincoln.
So why did I think an earlier war could mean a milder Reconstruction? Well, remember that Lincoln's views during all this were evolving, bringing him closer to the Radicals with each passing year -- at the start of his presidency, Lincoln had no intention of touching the institution of slavery where it already existed, and by the days before his death, he was not only waging a war of liberation and pushing an amendment abolishing slavery in the nation, but was openly talking about the need for at least partial expansion of the franchise to the formerly enslaved, and was open to even more in certain cases. End the war before the election, though, and now this evolution might be curbed in late 64/early 65, which will have effects in the months following the war however you look at it.
But that's just a thought I had in reading a previous post; could be I'm completely overcorrecting, and that Lincoln's political evolution would not be imperiled by an earlier peace, especially if it was so visibly won by USCT Troops. So I think I'm back to my original position here.