Who are "they" though? You seem to be trying to argue that no serious American political faction in Washington gave a damn about anything other than readmitting southern states in to the Union and preventing secessionist sentiments from continuing. This just isn't the case. If it was, then nobody would have had any issue with Johnsonian Reconstruction at all, because it was simply the fastest way of reintegrating southern states. Instead, this gave the Radical Republicans a mandate to attempt to remake the southern states into something approaching a Republican vision. Free soil, free labor, free men, and the ascendancy of forces other than the planter class. This course was attempted for over a decade and even up through Grant the entire question of Reconstruction was not simply how to admit them back into the Union, but how to destroy the political forces of Redeemerism and the Democrats in general.
The debate about human rights is not us projecting modern sensibilities onto historical figures. The struggles around 'Radical Reconstruction' was literally about how to ensure freedmen/scalawag/carpetbagger ascendancy over the forces of the old South. If nobody gave a damn about human rights, why did Thaddeus Stevens feel the need to declare that he was, "[...] for negro suffrage in every rebel State. If it be just, it should not be denied; if it be necessary, it should be adopted; if it be a punishment to traitors, they deserve it." Why did they attempt to impeach Johnson over grievances about how Reconstruction was handled? Why even draft the Wade-Davis Manifesto and fight Lincoln on the issue of a lenient Reconstruction if their plans were nothing more than abolition and reintegration. Many members of Congress were deeply concerned about trying to remake the South into something new which included civil liberties for African Americans and the destruction of the former Confederate's political power. Of course, it failed but you can't argue in any way that nobody cared out it. The entire politics of Reconstruction were about these issues.