Get what rolling though?
Congress did not embrace Freedman suffrage from choice. It was forced on them by Andrew Johnson's complaisance toward Southern Black Codes, reluctance to disavow the Confederate debt, and hearing damage suits against Union soldiers for wartime activities. Take away Johnson in favour of a POTUS who takes a tougher line about these matters, and they'll probably be satisfied with giving the vote to literate Blacks and ones who had served in the Union Army - enough to irritate the South but not enough to threaten white control. So you end up with the Blacks getting less rather than more.
I think you are correct.