There were plenty of cases where Reconstruction failed due to a lack of political will, boldness or vision. Giving the Southern states full rights so quickly, failing to aid the freedmen, giving out free land or withdrawing the army. These are obvious failures. Failing to deport thousands of Confederates is not one of them. It is an unrealistic option. The ideological concept isn't really there at the time. The great population transfers are awaiting the 20th century before they really get underway, and the Confederates were considered Americans. Also, logistically, the entire idea is just dead. The sheer cost of moving so many people would be outrageous, the effort gigantic.
It isn't happening, it just wasn't on the table unless the war was very, very different.