Part of me think that a victorious CSA would not be using any government buildings of the former United States, and instead, move their capital over to Chicago, to give themselves a clear break from the old regime. While transforming Washington DC's
(Which they will most likely rename into something else) government buildings such as the White House and the Capital into museums, sort of like in the amazing
'People's Socialist Atlas' TL:
... Assuming the CSA, especially if they go Totalist/Vanguardist, don't go full 'Year Zero' on the place, denouncing the former United States as a bourgeois republic founded by racist rich white landowners using slavery on stolen native lands, hence, irredeemably evil and needs to be erased from history, as all of their monuments and symbols are inherently counterrevolutionary and oppressive against the working class.
Ending with CSA militiaman dynamiting the Capital Building while burning the Declaration of Independence and the original copy of the US Constitution in a highly choreographed ceremony in front of a film crew invited over from the Commune of France, complete with a children's choir singing the Internationale, the same way that the COF will do to the Brandenburg Gate if they capture Berlin during the Second WK.