54% of the "free men of color" resided within the putative Confederacy in 1860.
No, 54% of the "free men of color" lived in the states where slavery was legal. 28% of the "free men of color" lived in the 11 states of the Confederacy; 72% lived in states that remained in the Union.
You're also ignoring that 6% of the blacks in slaveholding states were free, while 99.9% of the blacks in the non-slaveholding states were free.