Actually Cuba was not so white in the 1850s-1860s as today, but yeah, not really different from the Deep South.
According to wiki Cuba was 57% White and and 43% black in 1861, so probably around 52-54% in 1854, more black than it is now, but still majority white.
But here we are assumed that the annexation takes place during the Reconstruction, when the influence of the South is at its nadir. And there was a considerable pro-US opinion in DR during the 19th century. The annexation treaty failed in the US Senate, not because DR opposed it.
Their was support by President Baez (who also tried to get DR annexed to France and Spain), who I'd point out was deposed multiple times.
Whether or not the majority of Dominicans supported annexation or not I don't know.
The annexation is happening because of peaceful agreement, not conquest, so this concern is ASB for 19th century American public opinion.
Those reasons were raised by American politicians IOTL.