Just move annexation to the 1854-1856 timeframe and it's doable. Sure this would greatly exacerbate the sectional crisis between the North and South, but I doubt that even with Cuba as a slave territory the South could win. Cuba would be a running sore if handled incorrectly, but it is possible in the long term to make it work. Along with the ex-Mexican south west, Could could form a very powerful political bloc. Social Conservatism and Leftist Economics (the worst of both world's IMHO) could change the shape of one of the major parties.
This would also negate the Spanish-American War while increasing American interest in Latin America. Perhaps a replacement war concerning European debt collection by force that occurred during the late 19th century? The US Navy would probably be bigger to defeat Confederate blockade runners basing out of Cuba and later to defend the island. This too would affect American interventionism worldwide. Finally, a larger American navy could provoke a British reaction and lead to transatlantic tensions or conversely the British could realize that friendship with the US was more productive than confrontation and the Great Rapprochement may happen decades earlier.