Why didn't it kick off? Was it just because 'Cotton Was King'? Or perhaps it was because the main importer of tea at the time, who I assume was Britain, got their tea from India and China?
Sugar, cotton, and tobacco were much more lucrative. Plus the only ones really buying tea were China and their buddies, who had their own sources. Not to mention, it's not cost-effective enough to hire the laborers cultivating tea requires.
Why Britain drinks tea is due to a devastating pest ruining their coffee supplies in the late 1860s, leaving Brazil/South America with a monopoly. The US had abolished slavery for atleast a year at that point. The places Britain had colonized were already growing tea on a local level, Great Britain just ramped up the production to suit their needs.
So unless it is established early in colonial history, and sticks with the Americans, then there would be no need to cultivate it for the only ones buying it have their own supply.
Possible POD for tea cultivation in the US South/ Caribbean;
Coffee rust is more successful in it's global spread, choking out many of the last holdouts in Brazil. A worse Reconstruction and an earlier introduction of the boll-weevil leave the US south deprived of a source of revenue. A particularly smart entrepreneur (or maybe just a crazy one), returns to the US with tea plants to grow as an alternative to cotton. As a bargaining chip with the Japanese and British, tea quickly becomes a staple for wealthy Southern land-owners who can blow enough money to compensate for it's high cultivation costs.
