Could part of it be the llama's size, the fact that it is too small for an adult to ride it?
The llama is just the right size to use as a light to medium pack animal, but certainly too small for an adult to ride at least any real distance. The donkey or burro did well in places like Mexico, but not North of the Rio Grande in The USA.
People in The USA and in Canada tended to be more mobile and to travel longer distances. Both The US and Canada stretch from the Atlantic to the Pacific a distance of some 3,000 miles. Before the age of mechanized travel, to forge those two countries out of such a vast frontier and to settle people from coast to coast meant many of them had to ride. A pack horse can carry more than a llama, and horses and oxen are needed to pull large wagons.
I think their smaller size, the fact they are too small to ride, their load capacity and wagon pulling ability compared to horses and oxen made llamas impractical in North America.