I've always had the impression that Carson's base audience in the early 90s were basically the same people who had started watching him in the early 60s, IOW if you were a 25 year old watching him in 1962, you were a 55 year old watching him in 1992.
As such, I think Letterman's style, which was pretty clearly aimed at the more ironic sensibilities of late boomers and early Gen X, would have alienated a lot of the people who were already watching the Tonight Show.
(Caveat, I'm basing this mostly on personal impressions, plus the opinions of my father, who was born in the mid-30s and REALLY didn't get Letterman. He was quite happy with NBC's decision to go with Leno.)