Even if he didn't outright hate black people, his actions show that he was contemptuous of them. Clearly he thought that their very lives were less important than his political career. He was racist, definitely.
Honestly, that kind of prejudice is the most common one in the world -- not driven by hate, but by contempt. Not every racist is a frothing-at-the-mouth noose-wielding Klan member. Not every misogynist is a physically-abusive rapist. Most homophobes and transphobes would never bash anybody. But what they do feel is contempt, and often disgust. A conviction that Those People are fundamentally of less worth than they are. Less human.
While a bad position, especially in the light of our times where all the social struggles of that time have been won (at least legally), and obviously the wrong position, I have to say this: I don't believe he hated black people, nor do I believe he had contempt for them or believed they were an inferior race. What I do believe is that he was playing politics at a time when support for Civil Rights would have cost him support, at least in his mind, and would have lost him his political office.
EDIT:
I found this quote of what Smathers said of Kennedy and Civil Rights,
“I was telling him to go slow,” Smathers remembers. “He didn’t need to be told to do it. . . I was just saying, go slow; don’t run off all your votes in the South.”
That, along with other things I have run across leads me to believe he wasn't against civil rights for blacks, but was fearful of the issue from a political standpoint of how much it could cost him in support and voters, and for that reason if there were to be civil rights, it should be more slow and gradual so as not to upset political support, and that locally he couldn't make it an issue he could support even a little (well, maybe a little, given he voted for the CRA of 1957). In that way, he was the opposite of LBJ, and would probably be so as president. LBJ thought he'd lose the south for the Democratic party for generations, but that Civil Rights was too important a cause to ignore. Smathers in that same position of fearing losing the South would, at least in my opinion, have taken that seriously as a reason to be oppositional to civil rights and slow and gradual on civil rights. And the idea of losing the south over civil rights proved right, since it hastened the Southern Democrats out of the Democratic party, and made their presidential votes go elsewhere, first to the American Independents, then to Republicans. And on a local level of Senators and Congressmen and mayors and so forth in the South, that as well turned from Democrats to Republicans, though the process was much, much more gradual (finally completed, if I recall, only with the Republican Revolution of '94). Johnson was willing to have that since Civil Rights was such an important things and a humane things. Someone like Smathers may not have agreed. On a side note, I've often wondered how things would have gone had civil rights through legislation in the 60s been slower and more gradual; a turtle in the race rather than the rabbit.