One, it was the 1800s. Everyone was a racist. Two, he was a Democrat, which means he was even racist for the time period. Three, he helped orchestrate and personally oversaw a portion of the American Indian genocide. Four, he's Custer. He wasn't a man known for his laconic demeanor and magnanimity. He was known for being a loudmouthed, hypermasculine hothead. Custer in politics would be all about some racism.
Actually, Custer was reprimanded by, I believe, Grant for his outspoken opposition of the reservation system, if I remember correctly. In my readings of him, I've always figured him for the type of guy who sees himself mainly as a military man who shall, gallantly, due the bidding of the nation. Vain glorious, bombastic, a certainly a racist (who wasn't at that time), but I don't see him as a died-in-the-wool whitecap.