If I had a guess, the bantu peoples were hit nastily by smallpox-just as people in Kenya were (or that might have been measles...point is, not a lot of immunity).
However, the fact that their land was not good for European crops gave them a respite from the European invasion for a little time, which allowed them to recover.
In addition, IIRC early modern southern Africa got somewhat cut off from Indian Ocean trade routes due to the disruption of the Portuguese and others-which would mean less slave trading-but still had maize introduced by Europeans, allowing for a boom in population growth which would not be as badly counteracted by the slave trade as happened in west Africa.
In addition, the Khoisan were at least partially dependent on gathered wild plants. Environmental disruption caused by European settlement such as the introduction of invasive insects and plants could disrupt their ability to gather food, and leave them in some ways more vulnerable food wise than the bantu people who could create large stocks of stored food through agriculture for the bad times, be those bad times disease, drought, or war.
EDIT: There is no magical smallpox resistance gene people. Any group of people-Native American, black European, or Asian who are isolated from this disease will die in very large numbers when re-exposed.