The reason that the Roman Catholic church largely adhered less to scientific racism was because it saw itself as a universal church, above nations and kings. The fact that church doctrine recognised non-whites as having souls made the Catholic church challenge anti-miscegenation laws in the United States, arguing that these laws violated the religious liberty of Catholics. One of the primary goals of the church was to evangelise and seek out converts, all over the world, establishing a Catholic (universal) church under the Pope.
Protestant churches being independent could adapt and create their own theologies to the whim of preachers or parishioners, often choosing to focus or ignore parts of the bible to the suit their own ideologies. In the United States for instance, this led to various new sects and branches being invented, often able to justify or ignore institutionalised racism. During the 19th century the Baptist churches in the south began to provide biblical evidence supporting slavery, in contravention to Methodist and Quaker teachings, making the church popular in the region. Because the Baptist Church lacks a hierarchy, it was able to split into different factions, with the Southern Baptist Convention being founded in 1845 and with African-Americans forming their own sects after the Civil War. The white dominated SBC opposed desegregation and anti-miscegenation laws during the 1950s and 1960s.
In South Africa, the Dutch settlers adopted the Calvinist mindset based on a theology of predestination, that unlike in Europe, remained largely unaltered by the Enlightenment. This was a fatalistic viewpoint, whereby salvation cannot actually be guaranteed, leading people the Afrikaners to equate success with being in God's esteem, and thus included in His foreordained elect few. Whereas Calvinists in Europe seemed to base their evidence of being in God's favour on economic success, in South Africa the Boers based this on being successful in battles against the non-white population. The following quote does seem to illustrate the mindset that came to permeate the Dutch Reformed Church in South Africa:
"The Afrikaners saw in their own lives reflected in the Chronicles and Exodus of the Old Testament and, like the Hebrew tribes, came to feel that theirs was a special destiny. Like the ancient Israelites, the Afrikaners were patriarchal and semi-nomadic pastoralists, wandering in a harsh environment, and they too developed a sense of mission as representatives of the true Faith in confrontation with hostile disbelievers. Because of this Biblical identification, the Old Testament became a virtual manual of behavior as the Afrikaners moved increasingly away from the theological guidance of the organized church."
It is not difficult to see how being a member of such an exclusive religion rather than a more inclusive one would lead its adherents to believe that they are not only superior to other groups and that therefore keep interactions with them to a minimum.