During Dutch rule of the Cape, there were just over 2,000 European immigrants who arrived in the colony during the entire period of Dutch rule 1652-1806, with Dutch and Germans being the largest groups. As the Germans were overwhelmingly from regions neighbouring to the Netherlands they spoke nearly identical dialects to Dutch. Just under 10% were French Huguenots, and around 3% were Scandinavians, mainly Swedes and most immigration occurred in the 17th century, so that by the time that Britain formally took over the colony, some 99% of the 27,000 whites were born in the colony.
The reason for this seems to be two-fold, firstly the VOC was a profit-driven enterprise based on trade. The Cape was a necessary strategic base and nothing else. The reason for its colonisation was to make it self-sustaining. The colonists were treated a necessary to make the base self-sustaining, but often as a nuisance. As a result, the Dutch did not provide incentives or subsidies for settlers and barred them from competing with VOC monopolies. This was not unlike the Dutch colonies elsewhere, where profits and returns to share-holders limited the scope of settler colonialism.
In 1698 there were calls to open up the Cape to poor Germans, but only around 50 were settled, this demonstrates the VOC's lack of interest. Had the VOC been more encouraging, I imagine that Dutch would still be the dominant language, as it would have been the language of the administration.