Actually, this affects the entire population, we can see people from all races in all situations, there is even a 1932 newsreel (I know it is outdated) in which the american narrator comments how different from america, race doesn't mean social class in Brazil
By the standards of 1932, Brazil was indeed much more racially liberal than the United States, South Africa or many of the European colonies elsewhere. There was no formal segregation with signs designating areas for whites only, and the appearance of racially mixed social settings was a stark contrast to the United States. The legacy of slavery however did leave Brazilians of African origin economically and socially disadvantaged, particularly as they were often concentrated in the least developed parts of the country. Like much of Latin America, however the elites have tended to be overwhelmingly of European or mostly-European origin. As a result, what racial discrimination that did exist was often pointed to as based on class or caste, rather than race. The past government sought to address this imbalance, but due to the economic situation, Brazil has reverted to a situation that as predominated for much of its history with the current president and all of the ministers are what would be classified as "white". Only 2 out of 81 (2.5%) Senators are black or mixed and only 43 out of 513 (8.4%) deputies in Congress are of African ancestry. When one considers that just over 50% of the population is black or of mixed race, this shows that politically people of African ancestry are overwhelmingly underrepresented, having major effects on there position in society. Historically, Nilo Procópio Peçanha who served as Vice-President between 1906-1909 and briefly as President between 1909-1910 was often called a mulatto, and was ridiculed in the press of the day, but he denied having non-white ancestry.
The U.S. for all of its failings in the past seems to have at least attempted to address racism and its legacy head on. By a president whom would be considered "pardo" or mixed in Brazil, along with 42 (9.7%) out of 435 voting members of Congress identifying African-Americans, along with 2 non-voting members, it has become much more inclusive in government than say Brazil. In Brazil, the Senate a mere 2 (2%) out of 100 incumbent Senators are African-Americans, and though still under-represented African Americans were 12.6% of the U.S. population according to the 2010 census, whereas 50.74% of Brazilians identified as black or mixed in the 2010 census.
This problem is not wholly Brazilian, as one sees it in countries like Mexico, Peru, etc. where persons of European origin are smaller minorities than Brazil and perhaps even more overrepresented in politics, the economy and the media. However, when one simply claims that they are not racist because the lack of legal racism and the ability of a tiny number of non-whites to reach to the top of the pyramid, this often makes achieving any sort of equality difficult. Part of the legacy of Brazil's attitude towards race definitely stems from colonial times, and even in Portugal's African colonies the government often pointed to a tiny handful of Africans and mixed-race individuals in prominent positions along with the open attitudes towards mixed-race marriages as proof that they was no racism in the Portuguese Empire. For that reason, nothing was needed to be done to address it, rather if blacks were poor or disadvantaged, the government would point to the existence of poor whites and say that more needed to be done to address poverty. Meanwhile, Africans had far lower education attainment and economic opportunities than even the more explicitly racist South Africa or Rhodesia. Ironically, it has only been after abandoning its empire that non-whites have achieved their highest positions in Portugal, with the current Prime Minister being of Goan origin and the Minister of Justice being a black woman from Angola, but the more modern attitudes towards race in Portugal have undoubtedly been influenced from Europe.