Your challenge is to have things go significantly worse for the Brazilian Empire than they did IOTL. It doesn't necessarily have to outright end (either with a bang or a whimper), but it needs to be considerably less successful.
Perhaps keep Portuguese-Brazillian forces from taking over the Spanish Viceroyalty of Rio de la Plata in the Napoleonic Wars? After Napoleon was defeated in 1816 Portugal was awarded that land at Vienna and control of that river proved hugely beneficial to Brazillian success. Trouble is, I don't see how you could prevent it; I guess your two options are a Spanish victory in that campaign (doubtful) or Spain somehow leaving the Napoleonic alliance (impossible).
I’m not sure it’s quite possible to have Brazil not be a world power without a PoD in the 1600s. Their population is so massive, their mineral and agricultural wealth so vast, their dominance of South America and later the Southern Hemisphere seems inevitable imo.

I think that if you just want to significantly nerf them, the easiest answer is to have them loose the Cisplantine War. By winning and securing Cisplantina, they were forced to build a powerful martial tradition to defend against the Paraguayan-Argentinian Alliance. And due to that tradition, they were easily able to conquer Paraguay and seize dominance of the Rio de la Plata, which made them even richer, and allowed for even more internal and external development.

If they loose the war and then waste resources fighting future wars with Argentina, they would be significantly poorer. Add in Britain or France exploiting this weakness to leverage influence on Brazil, and maybe a Republican Coup or more powerful slave revolts, and you could have a halfbaked power. But I still feel like eventually they would go through the industrial revolution and come up to dominate South America at the very least.

As a side note, Bolivia would likely be far less rich in such a timeline. Their alliance with Brazil was founded on their mutual disputes with Paraguay, in such a timeline it might not exist if Brazil is focused on Argentina. And their wealth mostly originates from the Brazilian transcontintel railings running through their country. That wouldn’t exist without the Brazilians conquering Paraguay. Maybe ITTL the tech and trade hub of the South America is somewhere else? Perhaps Argentina if they’re more successful?
You're gonna need to get rid of Pedro II early enough to butterfly Alfonso The Great. Alfonso was the one who jumped off the Cisplantina victory to hyper industrialize and build a proper navy that they used to establish dominance in the oceans of the southern hemisphere and even conquer Angola from the natives/Portugal and eastern Indonesia from the still chaotic Dutch empire
Simple. Napoleon is hit with a stray bullet, crippling him and making him unable to command. The French never invade Spain and Portugal, and Brazil as we know it never comes to be.

Alternatively, have a massive slave population that is fed weapons and other aid by Portuguese rivals. The frequent slave revolts would either crush Brãzíl or force it to abolish slaves.
Prevent Portugal from getting the Majapahit territories including the parts of Luzon that remained loyal to Majapahit til the end, Bali, Celebes and the Lesser Sunda Islands
We'd probably need a slavery wank actually, as repulsive as the thought is. Otl, slavery was a glass canon of finances- very powerful, but easily broken. As the 18th century closed, sugar profits were declining, and while parts of the slavery driven north (god that feels weird to say as an American) did turn to cotton, it really hit mainstream with the industrial and urban south, especially since they could even produce fertilizer and whatnot cheaper. Somehow keep slavery strong, and you keep brazil weak. Might even get a Confederate style rebellion