Hmm, but they could have brought experts there from the Indies if there had been the incentive to cultivate there, e.g. losing some of their East Indian possessions to another power or to native rebellions, yes?
Well, they did lose most of their colonies as IOTL, but it didn't make them planting spices here. As Nugax said, the Portuguese were more merchants than farmers, when they first planned to start Brazilian plantation of some Asian plants they discovered they didn't have neither the right knowledge to do it neither have access to most of the seeds (as now they Dutch were firmly controlling most of the areas where they were produced).
Actually, since the second half of the 16th century the Portuguese actually tought about introducing in Brazil the cultivation of cinnamon trees, cloves, pepper, ginger and nutmegs. However, the monopolist groups that controlled the commerce with the Indies not only made the government avoid such transfer as manage to make the Crown declare any plantation of these products illegal outside Asia. Some seeds were brought illegally, but never developed large plantations. When the Asian colonies were lost, they finnaly reverted that policy, but then the problems explained above avoided it to happen.
The only Asian spice that that actually grew well in Brazil (to the point of becoming a pest in some areas) and that allowed farmers to have some profit out of it was ginger. In 1671 finally the king published a decree stating that the Brazilian colonists could finally send ginger to Portugal (recognizing the large presence of the forbidden plant in South America). when the news arrived to Rio de Janeiro, the governor and the Municipal Chamber went to principal church of the city to asked for a thanksgiving mass.
Finally, most of these plans were abbandoned when gold was discovered, as it gave the Portuguese much higher profits than any spice could give.