If the Napoleonic Wars never happened, would Brazil remained a permanent part of Portugal?
By 1820 Brazil had 4.7 million people (granted over 1/3rd were slaves), whereas Portugal had 3 million inhabitants, so that ought to give some perspective. By 1872, Brazil had 9.9 million inhabitants (3.8 million listed as white), by comparison Portugal had 4.6 million inhabitants by 1878, or just under half of Brazil's population.
In OTL many historians have characterised the 1820 revolution in Portugal as being a reverse of what happened in Spanish America. It was essentially the bourgeois in Lisbon and Porto declaring Portugal's independence from Brazil. Portugal had been under the authority of a Regency headed by British William Beresford. Also, between 1794-1807 Portugal (mostly Lisbon) had experienced a mini-boom in its economy, especially in trade with France which had been blockaded by the Royal Navy.
It's hard to say what would have or could have happened without the invasion of Portugal, but the gold and diamond production of Brazil had declined rapidly after 1760. This had brought about reform in Brazil and Portugal itself by the Marquis de Pombal. One of his reforms was to unify the armed forces and to promote people from Brazil to positions of power in local municipal camaras (city halls). Educated Brazilians were given posts in not only in Portugal, but in Angola, other colonies and Portugal itself. In addition, the army was unified as was the navy. He sought to encourage a centralized empire, and did encourage economic activity such as cotton growing in Pará and Maranhão.
Importantly, there was no legal distinction between people born in Portugal or born in Brazil. Historians have largely argued that this was done out of necessity because of Portugal's small population base. That is not to say that racial/socio-economic distinctions did not exist, but here too was a major difference with Spanish America. The African slave population was much larger and the indigenous population had been largely wiped out due to disease, warfare and outright massacres. Also, slavery by and large even more brutal than the experience in North America. Slaves were imported in large numbers until the 1850s and had such a high mortality rate that their rate of natural growth remained negative until around 1870.
One thing that is important is to note how people in Brazil self-identified in 1822. The concept of nationality and belonging to a nation was rather novel as most people simply saw themselves as subjects of a ruler. It appears that up until the end most Brazilians of European or mostly European heritage called themselves Portuguese. Brazil had a much larger wave of European immigration in the 18th century than Spanish America, so this might be part of the reason. On the other hand, the fact that whites in Brazil had not been excluded from offices and indeed were given a plethora of new roles in the increasing administrative apparatus, seems to have encouraged a sense of unity. Even Simon Bolivar in his writings referred to them as Portuguese. And it seems that during the events in 1821-1822 many of the Brazilian deputies in Lisbon still identified as Portuguese citizens.
Essentially what Portugal would need to keep the empire together for a longer period of time would be a strong monarch. King João VI was probably too weak of a ruler, but in OTL he died in 1826. However, his son Pedro IV (I of Brazil) and grandson would probably provide good monarchs for a state that would reform itself more gradually. Pedro IV would have ruled between 1826-1834. His son perhaps between 1834 to 1891 when he dies.
Brazilians would assume an increasingly larger role in the empire, but at the same time a sense of unity would have to be preserved. However, at some point, metropolitan Portugal may simply declare its independence and lose much of the colonial empire too (at least Angola and São Tomé & Príncipe).
If unity is preserved though, I can see the colonisation of Africa expanding much further inland. Even in OTL, Portugal recruited settlers for Angola in Brazil after 1822. Also, in Dahomey a large "Portuguese" creole community of freed slaves from Brazil existed in the late 19th century. Portugal had declared a protectorate in early 1885 over Dahoney, but France was able to usurp it. Perhaps with Brazil closer, the initiative in Africa would have been greater.
Finally, without the Napoleonic invasions I cannot see the royal court moving to Brazil. Not only was it too far from Europe, but the climate was not the most agreeable. In OTL, Rio would only surpass Lisbon in size in the 1870s, so it is also smaller with far fewer cultural institutions. What I can envision is a sort of Austria-Hungary compromise with two separate parliaments and prime minsters. A single head of state, and joint-ministries of war, navy, foreign affairs and finance for these. A common flag, a common anthem, currency, common citizenship would be kept so in some ways it would be more of a federation of two separate kingdoms.
One thorny issue would be slavery, if the Brazilians have their own parliament, I can imagine it is going to be dominated by the landed gentry of the kingdom. In OTL the institution was only abolished in 1888 by the Princess Regent. In this TL, Pedro V (Pedro II) will probably be able to find another princess as consort and Isabel will be butterflied away. More than likely there is going to be some external pressure to abolish slavery (probably from the British). In Portugal itself, unfortunately I can't see too many people really knowing too much about it or caring, indeed in the Portuguese colonies, the government enacted a law to abolish slavery in 1869, whereby all slaves were to be set free by 1878. However, in Angola even this proved unpopular amongst the settlers there. So this is going to be a thorny issue, perhaps a more gradual abolition will occur, with slavery being definitively abolished over the period of a decade.