Senseless nitpicking: Louis died at age 67; it was Lucien who died at age 65. Jérôme didn't quite reach 76. And while Napoleon's father died young (38), his mother lived to see her 85th birthday.
But yes, I agree with the general sense of the statement. Yet, what if he did "miraculously" die at a ripe old age? Even by doing nothing but sitting and waiting death on Saint Helena, this would still change history, as it would probably change the political carreer of his nephue. But I must admit not knowing enough about this time period to say how.
I think that the changes will start to appear much earlier than the 1850s: a Napoleon still alive during the 1820s and the 1830s will be always seen as a potential danger to the European order established at Vienna. This is going to work against his siblings, and even more so the children of his siblings. It can go in very different ways, depending how individuals are going to behave in this ATL. Is Louis Napoleon still going through a period of revolutionary leanings? In such a case, he might not survive his escapades, be them in Italy or in France. By the same token, he might refrain from this behavior and fade away into obscurity. Anyway you look at that, the likelihood for him to become president of the French republic and later on emperor is likely to be butterflied away.
I suppose that even the life of Napoleon's son will be quite different, he is likely to be kept under much stricter wraps. On the flip side, he will be kept away from the army, which may mean no pneumonia in 1831 and no death in 1832.
A surviving Napoleon will also change in some way the behavior of the Bourbons in France: it makes a difference to have the former emperor still alive in the 1820s.
These changes may well butterfly away the insurrection of 1830 (maybe it is repressed, maybe it comes earlier), and do away with the Orleans monarchy. In such a case, also the Belgian insurrection will fail, if it happens (and probably it will not happen without a sympathetic regime in France).
This does not mean that the Bourbons will necessarily remain on the throne, though. The issues which were at the basis of the insurrection will not go away because the insurrection has been crushed.
I would not really worry whether Napoleon will be allowed to retire to France in his old age, but rather about what is going to change in Europe during the 1820s and 1830s.