Not government, civilization. The people have reverted to hunting, gathering, and small scale agriculture. The people are still there but what they have is not sophisticated enough for shipbuilding industries to allow migration.
They lost the concept of currency. The Wu are a bunch of primitives now.
I'm sure many of them joined the aboriginals in their primitive ways when civilization deteriorated. However, it's hard to discount the possibility of a few leaving when economic conditions started to go downhill, way before total collapse. They didn't have to have a native shipbuilding industry when they could just buy passage on merchant/explorer ships that visited before collapse of Wu.
Furthermore, when I mentioned remnants I also included secret loyalists still in hiding in China and those who migrated to other places aside from Nan too. In OTL Ming resistance persisted even in the late 17th century.