Why wasn't the coast and coastal waters considered in need of territorial security?
What would be needed to defend the coasts?
The SE coast was indeed considered in need of territorial security, especially with regards to the
wokou pirates of the 16th Century. And, like darthfanta said, there were plenty of court officials, mercantile backing or no (though merchants could work with the pirates/smugglers as well) who urged the Empire to take a serious look at SE coastal problems.
However, ultimately both the Ming and Qing found solutions to SE security that did not require the construction of expensive fleets. The issue of the
wokou was solved in several ways: militarily, Qi Jiguang's strategy of using militias drawn from local men and local funds was deemed enough to solve the security issue; socially, reform of the Ming tax code and the lifting of the ban on oceanic trade lessened the burden of peasants who would otherwise be tempted to piracy. Diplomatically, granting trade rights to Portugal also meant that it now had no reason to support piracy and instead had every reason to
suppress it for the Ming. All of these cost the central government little, and largely solved the problems that underlay piracy.
Similarly, the Qing sought ways of dealing with piracy that did not require the construction of massive fleets like the one used to conquer Taiwan (especially given the Qing military finance system). The Canton system again gave foreign ships every incentive to help end - rather than contribute to - piracy for the Qing, and this fact was noted by officials who even memorialized against Qing economic retaliation against the Netherlands in response to the Batavia Massacre.
So the point is while issues of security did pop up on China's coasts from time to time, officials had other ways of dealing with them and did not see the construction of a fleet as the only solution. Same thing with the Mongols - the Chinese did not instinctively decide upon the military solution all the time (though generally when they were on the weaker side).