Would on Earth would they stop Settler from Mexico or the USA from overwhelming their new colony? Who would want to move how would they defend it? In truth it seems a little ASB
Not ASB at all - ABS would be "what if France discovered laser weapons and used them to conquer Mexico." Improbable doesn't equal ASB. Sorry, the overuse of ASB is a pet peeve of mine. I shall stop yelling at clouds, now
Anyway, one of the issues you WOULD run into is the Monroe Doctrine, though this could potentially make it a dead letter (it was really being enforced mainly by Birtain at this time, and I doubt Britain would intervene to force France to give the territory back (though they might saber rattle a bit to get some concessions that want elsewhere).
Now, as to what would prevent Mexico or American settlers from overwhelming the territory? Population. If Mexico had the people to heavily populate the region, they would have done so earlier. This is also about eight years prior to the Mexican-American War and shortly after the establishment of the Republic of Texas. That means that much of the surrounding territory was stil controlled by Mexico and unless the hypothetical American settlers are going to come through the Oregon Territory, they'd have to march through Mexican territory to get to California. First, they would probably not be allowed to do so, and second ... why would they want to? The US already has a LOT of land to fill up first where the hypothetical settlers can settle down in, and said land is much easier to reach. Hell, even Oregon at this point was pretty lightly populated at the time.
Things will change, of course, during the Caliornia Gold Rush (assuming that it happens on que, and it may or may not). During this period, you will definitely see American settlers reaching California and setting up shop. But, there's no reason to assume that said settlers would just shrug their shoulders and say "You know what guys? I got me a crazy idea. Let's rebel for no real purpose, and ask to be annexed by the US! Sound like fun, boys? Lets do it!" Especially since, due to this large divergence, there's absolutely no reason to assume that the US has annexed Texas in this timeline or that there was even a Mexican-American war - so the US may not even share a long land border with French California, nor established the precedent of annexing American Republics established on foreign soil.
Finally, there's leaving out the obvious issue that if the US tried this bold move, it would draw it into war with France. And considering the state of the American navy at that time, that's going to be a tough pill to swollow. This wouldn't be warring against the unstable Mexican Republic, or even the aging Spanish Empire. This would be France. And though the US could win that conflict, there's no reason to assume that it would be easy. Especially since, in order to protect its new colony, France is by its nature going to have to extend its influence over neighboring states: likely the French presence in Mexico is going to be greater and it could conceivably pull the Republic of Texas into its sphere. Should Nappy III come to power on schedule, you know he's gonna enjoy playing around in California and with Mexican politics.
I could see a scenerio where the US uses saber rattling in order to negotiate a deal where they purchase San Francisco from France at some point for a decent chunk of change (Webster has long dreamed of gettign San Francisco as a Pacific Port for the US), though.
So, yeah - as even the OP has stated, it's a somewhat unlikely scenerio. But it's hardly impossible, and should France get California there's every reason to believe that they conceivably could keep it for an extended period of time, and no reason to believe it would inevitably fall into American hands. This would be a huge POD and it would muck up the flow of OTL's events in the far west and south west. Honestly, improbable or not (and lord knows, improbable things have happened in OTL and continue to all the time), its a fascinating idea!
On a side note, I wonder if the Icarians would settle in California in this ATL and if they'd be any more successful than their failed Texas and Iowa colonies in OTL.