You don't understand the American mind-set. For that matter, Japan in OTL 1941 didn't understand the American mind-set, and might have made the same assumption you did. The US would not back down.
Quite right. The Imperial government spent a great deal of effort trying to get some sort of arrangement with the U.S. (on terms similar to those outlined). The intermediaries approached all explained to the Japanese that they were on a fool's errand.
There is also the practical matter of trying to achieve this sort of genocide. The population of Hawaii did not consist of sheep herds, not by a long stretch. Faced with assured death at the occupier's hands the population would as likely have filleted every Japanese soldier in the Islands as not, followed by an American Crusade of vengence that would have made OTL's Pacific Campaign look like a Scout Jamboree.
As far as the base question - Could Imperial Japan blackmail the U.S.? No.
Can a 3rd grader with a pea shooter defeat a Infantry company?
Could the U.S. set up an unbreakable blockade around the Japanese Home Islands, resulting in slow starvation for everyone living there? Yes. One existed IOTL .
Could the U.S. burn Japan to the ground, one city at a time? Yes; As IOTL.
Would the U.S. have just left that blockade in place for years while the Japanese people slowly disappeared, after some mad attempt by the Japanese to eradicate Hawaii's entire population? Hard to say for sure, but the sub blockade and the firebombing missions were fairly low casuality evolutions that could have been maintained as long as the U.S. felt the need.
The U.S., as a matter of national policy, more or less exterminated the Native American population, just because they were in the way. Imagine what the country would do in this case.