Do you really want a flamethrower on your ship which is filled with gunpowder and built of timber with flammable rigging and sails?
It isn't really something I would think one would want on a capital ship, given the risk in losing it, just in regular service. Imagine the terror each time one tried to train with it. This counts for incendiary shots too if it was used in cannons.
But, might it not make sense onboard smaller frigate ships? Reasonably expendable, fast-building, more capability to maneuver, and they don't take a significant chunk out of your line of battle for using them independently. I imagine that given the terror that sailors feel of fire, a few French fire-frigates would do much to make the British very wary of them, and since their destructive capacity vis-a-vis the enemy fleet at close range would be the same as an equivalently equipped larger ship, enough to make any fight with them suicide, having a few of them on the flanks might restrict British freedom of maneuver and intimidate them in attacking French positions.
Whether it would be a boon to the French though is questionable, since the English will inevitably capture some example of a greek fire weapon and will copy it themselves, and given the more aggressive and close-ranged tactics of the English navy it might be more useful for them than for France, who after all preferred combat at range.