You make the comparison to other 'genre' films (fantasy, sci-fi, action) in the OP, but the key difference is that they trade in positive emotions— wonder, spectacle and adrenaline. They're escapist fantasies for people who want to forget about the world for a few hours; in contrast, horror films depict the the bleakest parts of the world. There's just not an appetite for that sort of thing amongst the general public on a continuous basis.
Do you remember the downer sci-fi boom of the late 60s to the early 70s before Star Wars. Films like Soylent Green, Planet of the Apes (1968), Silent Running, Rollerball, Phase IV and other films. They had downbeat endings and a rather bleak endings. Yet they were box office successes despite of it.
Science fiction, fantasy and action films do not need to trade in positive emotions like wonder, spectacle and adrenaline. They can create bleak atmospheres and depressing plots. Take Nier for example, at first glance, it's a fantasy story about a father fighting monsters in attempt to find a cure for his sick daughter. Then you play more of its endings and then your realize how depressing the plot is.
Another example would be Man of Steel. Despite being a Superman film, it's actually a frightening film when you think about it. The horrific applications of superpowers, the devastation brought by superhumans fighting each other, the suffering of civilians during alien attacks, the desperation of a good person and the realistic exploration of superheroes in a mundane world.
On the other hand...
This is a movie that was somewhat more accessible to a family audience, and stayed away from morally twisted themes, in favour of a straight-ahead, old-school ghost story, albeit one delivered with hair-raising state-of-the-art special effects. I also seem to recall a body count of zero, and, with the exception of one gory hallucination, no extended scenes of harm to humans.
This is strangely familiar - you made essentially the same thread here six months ago:
https://www.alternatehistory.com/forum/threads/ahc-make-big-budget-horror-movies-possible.456758/
And the counterarguments are the same - family-friendly non-bloody non-scary horror is difficult to pull off.
It was an interesting conversation, so I decided to spin it off.
Family-friendly, non-bloody horror that is scary is actually not difficult to pull off.
Watch Coraline, The Watcher in the Woods, Paranorman and The Witches are scary not because of blood and gore, but because the effective use of clever writing and suspense.