One other possibility: An ecological niche that modern humans couldn't compete effectively in. Apparently African pygmy groups were very isolated genetically and could have eventually become a separate species given enough time. In their case, the separation was because a full-sized human just didn't do well in their habitat and pygmy people didn't do well outside it. Off-spring of a mixed couple wouldn't fit well in either world.
Neanderthals may have lasted as long as they did because of an ecological separation. Some theories say that they were specialized for ambush hunting in forested areas and that more open areas were always marginal habitat for them, partly because it required more mobility and those big, muscular bodies wore out quickly when Neanderthals were forced to move long distances to follow the game. That line of thought goes on to say that Neanderthals became extinct when the last ice age reduced the amount of their favored habitat to the point where their population was too low to be viable.
It's an interesting theory, with some interesting implications. Among them: modern humans were specialized open-country types, and developed weapons and tools designed to be light-weight so they could move around more easily, even though they took more energy to produce. When Neanderthals were forced into open country, they developed similar weapons and tools, but as mentioned, they weren't as good at open country living, and Neanderthal skeletons in probable open country almost invariably show signs of crippling arthritis.
The dates when Neanderthals became extinct in various places are in considerable dispute, but it seems that they may have had a last refuge in southern Spain, and may have survived there for thousands of years after they died out elsewhere, with some of the last survivors living near the Rock of Gibraltar, which brings up some cool images.
The 'end of their habitat killed them' theory says that our ancestors probably had little to do with the death of the Neanderthals. The worsening interglacial turned forest to open country which Neanderthals couldn't hand, but our ancestors were already adapted to. I'm a little skeptical about our ancestors being blameless. Even if open country wasn't prime Neanderthal habitat, it was territory they apparently used when there wasn't anything better available. Modern humans moving into that territory would have made Neanderthal adaptation to the ice age advances that much harder.
There may have been some luck factors involved in Neanderthal extinction. There are hints that Neanderthals in the Caucasus may have been wiped out locally by the climate aftermath of a large volcano eruption. There was also a rather nasty strato-volcano in Italy at about the time Neanderthals died out there, though I haven't seen it linked to the disappearance of Neanderthals in Italy.
Sicily was attached to the mainland during the peak of the last ice age, and I could see it being a last refuge of the Neanderthals in Italy--which would have made the Neolithic interesting there. Surviving Neanderthals versus early farmers. Not sure how that would work out. Probably the Neanderthals would have been wiped out.
You almost have to have continent-sized occupied areas to keep a separate species around through the waves of technology that swept out after the invention of agriculture. Most human groups didn't survive as distinct entities. What happened to all of the groups that lived in the way of the Chinese expansion, for example, or any one of several huge expansions in Eurasia?
