To do this requires two things: Sega needs to do better, while Nintendo has to do (significantly) worse.
The first shouldn’t be too hard; let Sega keep its focus on the Genesis instead of releasing the Sega CD and 32X, and then release a Saturn that’s less complex and easier to develop for. Also, release a proper 3d Sonic game for the Saturn, to maintain the franchise’s momentum into the 3d era.
But how to screw Nintendo? We need to keep in mind, Nintendo also had a successful handheld business which served as an additional revenue stream. To screw Nintendo enough to leave the hardware market entirely, we’d need to screw both their consoles (especially the N64)
and handhelds. Let’s start with the N64. One interesting way to make it less successful is something I’ve mentioned in another thread earlier this year:
Instead of reneging on their deal with Sony and going to Phillips, Nintendo decides to renegotiate the whole thing, so Sony’s work on the SNES CD continues for a while longer. However, just like IOTL, Nintendo eventually scraps the idea of a CD add-on for the SNES. IOTL, as part of dissolving their agreement with Philips, Nintendo gave them the license to use several of their characters, including Link, Zelda, Mario etc. in games for the CDi. In this case however, since they kept working with Sony, it’s Sony who gets the rights instead.
If you think about it, it was quite risky of Nintendo to not only allow another company to make Mario or Zelda games, but also to release them on a non-Nintendo console. IOTL, Phillips’ CDi games (most famously Zelda) were total garbage of course, but Nintendo couldn’t have known that beforehand. So what if a more competent company had gotten their hands on the rights for Nintendo characters? I think we can all agree that Sony was infinitely more competent than Phillips when it came to gaming, even though they were a newcomer in the market too.
So what if Sony, after getting the rights for Nintendo characters (and having decided to make their own console after the agreement with Nintendo is dissolved), had given the task of making Zelda or Mario games to developers that were actually competent? Imagine if instead of Crash Bandicoot, Naughty Dog was contracted to make a 3d Mario game for the PS1. Or Insomniac doing a Zelda game. Or Sony gave the job to their own internal teams, many of which IOTL made lots of original high-quality PS1 games. Not only would the PS1 have gotten its own ‘Nintendo games’ (which would’ve been crazy enough on its own), but it’s even possible that those games would’ve been released before Nintendo released their own Mario and Zelda games for the N64! In other words, Zelda and Mario could’ve made the jump to 3d on a non-Nintendo system first! By the time the N64 would’ve come out, the PS1 would’ve probably had an even greater head start in terms of sales, while the N64 would’ve seen quite a bit less hype since Nintendo’s biggest IPs would also have entries on the Playstation.
So, the PS1 comes out like OTL, except this time Sony has the rights to Nintendo’s biggest franchises, and releases 3d Zelda and Mario games on their system before Nintendo can do so for the N64. With their biggest IPs already having made their 3d debut on a different console, the launch of the N64 generates much less enthusiasm. Considering that IOTL Nintendo’s IPs are the main reason why most people bought the N64 in the first place, and in this ATL people can now get good Zelda and Mario games on the PS1, I could see the N64 being a flop similar to the OTL Saturn. Sega’s ATL Saturn meanwhile takes the place of OTL’s N64, being a system that, while not being as successful as the PS1, still sells a decent amount and has a great library of games (and, more importantly, is profitable).
So that takes care of the N64, but what about the Gameboy? The thing about the Gameboy is that it was actually in decline in the mid 90s, which isn’t surprising considering the system was more than 6 years old at that time. Nintendo was already working on a successor system, but then Pokémon came out and breathed new life into the Gameboy (almost half the Gameboy’s lifetime sales occurred after the release of Pokémon). So what if Sega decides to release a new handheld system of their own in the mid 90s (a proper handheld system, not the Nomad which was just a portable Genesis with terrible battery life)? And let’s also say that Sega makes a deal with Gamefreak, and Pokémon is one of the early titles released for this new Sega handheld. This means the Gameboy doesn’t get a second life, and keeps declining until Nintendo releases a successor.
This double blow of Mario and Zelda releasing on the PS1, and Pokémon being exclusive to Sega’s handheld, might be enough force Nintendo out of the hardware market by the 6th gen (maybe Sony even buys them to keep the rights to Nintendo IPs permanently). Sony is still the market leader, but Sega is doing well too, even though they’re second place.