We can get even bleaker--your scenario at least has the memory of Apollo to show it could be done.
Here's one I have in mind:
Al Shepard beats Gagarin to space (but not to orbit--though the broad mass of the US population can't tell the difference). There is no Kennedy moonshot goal, Apollo remains the LEO-focused program it was supposed to be initially, with some MOL-scale space stations as annexes to it. This chugs along until 1973, when the oil crisis and stagflation combine to render the program less popular with the public--and with the overall not-that-impressive (at least, to the public) scientific results from microgravity research and military disinterest, the program is quietly wound down, with a reusable lifting body spacecraft proposed as successor but also wound down (perhaps entirely killed under Carter). The Soviets, for similar reasons, ultimately wind things down as well--coming second, they were never able to squeeze as much propaganda value out of it as they got IOTL, and the Politburo is overall less interested (maybe they even squeeze some propaganda out of winding it down--'the warmongering capitalists continue to seek ways to weaponize space, long after experience has shown the lack of utility in that; while their citizens starve and die of drugs, they pander to their warmongering industrialists; we are more enlightened. Now, get in the truck, we're going to Afghanistan...'). There is talk of revival under Reagan, in the context of SDI, but that also dies when the Cold War ends and we get the thrice-damned Peace Dividend. On the unmanned front, we probably see Voyager, but maybe no Viking (by extension, no Mars revival in the 1990s?). Mars is written off as "as dead as the Moon." Hubble might still go up, launched on a Titan III rather than Shuttle. Not sure about Galileo, Ulysses, and Cassini ITTL--maybe the complicated politics of working with the Europeans on those ends up killing them?
Space advocacy functionally doesn't exist ITTL, except maybe the Planetary Society. In your scenario, the occasional space advocate can still point to Apollo as an example of, "it can be done, it has been done, and the results speak for themselves," comparing Apollos 15-17 to the later unmanned probes. ITTL, there isn't even that example, and space advocacy is entirely the domain of a relatively smaller population of planetary scientists and astronomers. Pretty much the only people who talk seriously about space colonies are the LaRouche cult, and even they say it comes after controlled fusion power plants.