As I recall, Carter's reputation was pretty abyssmal in the first few years of the 80s. At the '84 GOP convention, for example, speaker after speaker enthusiastically conjoined the names "Carter" and "Mondale", to obvious good effect.
Your scenario would require not only that Reagan be discredited, but that Carter be rehabilitated. Maybe if details of an October Surprise are released, revealing that Reagan deliberately prolonged the agony of the hostages in order to secure his own victory. Reagan then becomes the most hated man in American history, and people think "Aw, gee, the hostage crisis wasn't really Carter's fault after all", and he gets makes a comeback in '84 on the sympathy vote.
Not that I believe in October Surprise, but that's about the only scenario I can come up with.