That's part of it. Carter did have one advantage going into October: Reagan was widely seen as a right wing extremist, and in fact Carter was leading in the polls until late the the campaign. But the TV debate took that away, and Carter was done for.
A lack of a Kennedy candidacy doesn't change any of that. But perhaps Carter loses by a slightly smaller margin due to butterflies. He might win Massachusetts for example.
This doesn't even free Carter from having a primary challenger--there was always Jerry Brown (who might have done better than he did in OTL if Kennedy hadn't run).
But even with no primary challenger at all, Carter would lose to Reagan. No doubt a serious primary challenger is always a bad sign for how an incumbent president will do in November. But it is a symptom more than a cause of the incumbent's unpopularity.