A radical change of heart of Arab nobility ruling in North Africa. Some Caliphes actually tried to go smooth on this, and it was totally ignored.
I'm aware the Ummayad Caliphate was a state that could often be summed up as "by Arabs, for Arabs", but is there any way to assuage the Berber complaints and crack down on the Kharijites so revolt doesn't break out?
No. IOTL Berbers sent several queries, some being more or less answered, as the complain they were paying undue tax. The answer was of course "well, now, everyone would have to do that, and not only you".
Other more "smooth" measures were ordered, and totally ignored.
In order to make something about it, you need to have a change about Islam at best, not making Arabity extremly tied with this whole religious/judicial system (Arab being the language used by God). Probably something about Arabic clanism wouldn't hurt, but that would be pre-Islamic change.
Considering how Berber society functioned since IIIth century with a rough division between three regions ("imperial" Maghreb dominated by foreigners, hinterland Maghreb dominated by Berbers kingdoms, and desert as a peripherical) where Berbers political reactions were roughly the same against Arabs than they were against Byzzies, you'd still have huge tensions.
With the condition of IOTL conquest, a revolt was bound to happen sooner or later (I could see it delayed after 650's, but not outright butterflied).