I assume these are based on your reading in various nonEnglish sources, by the way?
I trust you, I'm just curious, as this is one of the many subjects the English speaking world doesn't cover very well.
You have english-speaking sources covers that. Thomas F. Glick by exemple, from Boston university.
For the Berbers, you had actually another factors : the guys that fought and came back to Africa, and others that replaced them. Fresh troops helps too. A lot.