Mongol and Kalmyk had buddhism and monastery, some monastery even nomadic.
Kushan in Afghan also worship buddhism
Celtic Ireland managed develop christian monasticism while lacking cities.
Monasticism can develop even in odd place, there are no need of cities for monastery.
These are good points to bring up, but it's important to note how they are different from this situation.
Buddhism was already a major religion when the Mongols, Kalmyks, and Kushans converted. There were already huge communities in India and China for them to encounter and convert. There won't be any in Africa. Those wandering monks are by themselves.
Similarly for Celtic Christianity in Ireland. Europe and Britain were already Christian. Indeed, the reason monasticism succeeded there was because it was a better alternative to the city/diocese base of the bishopric model for the Irish. But it was in constant contact with the much larger Christian communities across the sea. So it does give an example of monasticism as not a reaction to worldly urbanism, but I don't think it's strongly applicable to this discussion.
Furthermore, all these areas were far more rich than East Africa. East Africa has an extremely high incidence of disease since man evolved there and the local pathogens involved with it. The people there are cut off from most trade and technological and cultural exchange. There won't be repeated interactions with Buddhist culture. People are going to be much more concerned about survival than listening to monks talk about the 4 Noble Truths and give up all these distractions so you can find nirvana.
This is why I said that if Buddhism is to succeed, you need to greatly increase the actual amount of trade and interaction than what the original poster suggested (some Buddhist monks arrive in Zanzibar and somehow a successful monastery which converts the Bantus).
The Swahilis and Arabs Muslims develop many cities in East Africa. Mombassa, Kilwa, Malindi, etc. Some cities even exist pre-muslim era. Earlier and developed trade with India will create the same cities.
I am not an expert on African history, but I don't think the coastal settlements really count as cities before the arrival of Muslim trade. If I'm wrong, I'd love to know where I could read about this. The Periplus of the Erathyean Sea gives lot of Africa ports in the Red Sea, but very few south of the Horn of Africa. Are these actually "cities" or simply small towns with good natural harbors? There's a big difference. A small town doesn't have the same "dehumanzing" or "wordly" impact that living in cities do.
I think it's more accurate to say that the increase in Muslim trade in the region propelled city formation. Mombassa was founded around AD 900. Wikipedia states that Persians settled in Zanzibar early enough to found a Zoroastrian temple, but doesn't give any indication to size or importance.
Plus, if we are talking about the original post, Buddhism must arrive before the Bantu migration which is substantial. Bantu moved there around 500 BC to AD 100. That is just around the creation and spread of Buddhism in India. There won't be many settlements there at all during that time.
For Buddhism to convert East Africa, we need to boost interaction between Africa and India A LOT.