I think you're forgetting about the entire eastern coast of Africa, which had a whole string of important port cities - Adulis, Zeila, Berbera, Mogadishu, Baraawe, Kismayu, Lamu, Malindi, Mombasa, Pemba, Zanzibar, Dar es Salaam, Mafia, Kilwa, Sofala, and Quelimane. The Ethiopians, Somalis, and Swahilis, all had strong maritime traditions. By some accounts, Sofala along the coast of modern Mozambique was, in fact, a port subject to the Great Zimbabwe polity - a civilization based in the interior of the continent but economically tied to coastal trade networks.
As for West Africa, there really wasn't much need for maritime trade. The civilizations that arose there were heavily tied in with the overland, trans-Saharan, camel caravan routes that connected to North Africa and beyond. The known sources of gold to the south, likewise, came from inland regions as well. These inland trade routes were what their economies were based on and there wasn't much need for seaside ports until the European ships came down the coast in the 15th Century.
The Bantu peoples, likewise, were cultures that developed inland and that had no reason to change this socioeconomic orientation until they had maritime cultures to trade with. It's believed that the Bantu element to Madagascar did not arrive on their own accord (they had no proper seafaring technology, having developed inland), but were brought over by Arabs. On the western coast, there was nothing but the Atlantic Ocean and fledgling Bantu cultures that hadn't even been in the area all that long. The Bantu peoples on the eastern side did have a reason to develop ports - lucrative trade prospects in Arabia, Persia, India, and China - and so the Swahili coast culture was born.