To begin with a short disclaimer, the origins of the Mamluk or Ghilman soldiers that would come to dominate the Islamic world, is shrouded in some ambiguity. In a similar mode to crucifixion in the Roman world, chroniclers often neglect to describe certain intricacies that are interesting to the modern reader. One is reminded of the stories within most Islamic chronicles related to due the siege of Makkah, the chronicler describes the intricate dispute over whom the citizens of the city should make the call to prayer to and whom they should honor and the tale was to be very humorous and also was something that likely interested the people who would have read this work. However, to the modern historian, whose goal is to extract as much from the text as possible, it becomes difficult when the chronicles mention various intricate disputes over prayers and yet refer to the purchasing of Mamluks in the slave market in 814 as just as simple as purchasing bread at the local market. Thus, the disclaimer is clear, the opinion is simply that, my opinion regarding the origin of the Mamluk system and its reasons.
Within the past and today in Islamic rulings generally, the view held is that conscription is considered a disliked function and or haraam (my opinion). Any sort of government action that has the effect of removing one from his duties to Allah is to be seen as a great evil and to be rejected. For instance, all Islamic jurists agree, the ideal of the western income tax or sales tax, wherein percentages of money are taken from individuals, is considered a major sin of those who perform such and they will enjoin with the hellfire. The reason is, a tax that is levied upon the people without stipulations that such tax should last only a short time and period, have the effect of lessening the ability of the Muslim to perform his duties to Allah, as he is forced to have less money that can be spent on his/her family and less money for the tithe incumbent upon him/her. Further, it is considered an action that disgusts Allah, the idea of prohibiting or taxing the fair livelihoods of the normal folk. Thus, if possible without lying or deception, it is allowed for a Muslim to avoid any tax that directly targets the person when not in the fashion of say, a poll tax wherein one must pay a set amount at an appointed time.
In the same vein, the concept of a government, Islamic or Kufr, which directly conscripts through coercion the people of its land, is considered the same. For most of the life in military, a solider when not at war, spends his time training, sitting in fortresses, patrolling aimlessly, eating off the people in his community as he is often not allowed to go forward and make his own production, etc... This sort of wasteful activity is permissible for those who volunteer to do this, but for the state, it is not permissible to force soldiers into an army without a clear reason, which the reason is direct war and threat of disaster to the lands in which you reside. So, the ruling is, Islam does not allow the peace time military conscription of soldiers as this 1,takes them away from their productive lives with families 2, lessons their abilities to do their duties to Allah 3, they are forced to follow orders of masters who may lead them astray.
Despite this, the ruling on standing armies is affirmative, a Muslim ruler or land may maintain a standing army of volunteers or slaves, such as a tribal council for fear of war, raises funds to supply a militia and the local men volunteer to fight. Or in the second case, a master within Islam has control over his slave as his property and has certain rights to place this person in any occupation. So, the two types of armies that are accepted when at peacetime for the Muslim is the volunteers, which are sometimes referred to as mujahadeen (those who wage jihad) and an army composed of slaves and such or mix between these two.
During the days of the Abbasid caliphate, this ruling was known and enforced generally. Rulers who coerced soldiers to war, such as Hajjaj bin Yusuf, the governor of Iraq (during the middle Umayyad regime), were seen as extreme levels of tyranny and derided as evil without equal. Hajjaj bin Yusuf's conscripted army (Peacock Army), further, rebelled against him in a spiral of Kharijite-Shi'a revolt that was aligned even with Sunni sympathies that rejected the tyranny of Hajjaj bin Yusuf. There was also the case of the rebelling Egyptian fleet at the siege of Constantinople against their Umayyad masters and allowing the Byzantines to gain the critical victory at sea against the Caliphate. Thus the idea of a conscripted army so popular in other areas of the world which focused on state control, was not to be in the Islamic world and its own complex legal codes and cultural situation.
This however begs the question, how did the Umayyad and Abbasid do so well in war? In the case of the Umayyad, there was a large quantity of volunteers, who surged to the edges of the Islamic world to battle the infidels and pushed the borders forward on either end of the Caliphate and in all directions. Thus, a type of decentralized warfare, which would be interrupted by focused campaigns by the Caliph who through his authority could call existing warriors and generals into his command for a focused invasion, most importantly in the various sieges of Constantinople during the Umayyad period or in internal conflicts or rebellions (in some cases, this warrants a conscription).
In the case of the Umayyad, their armies focused on the edges of the empire and fixated upon Arab tribal customs of loyalty and what have you, remained less at arms to deal with internal conflict and less effective. The amount of revolts the Umayyad incurred during its rule and the ineffective responses often betray the success that the Umayyads usually held when fighting exterior foes. Through the early sections of the Umayyad period, the Caliphate had more or less kept Byzantium on the defensive, conquered Iberia, Africa, Berber tribes, raided the Nubians, pushed into the Indus, pushed into the Kwarezm, more or less defeated the Khazars, managed to spread its naval capacity to the level that exceeded most powers at the time, including the Byzantines. However, the Umayyad often had great issues internally, especially with Kharijite and Shi'a factions, mostly deriving from rebellions in Iraq and Africa. After many decades of decline and weakening military success on its borders, rogue elements of the armies on the frontiers in the east, would rebel and take the throne and establish the Abbasid Caliphate.
Reasoned well with its rise to power, we may assume that a certain consciousness developed in the early Abbasid realm relating to the shaky ground with which they sat. Howe exactly could they stop what happened to the Umayyads from happening to them? As we see, the Abbasid set about reforming the Islamic world under a new paradigm and asserting its many privileges. The most paramount action was the inclusion of non Arabs into the ruling circles and into Islam and the slow removal of the Arab chauvinism that was exhibited in the Umayyad period. The most early example of this was the creation of Baghdad using the ruins of Cteshipon and Seleucius as its bricks and stones, a fresh start for the new regime that would find uses in the tools of the Syriac world within its realm and the Jewish inhabitants. Namely employing these groups as scribes, further the rapid Islamification that began to spread in Iran, influenced by both a more strict interpretation of Islamic law (the Abbasid instituted laws that allowed any Muslim to vandalize non Dhimmi property that was associated to other religions without punishment) and the allowance of Persian converts to Islam, further increased the diversity of rule and influence in the Abbasid court and ruling class.
During the Abbasid period, we also begin to see more formal feudal (not European manorialism that is related to Rome, more along lines of governors who ruled lands as strongmen and owed loyalty to the Abbasid caliph) creations such as the Tahirids in Iran who would become the foremost supporter of the Abbasid power. This innovation also included newly converted Muslim as governors, such as al-Afshin, the governor of Sogdiana and Mayzar al-Qarinvand the governor of Daylam. In this scenario, the Abbasid were building a base of support that insulated it from much of the perceived threats that the Umayyad experienced and protecting further the stability it perceived.
One innovation that began along with these, when first mentioned, is of the purchase of Turkic slaves ghilman/mamluks by Abu Ishaq (future caliph al-Mu'tasim, 8th Abbasid Caliph) and also by al-Ma'mun (7th Abbasid Caliph) who unlike slaves to toil in the field or in the kitchen or in harems, was to be sent to battle and used as warriors and generals for their master and equally be conferred status. One imagines the scene, in the Islamic conquest, much of the old order of Asia was overturned as this movement and surge of Arabs/Muslim spread across Iran and struck deep into the hearts of Asia and into the heartland of the Indus Valley. In this spread, the old powers of the Asia are seen to be receding in power and that of the Muslim in the ascent, especially in the victory of the Abbasid against the counter moves by the Tang Anxi/Western Protectorate who was defeated critically in Talas and the Abbasid would be seen to make gains in the rest of Central Asia west of Qashqar and Qhotan. Eastern fronts of the Abbasid would wage a constant war of raids and attacks into the steppe and Hindustan, bringing in slaves from these lands, many of whom former warriors and even so, people caught in the grips of war, selling their own children on the frontiers to slavery. Of these, Turkic soldiers with great skills from the north and east were brought in fair numbers, wherein they would by these two brothers (al-Ma'mun and al-Mu'tasim) to serve as their warriors and commanders and in theory, would be loyal and always at the ready to defend the Abbasid throne.
In the regime of these two brothers (which can be considered very similar in philosophy,) we see the beginning and intense usage of these Mamluks, greater support of the Abbasid throne of the scholarly pursuits of the intellectuals (especially in the domain of discovering or defending older Greek philosophies), increased aggressiveness in military matters and the support of the Mu'Tazilism against Sunni Islam, Kharijites and the Shi'a. The undertones of all of this, is the final point, the ascendancy of Mu'Tazilism and the usage of these scholars as bureaucracy, partisans and inquisitors. In the waning stages of al-Ma'mun, the regime shifts to openly enforcing Mu'tazilite views upon its people in the form of the Mihna or the inquisition, wherein the Abbasid throne made it an illegal offense to propagate some of the more fundamental beliefs of the Sunni or Shi'a. Also as an accomplice or consequence of the new religion at the helm, there was a newly increased legalism and culture of offenses and legal tyranny associated with this period, with which the Mamluks would become the heralds of the overbearing rule of the traitor Caliph. It was also in this period we see the Shi'a subjected to essentially protocols calling for their execution if found to be a Shi'a that was actively believing in those creeds seen to be traits of the Shi'a.
It should be reminded that these effects occurred during the reign of two very powerful and reformist Caliphs, who supported all of the innovations previously mentioned. Whom my opinion is, they knew their objectives, to deviate from the Arab traditions, lessen the influence of both the newly converted Muslim governors and the Arab tribes and clans. This thus included bringing together many forces under its wing and using these as hammers to block out the power of these hated groups. Thus, the growth of a hostile and scheming bureaucracy and intellectual community of Mu'Tazilah and the growing powerful and independent Mamluks (and associated Abbasid family members who in theory were their owners), was under rulers who supported these functions and were also strong enough to contain their excesses. The subsequent rulers however, would not both support these factions to the same degree and were not strong enough to reign them in under Abbasid authority and over time, these two aligned factions under the 7th and 8th Caliph, would become enemies. During the Anarchy of Samarra, following the reign of al-Wathiq, the Caliphate would be controlled defacto by factions in the court, especially the varied Mamluk leaders, who routinely made it their power to elect Caliphs and also frequently assassinated or outright murdered without discretion the Caliph with whom they had enmity.
The last strong reformer of this period, al-Muhtahdi would be one of the few who showed signs of restoring the Caliphate to its office in the Samarra chaos. The son of al-Wathiq, also the son of a European slave (his father al-Wathiq [whose father was al-Mu'tasim], was also the son of a European slave, supposedly a Greek woman), he was known for his fair tone and blond hair. However, his more important features were in his strict adherence to Sunni Islam and his extreme ferocity in politics. Upon ascension to the throne, seen to be weak by those who allowed his ascension, he saw his authority weakened by his predecessors and sought to restore Caliphal power and did so by extending his power to decimate the massive court corruption, vice (such as music played in court) and restore the ancient alliances with the Arab tribes and counter the Mamluks in the military. He in a very Machiavellian fashion would also seek to cause his enemies int he court to attack him, whilst making appeals to the people and gifting them charities and making shows of his support to them; in short he had the support of the general populace of Iraq who was not connected to power. However, his reign would be cut short, his fearsome reform methods and anti corruption campaigns radicalized his foes, who like many caliphs previous, moved against the Caliph overtly and as it is described they attacked the Caliph in his palace, who was forced to flee to the roof where he was chased by the mamluks, who once capturing him, kicked his crotch area and his face/stomach until he died.
Following the death of the reformer al-Muhtahdi, the Calipahte would be held supposedly by al-Mu'tamid, who was in effect controlled by his brother al-Muwaffaq who held the title of regent for his life and the Mamluk faction who held the military alongside al-Muwaffaq, while the real Caliph would spend his time under house arrest his entire Caliphate. While in the rest of the regions, the Caliphal authority had broken down, in Egypt, the ruler was the Tulunid Mamluk factions who technically swore fealty to the Abbasid, yet their loyalties betrayed them generally (aside from the Tulunids joining the Abbasid attack on the Zanj capital of al-Mukhtara, which then had been besieged for nearly a year by al-Muwaffaq) and Iran which due to the rebellion of the Yahya ibn Layth al-Saffar had exited the Abbasid realm in both name and in reality (the Saffarids had attempted to conquer Baghdad, but were defeated by the Abbasid armies and the Saffarids were also defeated by the Zanj armies in Ahvaz).
I hope this answers some of the questions you had, if there is any questions, I am open to ask all.
Within the past and today in Islamic rulings generally, the view held is that conscription is considered a disliked function and or haraam (my opinion). Any sort of government action that has the effect of removing one from his duties to Allah is to be seen as a great evil and to be rejected. For instance, all Islamic jurists agree, the ideal of the western income tax or sales tax, wherein percentages of money are taken from individuals, is considered a major sin of those who perform such and they will enjoin with the hellfire. The reason is, a tax that is levied upon the people without stipulations that such tax should last only a short time and period, have the effect of lessening the ability of the Muslim to perform his duties to Allah, as he is forced to have less money that can be spent on his/her family and less money for the tithe incumbent upon him/her. Further, it is considered an action that disgusts Allah, the idea of prohibiting or taxing the fair livelihoods of the normal folk. Thus, if possible without lying or deception, it is allowed for a Muslim to avoid any tax that directly targets the person when not in the fashion of say, a poll tax wherein one must pay a set amount at an appointed time.
In the same vein, the concept of a government, Islamic or Kufr, which directly conscripts through coercion the people of its land, is considered the same. For most of the life in military, a solider when not at war, spends his time training, sitting in fortresses, patrolling aimlessly, eating off the people in his community as he is often not allowed to go forward and make his own production, etc... This sort of wasteful activity is permissible for those who volunteer to do this, but for the state, it is not permissible to force soldiers into an army without a clear reason, which the reason is direct war and threat of disaster to the lands in which you reside. So, the ruling is, Islam does not allow the peace time military conscription of soldiers as this 1,takes them away from their productive lives with families 2, lessons their abilities to do their duties to Allah 3, they are forced to follow orders of masters who may lead them astray.
Despite this, the ruling on standing armies is affirmative, a Muslim ruler or land may maintain a standing army of volunteers or slaves, such as a tribal council for fear of war, raises funds to supply a militia and the local men volunteer to fight. Or in the second case, a master within Islam has control over his slave as his property and has certain rights to place this person in any occupation. So, the two types of armies that are accepted when at peacetime for the Muslim is the volunteers, which are sometimes referred to as mujahadeen (those who wage jihad) and an army composed of slaves and such or mix between these two.
During the days of the Abbasid caliphate, this ruling was known and enforced generally. Rulers who coerced soldiers to war, such as Hajjaj bin Yusuf, the governor of Iraq (during the middle Umayyad regime), were seen as extreme levels of tyranny and derided as evil without equal. Hajjaj bin Yusuf's conscripted army (Peacock Army), further, rebelled against him in a spiral of Kharijite-Shi'a revolt that was aligned even with Sunni sympathies that rejected the tyranny of Hajjaj bin Yusuf. There was also the case of the rebelling Egyptian fleet at the siege of Constantinople against their Umayyad masters and allowing the Byzantines to gain the critical victory at sea against the Caliphate. Thus the idea of a conscripted army so popular in other areas of the world which focused on state control, was not to be in the Islamic world and its own complex legal codes and cultural situation.
This however begs the question, how did the Umayyad and Abbasid do so well in war? In the case of the Umayyad, there was a large quantity of volunteers, who surged to the edges of the Islamic world to battle the infidels and pushed the borders forward on either end of the Caliphate and in all directions. Thus, a type of decentralized warfare, which would be interrupted by focused campaigns by the Caliph who through his authority could call existing warriors and generals into his command for a focused invasion, most importantly in the various sieges of Constantinople during the Umayyad period or in internal conflicts or rebellions (in some cases, this warrants a conscription).
In the case of the Umayyad, their armies focused on the edges of the empire and fixated upon Arab tribal customs of loyalty and what have you, remained less at arms to deal with internal conflict and less effective. The amount of revolts the Umayyad incurred during its rule and the ineffective responses often betray the success that the Umayyads usually held when fighting exterior foes. Through the early sections of the Umayyad period, the Caliphate had more or less kept Byzantium on the defensive, conquered Iberia, Africa, Berber tribes, raided the Nubians, pushed into the Indus, pushed into the Kwarezm, more or less defeated the Khazars, managed to spread its naval capacity to the level that exceeded most powers at the time, including the Byzantines. However, the Umayyad often had great issues internally, especially with Kharijite and Shi'a factions, mostly deriving from rebellions in Iraq and Africa. After many decades of decline and weakening military success on its borders, rogue elements of the armies on the frontiers in the east, would rebel and take the throne and establish the Abbasid Caliphate.
Reasoned well with its rise to power, we may assume that a certain consciousness developed in the early Abbasid realm relating to the shaky ground with which they sat. Howe exactly could they stop what happened to the Umayyads from happening to them? As we see, the Abbasid set about reforming the Islamic world under a new paradigm and asserting its many privileges. The most paramount action was the inclusion of non Arabs into the ruling circles and into Islam and the slow removal of the Arab chauvinism that was exhibited in the Umayyad period. The most early example of this was the creation of Baghdad using the ruins of Cteshipon and Seleucius as its bricks and stones, a fresh start for the new regime that would find uses in the tools of the Syriac world within its realm and the Jewish inhabitants. Namely employing these groups as scribes, further the rapid Islamification that began to spread in Iran, influenced by both a more strict interpretation of Islamic law (the Abbasid instituted laws that allowed any Muslim to vandalize non Dhimmi property that was associated to other religions without punishment) and the allowance of Persian converts to Islam, further increased the diversity of rule and influence in the Abbasid court and ruling class.
During the Abbasid period, we also begin to see more formal feudal (not European manorialism that is related to Rome, more along lines of governors who ruled lands as strongmen and owed loyalty to the Abbasid caliph) creations such as the Tahirids in Iran who would become the foremost supporter of the Abbasid power. This innovation also included newly converted Muslim as governors, such as al-Afshin, the governor of Sogdiana and Mayzar al-Qarinvand the governor of Daylam. In this scenario, the Abbasid were building a base of support that insulated it from much of the perceived threats that the Umayyad experienced and protecting further the stability it perceived.
One innovation that began along with these, when first mentioned, is of the purchase of Turkic slaves ghilman/mamluks by Abu Ishaq (future caliph al-Mu'tasim, 8th Abbasid Caliph) and also by al-Ma'mun (7th Abbasid Caliph) who unlike slaves to toil in the field or in the kitchen or in harems, was to be sent to battle and used as warriors and generals for their master and equally be conferred status. One imagines the scene, in the Islamic conquest, much of the old order of Asia was overturned as this movement and surge of Arabs/Muslim spread across Iran and struck deep into the hearts of Asia and into the heartland of the Indus Valley. In this spread, the old powers of the Asia are seen to be receding in power and that of the Muslim in the ascent, especially in the victory of the Abbasid against the counter moves by the Tang Anxi/Western Protectorate who was defeated critically in Talas and the Abbasid would be seen to make gains in the rest of Central Asia west of Qashqar and Qhotan. Eastern fronts of the Abbasid would wage a constant war of raids and attacks into the steppe and Hindustan, bringing in slaves from these lands, many of whom former warriors and even so, people caught in the grips of war, selling their own children on the frontiers to slavery. Of these, Turkic soldiers with great skills from the north and east were brought in fair numbers, wherein they would by these two brothers (al-Ma'mun and al-Mu'tasim) to serve as their warriors and commanders and in theory, would be loyal and always at the ready to defend the Abbasid throne.
In the regime of these two brothers (which can be considered very similar in philosophy,) we see the beginning and intense usage of these Mamluks, greater support of the Abbasid throne of the scholarly pursuits of the intellectuals (especially in the domain of discovering or defending older Greek philosophies), increased aggressiveness in military matters and the support of the Mu'Tazilism against Sunni Islam, Kharijites and the Shi'a. The undertones of all of this, is the final point, the ascendancy of Mu'Tazilism and the usage of these scholars as bureaucracy, partisans and inquisitors. In the waning stages of al-Ma'mun, the regime shifts to openly enforcing Mu'tazilite views upon its people in the form of the Mihna or the inquisition, wherein the Abbasid throne made it an illegal offense to propagate some of the more fundamental beliefs of the Sunni or Shi'a. Also as an accomplice or consequence of the new religion at the helm, there was a newly increased legalism and culture of offenses and legal tyranny associated with this period, with which the Mamluks would become the heralds of the overbearing rule of the traitor Caliph. It was also in this period we see the Shi'a subjected to essentially protocols calling for their execution if found to be a Shi'a that was actively believing in those creeds seen to be traits of the Shi'a.
It should be reminded that these effects occurred during the reign of two very powerful and reformist Caliphs, who supported all of the innovations previously mentioned. Whom my opinion is, they knew their objectives, to deviate from the Arab traditions, lessen the influence of both the newly converted Muslim governors and the Arab tribes and clans. This thus included bringing together many forces under its wing and using these as hammers to block out the power of these hated groups. Thus, the growth of a hostile and scheming bureaucracy and intellectual community of Mu'Tazilah and the growing powerful and independent Mamluks (and associated Abbasid family members who in theory were their owners), was under rulers who supported these functions and were also strong enough to contain their excesses. The subsequent rulers however, would not both support these factions to the same degree and were not strong enough to reign them in under Abbasid authority and over time, these two aligned factions under the 7th and 8th Caliph, would become enemies. During the Anarchy of Samarra, following the reign of al-Wathiq, the Caliphate would be controlled defacto by factions in the court, especially the varied Mamluk leaders, who routinely made it their power to elect Caliphs and also frequently assassinated or outright murdered without discretion the Caliph with whom they had enmity.
The last strong reformer of this period, al-Muhtahdi would be one of the few who showed signs of restoring the Caliphate to its office in the Samarra chaos. The son of al-Wathiq, also the son of a European slave (his father al-Wathiq [whose father was al-Mu'tasim], was also the son of a European slave, supposedly a Greek woman), he was known for his fair tone and blond hair. However, his more important features were in his strict adherence to Sunni Islam and his extreme ferocity in politics. Upon ascension to the throne, seen to be weak by those who allowed his ascension, he saw his authority weakened by his predecessors and sought to restore Caliphal power and did so by extending his power to decimate the massive court corruption, vice (such as music played in court) and restore the ancient alliances with the Arab tribes and counter the Mamluks in the military. He in a very Machiavellian fashion would also seek to cause his enemies int he court to attack him, whilst making appeals to the people and gifting them charities and making shows of his support to them; in short he had the support of the general populace of Iraq who was not connected to power. However, his reign would be cut short, his fearsome reform methods and anti corruption campaigns radicalized his foes, who like many caliphs previous, moved against the Caliph overtly and as it is described they attacked the Caliph in his palace, who was forced to flee to the roof where he was chased by the mamluks, who once capturing him, kicked his crotch area and his face/stomach until he died.
Following the death of the reformer al-Muhtahdi, the Calipahte would be held supposedly by al-Mu'tamid, who was in effect controlled by his brother al-Muwaffaq who held the title of regent for his life and the Mamluk faction who held the military alongside al-Muwaffaq, while the real Caliph would spend his time under house arrest his entire Caliphate. While in the rest of the regions, the Caliphal authority had broken down, in Egypt, the ruler was the Tulunid Mamluk factions who technically swore fealty to the Abbasid, yet their loyalties betrayed them generally (aside from the Tulunids joining the Abbasid attack on the Zanj capital of al-Mukhtara, which then had been besieged for nearly a year by al-Muwaffaq) and Iran which due to the rebellion of the Yahya ibn Layth al-Saffar had exited the Abbasid realm in both name and in reality (the Saffarids had attempted to conquer Baghdad, but were defeated by the Abbasid armies and the Saffarids were also defeated by the Zanj armies in Ahvaz).
I hope this answers some of the questions you had, if there is any questions, I am open to ask all.