An Austro-Hungarian Libya would've been governed differently to how other parts of the empire were governed. In OTL, the Austro-Hungarians governed Bosnia in a way that took into account the fact that there was no organization of the disparate communities resembling the organization of the churches in other parts of the Empire. In OTL's Bosnia, the mufti of Sarajevo, Hilmi Hadžiomerović, was made mufti of the whole of Bosnia and was given control of the sharia court. On the initiative of the Austro-Hungarian authorities, this court was to be presided over by a four-man council – like an episcopal synod. The Muslim communities were granted autonomy in their financial administration and organization of educational establishments and religious institutions. The structure of the Islamic communities was based on the Christian bishoprics and parishes. This was an attempt to organize the region's religious life into a church-like organization under Austro-Hungarian control.
As for legal jurisdiction, allowance was made for the specific situation. As citizens of the Habsburg Monarchy, Bosnian Muslims were subject to Austro-Hungarian civil law, but questions of family and inheritance law were left to the sharia. The Muslim legal scholars were not only appointed and paid by the Austro-Hungarian administration but also constrained to cooperate with the civil jurisdiction, also in terms of court procedure. To provide training in both legal traditions, a state Islamic school of law was established in Sarajevo in 1887.
So a mufti, through a four man sharia court, would rule over an AH-Libya, with autonomy in financial, religious and educational institutions, with Imperial legal jurisdiction only affecting areas outside of family disputes. Libya would effectively have more autonomy in it's affairs than anyone else in the Empire, except for the Dual Monarchy of course.