What's the best way to stop the Ottoman state's setbacks on many of its fronts (Persia, Austria, etc) in the 17th century?
Disband the Janissaries.
Not going to be easy, but yes, that would help.
It is not clear, however, what would replace them.
Disband the Janissaries.
Couldn't the Ottomans just train soldiers like a regular empire that would disband in times of peace?
Most empires tend to have standing armies, actually.
Conscription was rare at the time, and conscripted armies somewhat less effective in general. Compare Europe, where most countries (not really empires though) still heavily realied on hiring mercenary companies at this point, the main exception I know of being Sweden (with a semi-permanent core of standing army that was highly trained, giving her a considerable military edge).
Actually the proprietary system might well work for the Ottomans. Given that the social structure was divided into millets according to faith (i.e Jewish, Muslim, Orthodox Christian, Latin Christian etc) and only Muslims could serve in the military the Sultan's regime could impress upon wealthy non-Muslim individuals and communities that paying for the upkeep of a regiment of Muslim volunteers would be a really good idea.
This would preserve the division between Muslims as fighters and the rest as wealth producers and of course prevent the Muslim elites from establishing their own military power bases. Officers would not be paymasters and the usual link between paymaster and loyal soldier would be severed by religion leaving only allegiance to the Sultan as a common thread. In addition the rivalries between regiments could be encouraged to prevent them uniting together like the Janissaries over grievances and certainly they should have no common accord with the timariot cavalry...though those always boasted that "horsemen do not mutiny".
Of course as with any system it would likely present its own flaws and be open to abuse but it does have the potential to keep the Ottomans more current in military affairs into the C18th.
This would sound perilously close to giving Timar titles to non-Muslims though.
Also, it is not exactly true that non-Muslims never fought in the Ottoman army. The Sultan had Christian vassals whose forces he could and did call upon, most notably Moldavia and Wallachia, but in earlier times, also Serbia.
Well yes I did somewhat give a simplified answer in reference to complexities of Ottoman society as you did in the post to which I replied but it is possible to see from your own nitpick that well the Ottomans did skirt with treating Christian vassals as timariots.
The problem is that you'll find a lot of conservative opposition to do it.
Disband the Janissaries.
Not going to be easy, but yes, that would help.
It is not clear, however, what would replace them.
Except for the ways that it... well, wouldn't.
People tend to forget that the Auspicious Incident was followed by Russia forcing a highly unfavorable treaty on the Turks, who now lacked a military capable of resisting them.
Sure, but it happened at a time when the correlation of forces was much more heavily stacked against the Ottoman state than it would be two enturies earlier.
Unless the Ottomans modernize their economy and state, how much will this help? What use is a more modern army if you still don't have a printing press in your capital?