I think there's a number of factors that don't just apply to the Habsburgs, though they were obviously one of the worst manifestations. The big one was the limited number of Catholic houses left after the reformation which shrank the pool in Germany. Then, ironically, the fecundity of Ferdinand I and Anna, their daughters married into the house of Bavaria, Mantua and Tuscany such that the remaining Catholic houses ended up at least somewhat related going into the 17th century. This also brought about the first round of avuncular marriages when their sons Ferdinand II and Charles II both married nieces as the pool of available Catholic brides was so small (especially once you eliminate the French, who had no prospects available in the late 16th century as they were going through a succession crisis of their own). Which leads to another point, many other houses, such as Bavaria and France were almost just as bad as the Habsburgs. As pointed out double first cousins statistically share about the same amount of DNA as uncles and nieces so Philip IV's avuncular marriage to Mariana of Austria is no worse really that Louis XIV's to Maria Theresa of Spain. Not to mention that Marie de' Medici, as a grand-daughter of Ferdinand I, was a first cousin of Philip IV's mother, Margaret, thus Louis XIV's parents Anne of Austria and Louis XIII were already second cousins.
Bavaria was, I think, worse; Albert V was a second cousin once removed to Ferdinand's daughter Anna, their son William V married Renata of Lorraine who was his second cousin, again through a Habsburg line (maternal descendants of Philip the handsome), their son Maximilian I first married his first cousin Elisabeth of Lorraine then his niece Maria Anna of Austria (who's father Ferdinand II was the child of an avuncular marriage between Charles II of Austria and Maria Anna of Bavaria and so she was also Maximilian's first cousin once removed), their son Ferdinand Maria, somehow turned out just fine and married Henriette Adelaide of Savoy who, as a grand-daughter of Marie de' Medici and Catherine Michelle of Spain, already had both Spanish and Austrian Habsburg blood, their son Max Emmanuel then married Maria Antonia of Austria, his second cousin (who was herself the product of an avuncular Habsburg marriage). So it's not surprising that their son Jose Ferdinand died young as he had a pedigree just as bad as Charles II. Thankfully Max Emmanuel then married a Sobieski princess injecting desperately needed fresh blood that probably saved the Bavarian Wittlesbachs from their own extinction event.
Anyways, point of this is that this issue wasn't limited to the Habsburgs, if you dig around all the major Catholic houses had this problem, it's just not as evident because they didn't always have the same name. Perhaps what made the difference in other cases is that the other houses managed to get a break every 2nd or 3rd generation and 'only' marry a second cousin or at least someone more distantly related than niece or 1st cousin. I think the Habsburgs lost that opportunity only though a few unfortunate deaths, Balthasar Carlos, for example, was from parents only distantly related (2nd cousins) and his marriage to Mariana would have been 'only' to a first cousin. Likewise Ferdinand IV of Austria could have married someone more distantly related than his brother Leopold's OTL marriage to his own niece. They really needed to get some fresh blood in the early 17th century after the previous series of avuncular and first cousin marriages. The only real move where I think there was a better choice available was Philip IV's marriage to Mariana, if he had married someone else, say Anna de' Medici or Eleonora Gonzaga, or maybe even Isabella Clara of Tyrol as a second wife he could have saved his line. This in turn would likely have helped the Austrian line as well.