I think people are developing a very simplistic view of the Crusades (the real ones, not the later ones after they gave up on the Holy Land). It was not a matter of looking at the map and going "Non Catholics! Attack!"
If the Crusaders take the Levant, they'd try to go for Egypt next, as historically, because you need a friendly Egypt to keep the Levant safe. Beyond that, if they were still in the area, its likely that future Crusades would be focused on keeping what they held, which would be a task enough in and of itself. Europe does not have an unlimited pool of manpower and resources and a logistical base from which to launch invasions through the Red Sea...