Not true, crusades were also declared to deal with the pagans of northern and eastern Europe, various Slavic and Baltic tribes. And don't forget the 13th century Albigensian Crusade. And other heresy oriented crusades that occurred in Europe
Only after the institution of crusade was established because of the earlier First Crusade to take Jerusalem and the subsequent crusades to defend it. If that never happens, then the institutional legacies around the Crusade ideal - Papal involvement, theological justification, the Holy Orders, even the idea of a Christendom wide movement - never happen.
Now this does not mean there are no wars where Christian powers conquer non-Christian lands. The Charlemagne's wars against the Saxons, the German invasion across the Elbe, Norman conquest of Sicily, and the Reconquista (although here eventually some of the Crusading institutions joined with it in its later stages) all occurred without the idea of Crusade. However, these occurred for more traditional reasons of conquest than the specific Crusade ideal and lacked the distinguishing characteristics of the well known Crusades.
If an "armed pilgrimage" to Jerusalem called by the Pope never takes place in order to keep the route safe for pilgrims, then the distinguishing features of Crusades will likely never happen and so can't be used later on.