The Crusades were actually a response to a plea from the Byzantine Emperor for help against the Turks in Anatolia. Pilgrims to the Holy land were not really a primary target for harassment. Of course, the Pope and others said they were, in order to create greater fervor for the Crusade. The plight of the Byzantine Empire didn't really register very high on the concerns of Western Christianity, hence the added incentive.
