Remember that perhaps 500-1000 years went by between the incorporation of Palestine/Israel in to the Roman Empire which accelerated the dispersal of the Jews in Western Europe and elsewhere until the more formal separation of the Jews from the gentile population. During this period of time there was inevitably some "mixing" with the "non-semitic" population. After the triumph of Christianity in Western (and Eastern) Europe and the final demise of the Byzantine Empire a consequence of repeated assaults on Jewish communities was a non-zero incidence of children born of rape who became members of the Jewish community (matrilineal definition of Jewish identity) and thus brought with them genetic material form the surrounding population. It is worth noting in the more eastern parts of Europe this included genetic material from Mongol conquerors.
While in many instances dispersal of Jewish communities included families and women, often the dispersal was heavily male dominated. As in other male heavy communities, this resulted in "local" women being taken as wives, and accepting conversion.
If one looks at Jewish communities around the world, at least before the major movements of the last 100-150 years, it becomes obvious they tended to look very much like the local communities. The Jews of the Arab world looked much like their neighbors, the Jews of Western Europe likewise, the Jews of India or the Jews of Ethiopia and so forth. Also note the Jews of China, who while finally absorbed in to the larger community were physically almost indistinguishable from their neighbors. I do have to disagree with the proposition that those who "looked" more local were more successful in the Ashkenazic/Christian world. Until the breaking down of ghetto walls, distinctive clothing and hair/beard styles among other things forced Jews to look different from their Christian neighbors so eye color/hair color/skin tones were less important to separate one from the other.