The unarguable existence of massive slave-hunting and slave-trading networks (mostly Barbary and Ottoman) of early modernity that specifically sought out slaves from European people racialized as "white" or its Arabic category-equivalent would instantly dismiss that notion. There were racial or colour categories outside of Europe, and "white" was one of them. Capturing "white" slaves was something people did very deliberately, same as they deliberately captured "black" slaves. Nor is this something that simply stopped once Europe started sailing the world. It predated and coexisted with the Atlantic slave trade throughout the latter's existence. It outlasted it, too.
Were they targeted because they were thought of as "white", or because they were Christian though? Genuine question, I'm not an expert on this part of history.