Celtic loan words could perhaps come in via Ireland, it could perhaps establish itself as a part of the modern world with no Norman invasions. But generally yeah, they wouldn't be very many.
The cliched thing in this scenario is for English to remain a utterly Germanic language. This though is a big no.
Just look to Dutch and the Scandinavian languages- no French rule there yet still a fair few French words. With the French as rich neighbours this would be very apparent on English. Not to mention the latin effect which will be strong.
I completely agree. I think where it's going to diverge significantly from OTL is in administrative words. In proto-Middle English (only thing I can think of calling transitionary 11-13th century English ) a lot of Germanic words remain- what declined was the use of declension, gender and Germanic words for administration. Because the gov't was Norman French, we adopted a lot of their words to describe things. So instead of "empire", "reign" and "tax" we might have rīce, rīcsung and scat (for a time). However I imagine just as IOTL, French words would be imported and corrupted en masse during this time as well- so you might very well have something like "excise" imported via. Dutch (via. French) to mean tax in the end anyway.
The main difference I think will ultimately come up in that the language will not be forced on the English necessarily. Without the Normans to entrench a lot of French words in basic English vocabulary, much of common speech is going to remain primarily Germanic. Most notably, it's probably going to stay that way largely in legal language (where IOTL it's very French/Latin influenced). On the other hand, the English should still do a ton of borrowing from Latin, French, the Low Countries and Scandanavia- particularly in emerging areas of speech such as foreign policy, probably a lot in religion, etc. I would still expect French words to make up a significant part of our vocabulary but like German and Dutch, they've largely been imported as administrative descriptors and first "translated" via. English declension rules (which I would surmise hold together better in an Anglo-Saxon England).
The big thing I wonder- what of the north?
Would England reconquer the lands recently lost to the Scots?
What of Straithclyde?
I would well see Scotland kept as a small celtic highland nation with the English reasserting themselves in the low lands.
Would this have long term demographic impacts? Other threads seem to make it sound as if the Anglo-Saxons would ultimately reconquest some of the lands lost to the Scots, but that by and large their rulers (House of Wessex) weren't incredibly interested in uniting the isles or anything.