They don't seem to have been numerically many(plus many emigrated afterwards anyway), also it was not lasting contact, if anything it was very brief and it would weird if Scandinavian actually influenced English that much.Because there was a fair bit of settlement by the Danish Norse - Anglo Danes were an actual thing - and lasting contact with a language that has similar words but different inflections has an affect. Look at Afrikaans for example.