Maybe since they are smaller and higher speed they can escape easier from the furthest away port. Smaller ships can land at smaller quays. The British were mining the area and destroyers can go over or around the fields easier, something practical like that maybe.
How would you like to be one of those soldiers on the deck on one of those destroyers all the way up (would they have been on deck or below)?
they were below deck - the seas were just too rough.
The smaller ships didn't have the endurance needed in the first place, that's why they were still refueling some 36 hours later.