I think a population in the hundreds or even thousands is entirely plausible. Climactically, Kerguelen is perfectly habitable when you compare it to places like Iceland, parts of Alaska, or the Falklands. It’s just very remote.
So let’s imagine the settlement is initially founded as a resupply...