There is no reason to think we'd detect them if they are far enough.
Any galaxy which has every star in it surrounded by a Dyson sphere would become one of the strangest galaxies in the universe. Same thing with supermassive black holes being tapped for energy, they'd look utterly strange, and while it's obviously totally theoretical, there's no reason to think that many usual handwaves for why aliens don't use Dyson spheres (i.e. "they tap into energy through some means we can't even theorise") wouldn't produce effects that would be unexplainable under known physics.
I think the evidence is that if they exist, spacefaring alien civilisations are rare and separated far enough in distance that the light from their activities has yet to reach us.
Sure, maybe in another 100 years we are all space Amish reproducing through robotic wombs, but we don't know that. The point is that exponential population growth is not an inevitability. A steady state seems equally probable if not more so. Which undercuts one of the key assumptions of the Drake model: which is that any civilization grows exponentially to fill up space.
Well, not Space Amish per se, although perhaps the model of O'Neill cylinders full of agricultural land being built everywhere and exponentially filling up will resemble that idea to far future humans or aliens. And for the reasons I said, it's easier for a family (or individual) to have one or two more kids than they otherwise would if they don't have to deal with the difficulties of pregnancy, which results in increased fertility in the society. Other factors which decrease fertility, like urbanisation combined with high housing prices, might be alleviated in a future human or alien society by increased remote work or economic reform.
Similarly, I mentioned groups like the Amish or other religious groups who encourage high fertility. There's no reason to assume aliens don't have such groups. Indeed, evolution
favours such groups based on their high fertility. All you need is a single prominent group in an alien species reproducing at higher than average fertility rate and they suddenly have a reason to exponentially expand.
Sure, there might be species who have static population forever, but this seems like it would take a lot of effort to maintain for questionable gain, at least for the spacefaring species relevant to this question. But space resources imply an era of plenty and one with the technology to have artificial wombs, cloning, and more longevity which increases population sizes. All you need is one group with 2.5 fertility over a few centuries.
This is way too much speculation on how future fertility rates are gonna go.
I don't think it's any more speculation than saying "fertility rates will forever stagnate by the 22nd century" since this is based on current trends using current paradigms. It cannot be understated how much of a game-changer artificial wombs would be for fertility. And one key factor in regards to the Fermi Paradox is that if even a single alien species--or even just a single group among them--decide to buck the trend and have higher fertility rates, then that species will expand.
This is an absolutely enormous amount of guessing at how future technology is gonna work like 500 years later when we aren't even good at guessing what's gonna happen in 50 years.
I think it's vague enough to be believable since it's all based on known physics, known proposals, and common desires such as the need for cheap energy (which any star readily provides but OB associations, globular clusters, etc. provide insane amounts of) and the prestige of colonising another star system. Although very challenging, to a spacefaring Kardashev 1 civilisation sending a colony ship to Alpha Centauri should be like us today sending a colony ship to Mars--barely practical and a poor use of resources but one which a lot of people (i.e. your future Elon Musks) would dream of and have no trouble recruiting tens of thousands of people. And like a Mars colony, the technology developed for it has plenty of uses elsewhere (the big one in this case being whatever you're using to accelerate/decelerate a ship from perhaps 0.05
c, be it antimatter/fusion engines, a massive solar-powered laser, etc).
What about communication times? It takes between 4 and 20 minutes for radio to reach Mars, depending on the positions of the planets in their orbits. For the moons of Jupiter, about an hour and a half. We do not see many sci-fi stories about "real" future space travel because they are too dull. Hollywood gives us warp speed, subspace, worm-holes, hyper-drive as short-cuts to create westerns in space.
Ironically before the telegraph (and in those small frontier towns without the telegraph) this sort of communication lag was quite similar in the "Wild West". I think the more questionable thing is why interstellar colonisation is assumed so often in science fiction when the diversity and sheer size of the Solar System makes for a perfectly compelling place to set all sorts of SF westerns, space opera, etc. Hell, you even have travel times comparable to the mid-late 19th century if you want to keep it within the orbit of Pluto or so since there's realistic rocket designs (fusion and antimatter) which could get you to Pluto in 2-3 months (albeit very inefficiently since this is torchship-tier stuff) and laser highways using solar and fusion-powered lasers to propel laser sail ships at a very high speed are perfectly plausible and of course a substitute for the classic railroad. The classic robber baron railroad boss is now the boss of the laser highway trying to extend his network to a profitable asteroid mining colony, for instance.
More on topic (and fitting for this forum instead of Future History or Chat), maybe a writer or two could find the idea of the Fermi Paradox interesting in the 80s or so and reject the idea of a setting like Star Trek full of FTL travel and aliens and set everything within the Solar System. Like an early version of
The Expanse, or perhaps a setting like
Mobile Suit Gundam (although it doesn't have to be military science fiction of course). Or perhaps the popularity of a setting like that leads to more rejection of the Fermi Paradox.