Baltimore and DC are and have always been pretty much the same size, so otl kinda fits.
DC got big not only because the federal infastructure is huge (Capitol, white house, fuckton of museums and monuments) but also because it was far enough away from Baltimore that it served as the city for it's own area. Ottawa (desipte being about the same distance) never was able to do this because Montreal sucked up all the development instead, due to the area having less overall population and density (compared to the mid Atlantic which is the most densely populated part of the u.s.)
So if the capital is at Havre de grace or closer to baltimore the (much smaller) federal district could end up only as a campus for the fed's administration, about the size of wilmington, delaware, and Baltimore be much larger, parraleling Philadelphia in size.