IIRC, they nearly lost an X-15 because, despite 'heat shielding', it caught a nasty case of 'heat soak' on a 'speed run' and suffered a lot of internal damage. Pilot was lucky to get it down safely, but it never flew again...
That may be the one that, um, bent like a 'nana upon landing.
You could not 'grow' the basic X-15 design much beyond strap-on tanks. Back to the drawing board, ~ 50% bigger to carry the weight of spaced heat-shield, probably with boil-off coolant, and you've just got a bigger, heavier X-15 with much the same issues due higher loading. Rinse & repeat, gets further from the X-15's sub-orbital sweet-spot...
IIRC, the math gets better again when you reach something the size of the XB71, as area rises as the square, but volume as the cube...
Happily, Neil Armstrong made 'Astronaut' in an X-15, put his test-pilot skills to taming the Lunar Module and the rest is history.