Yeah, I'm aware of the running joke about the Chinese having invented everything. But in this case it's barely an exaggeration: the idea of filling a paper sac with hot air to make it fly has been around in China since the Three Kingdoms period, and military strongman Zhuge Liang (181–234 CE) usually gets credit for it. Of course, if you listen to Joseph Needham, then the idea actually dates from the late Warring Kingdoms period, almost half a millennium earlier.
These contraptions are known as Kongming lanterns, after Zhuge Liang's courtesy name, and can be the size of a human being. Presumably, they can lift a few dozen grams, as they are fitted with a candle or other incandescent material to keep the air hot. How difficult can it be to imagine, one day, some Chinese figuring that if a small lantern can lift a candle, a bigger lantern could lift, say, a small animal? And what then of a really big one--could it perchance lift a man?
So, sometime between the Three Kingdoms period and, say, the early Tang dynasty, full-flegded hot air balloons are developed. What next?
