Nice anecdote there
The Romans did have silk, imported from China - in fact Roman officials worried that Rome's coffers were being emptied as rich Romans bought up silk and trade was only flowing in one direction.
They did not know the secret of its origin, though, so getting enough to make balloons would have been difficult (unless they do find out in TTL).
Which reminds me; IIRC, the OTL Byzantines learnt the secret of silk production during the 6th century.