Here’s the thing-the ancients did improve the trireme. The added more oarsmen and created the quadrireme and quinquereme of Punic War fame. And they seem to have consistently felt the quinquereme was a superior warship across both time and place, as evidenced by the fact it was used both by traditional naval powers like Carthage, Rhodes, and Athens and land powers like Rome and Macedon. Moreover, all of those navies that used quadriremes and quinqueremes used them alongside triremes. So they clearly understood the capabilities of both kinds of ships, and none of them chose to abandon quinqueremes to return to triremes. To me, that is quite telling.
Now, there are certainly other things you can do to improve ancient triremes (and by extension heavier warships) by giving the ancients certain technological developments:
-A rudder. I think this is the most significant improvement we could make. Ancient Mediterranean ships were steered using a steerming oar, which was usually placed on the starboard side of the stern, although instances are known of the Romans placing the steering oar off of the stern. This system has a number of problems. Adopting a true rudder like the ones developed on Chinese ships in the first couple of centuries AD would be useful in that it is more effective and requires less effort on the part of the helmsman. Additionally, a rudder in this style would retain the ability of the steering oar to be lifted out of the water, which is necessary to allow beaching.
- Sailing rigs. Lateen sails have already been mentioned, and they were either invented or adopted in the Late Roman period so that seems like a pretty clear possibility. Related are other sail rigs which might offer advantages to the trireme. The junk rig comes to mind as one that might be useful to ships with a small compliment of sailors, as it is easier to use effectively with a small crew. I’m sure there are others, but I don’t know enough about sailing to comment more.
- Magnetic compass. I think the value of this is fairly self-explanatory.
The real issue here is that the trireme was already very advanced for the technology people of the time had. There aten’t a ton of things one can improve without simply turning it into a different type of ship.