I may have misread your proposal before. I'm not sure if it works because if we take a reasonable "longest side" figure like 600px, someone could still enter a 600x600 square flag, and that would fit the rule but is clearly too large. That's why I thought it might be better to constrain both sides, but you're right that it borders on being too complex that way. Or maybe that's just because of the way I worded it?
Whoops, late reply. Yeah, I can see your point there. I guess I was kind of assuming good faith though that we wouldn't have anybody abusing it though. I mean, I guess we could have an exception rule for square flags, but that indeed might be overcomplicating it. We can't exactly ban square flags either, heh.
Maybe we just need to think outside the box (I had a complicated box metaphor worked up but it was too crazy). Instead of saying X by Y, let's just say Long Side = X, Short Side = Y, Square flags are limited to Y by Y. It would allow people to make whatever shape they want, and while square flags wouldn't take full advantage of the longest side, I think this might be a fair compromise.