I am leaning against it. As far as I know, Quakers don't really have any distinctive clothing like yarmulkes or other Jewish religious clothing. There's just a general trend towards simplicity.
Also: Those hats. You might not wear them now, but if I were you I would start doing it again. Any excuse to wear such nice hats.
In a heavily nationalist society, you could get anger against Quakers over their pacifism. Say there's a bloody war, and the country in question allows Quakers to serve as conscientious objectors. Quakers could easily be seen as 'not doing their fair share' or something like that, and that could inspire resentment. They could be persecuted for similar reasons to the way the Jehovah's Witnesses were treated in Nazi Germany. Given how much Quakers get involved in social-justice activism, they could possibly be made into some kind of 'other'.
The more militaristic and unjust a society is, the less Quakers will be liked? Hmm. Maybe a more unpleasant US or UK could be the POD I'm looking for, then.
Quakers also are wealthier than the general population (IIRC we're the single wealthiest per capita religious denomination). You could do something with that.
Isn't that similar to the situation of many Jews in Europe, that many tended to be wealthy due to their ancestors not having been allowed to own land and thus becoming merchants, bankers and so on and so forth?
Do you think it's a pure coincidence that Quakers are relatively wealthy, or does it have anything to do with Quaker beliefs? For example, perhaps Quakers don't tend to spend money on expensive luxuries due to the testimony of simplicity?
(Also, out of curiosity: Are you a believing Quaker, or do you just happen to be born into a Quaker family? Also, what kind of Quaker? Hicksite? Beaconite? Wilburite? Gurneyite? I ask because I wonder how much I can mine you for information.

)