There *was* a Brown Scare in OTL:
"Against such people, Roosevelt and other politicians helped to foment a 'Brown Scare" before and after Pearl Harbor.77 Although directed at different targets and practiced on a lesser scale, it resembled the Red Scares at the close of World War I and after World War II in its sloppy regard for civil liberties, affinity for conspiracy theories, manipulation of concerns for national security, and institutional machinery. Indeed, the Brown Scare was another act in the rehearsal for the Cold War, going far to generate the mentality and the apparatus mobilized against the left after World War II." Michael S. Sherry, *In the Shadow of War: The United States Since the 1930s, * p. 51
https://books.google.com/books?id=3VkST5a9Mo8C&pg=PA51
"Dark tales of fascist 'fifth columns' had abounded in progressive circles during the war, and even supposed civil libertarians like the Nation's Freda Kirchwey agreed that the 'American fascist press' had to be suppressed." Thomas W. Devine, *Henry Wallace's 1948 Presidential Campaign and the Future of Postwar Liberalism,* p. 42.
https://books.google.com/books?id=oOaJCpyZFW8C&pg=PA42
Wallace didn't go quite so far as to accuse all enemies of FDR of being fascists but he did say that "The name Roosevelt is cursed only by Germans, Japs and certain American troglodytes."
http://www.ibiblio.org/pha/policy/1944/1944-07-20d.html