Lots of people wrote 'Latin swing' and 'Bossanova' music Stateside, no? And, yes, there were groupies and fans of that music.
Yes, I will agree that it was not as deep a penetration as Japanese today, but there was no TV and certainly no internet, which slows things down.
Also, I think people here are massively overestimating the penetration of otakus in the US.... Because we (or our friends) are the kinds of people who like that stuff. How many Trump supporters, for instance, do you suppose even know the words 'manga' and 'animé', and can distinguish between them?