This is really ASB territory where the past is concerned. As for the future, Rchard Posner has suggested a distinction between SSM and polygamy:
"But later in his opinion the chief justice remembers polygamy and suggests that if gay marriage is allowed, so must be polygamy. He ignores the fact that polygamy imposes real costs, by reducing the number of marriageable women. Suppose a society contains 100 men and 100 women, but the five wealthiest men have a total of 50 wives. That leaves 95 men to compete for only 50 marriageable women."
http://www.slate.com/articles/news_...issent_in_obergefell_is_heartless.single.html
Jonathan Rauch takes a similar line:
"Here's the problem with it: when a high-status man takes two wives (and one man taking many wives, or polygyny, is almost invariably the real-world pattern), a lower-status man gets no wife. If the high-status man takes three wives, two lower-status men get no wives. And so on.
"This competitive, zero-sum dynamic sets off a competition among high-status men to hoard marriage opportunities, which leaves lower-status men out in the cold. Those men, denied access to life's most stabilizing and civilizing institution, are unfairly disadvantaged and often turn to behaviors like crime and violence. The situation is not good for women, either, because it places them in competition with other wives and can reduce them all to satellites of the man."
http://www.politico.com/magazine/st...t-next-gay-marriage-119614.html#ixzz3gNqh3mQl
This argument has been criticized as "marriage socialism"--a preference for equality over liberty, which is hardly consistent with the quasi-libertarian views Posner and Rauch otherwise espouse:
http://balkin.blogspot.com/2015/06/richard-posner-marriage-socialist.html