I don't know anyone that has self-identified as a swinger. Rather there's a segment of people who use terms like poly or open for poor behavior. (By the way, I'm not saying that ethical non-monogamy isn't possible, though I do think it's likely harder than traditional monogamy, so I don't think it's likely to be the panacea some seem to think it could be.)

Your point that women faced and face harsher consequences for non-monogamy than men is valid. But men had to be sleeping with someone else. Infidelity and non-monogamy is far more balanced between the genders than anyone still seems to want to admit.