Indeed all good points. Here's the kicker though... All laws are a restriction on the citizens that would choose to break them.
"Rights of others" is very open ended, and can be applied to literally ANY desire by ANY group of people.
We tell people they can't walk around down town naked, but who would it really hurt?
Farmers go to jail if they sleep with their sheep, but who are we to tell him he can't?
Why can we tell people they're not allowed to smoke crack in the privacy of their own homes?
How dare somebody tell me and my wife we can't have sex in the movie theator... We'll keep it quiet.
See what I'm getting at? Free society has the right to put certain limits down. While nobody in their right mind would argue with womens suffrage, or equality for blacks, it is not unreasonable to sit back and ask ourselves how much we're willing to accept as a society.
For a matter of record, I know these views are unpopular here. I've been called some pretty vile (all untrue) names for them. The gays that I know, and the gay that I'm related to all know where I stand and why. In real life, I've NEVER, EVER been accused of hatred, because it so obviously far from the truth. Even though I don't agree with the lifestyle, I disagree with it in the exact same fashion as I disagree with any lifestyle I find to be wrong. I understand that we all have our challenges in life. We all have our weaknesses. We all do things that are wrong. I don't consider myself better, and I don't consider them worse for it. I'm certainly not on any sort of pace to become the perfect human before I die.