You might not want to be topless in public places, but if topless men don't bother you, topless women shouldn't either - especially if you are a woman!
The tendency for many people to judge and demean women based on their choice of attire is a type of subjugation, it keeps women down, which keeps people down. It's just one more way that we separate ourselves from our neighbors, and the attitude that a woman's style of dress makes them less of a person and more of an object IS harmful to ALL of us.
If you haven't already, visit the Topfree Equal Rights Association for news concerning the movement, and plenty of great information for why allowing men to be topless in public while restricting women is hurtful and dangerous to our society.
Don't forget that in many great civilizations of the past, public partial nudity was normal, and full nudity was freely accepted. Clothing was donned to show wealth, class, or status and protect from the elements; modesty is a modern concept.