I don't see what is potentially damaging about people knowing you signed it anyway. It doesn't make you a gay-bashing red-necked evil conservative. In fact, it could simply mean that you prefer a direct vote to a vote of representatives.
"Think of your favorite song/artist. If GW Bush wanted to use that song as his campaign song, wouldn't you feel as though the artist were somehow endorsing that campaign?"
I totally agree. When I first read about Charles Manson, my very first thought was: "How could the Beatles be endorsing that psycho?"
Take everything in stride. Trample anyone who gets in your way.