Owning a plushie and a couple of figures, watching the show, visiting internet forums would imply "big fan" to most people. None of those and even all of them together implies any kind of inteference with "life" as you put it.
None of these things are a bad thing to do with your life if you are an eight year old girl.
If you're a 40 year old man, sorry, it's just weird. I know at this point the slashdot libertarians will bring out the "you should be free to do anything you like as long as it doesn't harm others" card.
Fine, I didn't say it should be illegal. I just think that, taken seriously and not as a camp joke, being a brony is going to make people think you are weird. That's your right, but it is ridiculous to pretend you don't see why it's weird.