Looking back at my trip history, I've been out roughly once a week most months with some gaps when the weather was particularly bad or things got too busy at work to get out for lunch. It's enough to be a nice change of pace for staying in the office.
Cold isn't too bad. I keep an old ski helmet at my desk and ride with the ear pads on in the winter & off in the summer. Between that, a coat, a balaclava & a good pair of gloves it's not too bad when it's cold -- keeping face & hands covered is probably the most important part. The sidewalks & streets are usually cleared of snow / ice well enough to manage.
Summer does get pretty hot (90's occasionally) -- when I rode last summer on hotter days, I needed to bring a towel to dry off & an extra shirt to change into.
I tend not to ride much in the rain -- tried it a few times, but found that while a Goretex jacket keeps me dry while walking, my legs are extended far enough forward on a bike that my pants just get soaked.