The serialized shows are a double edged sword. They might encourage me to stick with a series, but they also discourage me from starting one, especially if I don't start until Episode #5.
Indeed... and as a corollary, if episode #1 of a season totally turns me off or I decide that it's crap, I may just decide to not bother with the rest of the season. If I change my mind later and decide maybe to give it another shot, I'd immediately think 'why bother? I'd have to catch up first, and I'm not really sure if it's worth the time to do so.'
Best example I can think of is The Walking Dead, when they killed off a character at the end of last season that a huge chunk of viewers really liked, and they did it with a megaton of gratuitous violence. I'm willing to bet that AMC lost at least 10-15% of their viewership for the show right then and there, and most of them probably haven't come back.