I think it depends on the type of game. For FPS games, where you "are" the character, graphics really help with the feeling of immersion. For simpler RPG games, graphics are less important and the storyline becomes more important. For more complex RPG games, where exploration of the virtual world is big on the agenda (like RuneScape and WOW), graphics are a larger part of the experience than simple RPGs like Pokemon.
Then there are the strategy games, like Freeciv, which depend on one's own experience and playing style to deliver a pleasant gaming experience. Of course, we can't forget older games (e.g. Super Mario Bros, Need For Speed II, Pacman), in which the fun is in the familiarity. For someone born recently who didn't grow up in a Super Mario Bros household, the original Super Mario might seem to suck because of its crappy graphics. But someone who grew up with Super Mario Bros would still enjoy it today due to that feeling of nostalgia and familiarity. Personally, I still prefer the older Need For Speed games over the new ones. Because I grew up with them, I feel more comfortable with their familiar appearance and gameplay, despite their crappy graphics (compared to today's standards).