Exactly. This is why I don't use any AV product at all. As long as you're reasonably careful not to download and install unknown programs, there's no way to justify incurring a huge performance hit on a daily basis. For example, I once "fixed" a friend's PC in which she had installed two AV programs - Avira and McAfee - for additional protection and security as I heard. File copying had dropped to something like 150Kb/sec between two hard drives because both anti-viruses were scanning it. Disabling one increased the speed to about 1.5Mb/Sec. Disabling both improved it to about 6Mb/Sec (figures according to rough recollection, to be taken with a pinch of salt). I eventually left one on since she wasn't an experienced user and needed some anti-virus program, "just-in-case".
But experiences like these over the years have convinced me that the wisdom about adjusting your process is far more valid than having an army of products. I haven't had a single virus infection for as long as I can recall and if I did, that was because I'd been careless and run some program off the net without finding out what it was. Also, I don't think AV programs offer any meaningful protection against things like browser flaws. If someone decides to exploit say a buffer overflow vulnerability in your browser and you simultaneously decide to browse to that very site which does so, well, so sad, too bad. Might as well wait for the browser vendor to release a patch which fixes that flaw and use a more secure browser like Chrome to browse dodgy sites, rather than pray an ineffective AV magically detects it with its "heuristics". Most often, all that DLL injection and the like result in an unstable browser, rather than providing any real protection.
Having said all that, I do see the utility in being able to do an occasional on-demand scan on an executable. I also see why AV vendors are going for the nanny philosophy to deal with the armies of inexperienced users who have no idea about the "process" behind security. But for those with a reasonable idea of it, it's probably better to suffer the rare virus infection than endure a crawling system on a daily basis thanks to some overzealous AV product.