Linux the OS certainly has had numerous real-world security problems that need to be addressed. I don't particularly care about the semantics of "Oh it's just a kernel!" because I could play the exact same game with Windows where Windows kernel vulnerabilities aren't super common either. Guess what: Linux and Windows both run the same web browsers these days, and that's a cross-platform security hole no matter who wrote the kernel.
Additionally, the biggest security hole I see now is Android due to the fact that it's damn near impossible to actually get upgraded software to fix the numerous holes.
However, Torvalds' direct responsibility is the kernel, so in this particular context I'm not going to give him too much grief. The Linux kernel does actually include extremely sophisticated mandatory access control systems like AppArmor, SELinux, etc. However... and this goes to his point... these systems are used sparingly because they are REALLY complex and lead to all kinds of usability issues for unsophisticated users (And "unsophisticated" here could easily mean a skilled Unix sysadmin with years of experience. These MAC systems are *not* considered "normal" in UNIX).
So basically: Yeah, Linux is not perfect. Nothing out there is perfect. However, the kernel actually does have a bunch of sophisticated security facilities. Maybe more work should go into making these sophisticated security features more accessible and useful to regular people.