I'm not young but I tried to get into kernel driver development (out of necessity, not so much itch-scratching) and was thoroughly frustrated by:
1) Lack of useful documentation
2) Constant churn in the kernel
I wanted to allocate memory in a driver. Lots of memory (64MB on an embedded system). I read all of LDD (latest revision) and tried everything: memory mapping, kalloc, vmalloc, ioremap, etc. etc. Nothing worked, and the book was so i386-centric that on my ARM system the explanations didn't really make sense. Eventually (we're talking several DAYS) I made something work using mmap.
And don't get me started on interrupts. Where do they even start? Where's the interrupt handler? WTF is the difference between a hard interrupt, a soft interrupt, and a virtual interrupt? I spent DAYS trying to figure out the exact sequence of code that represents travel from the hardware event to the final driver code (because it was necessary in this case) and had to give up. Interrupts are just magic.
And yes, I can read source (and I did).
As for churn, it seems like every week there's a new scheduler, a new memory allocation strategy, a new filesystem, etc. etc. QUIT IT! Is it really improving things that much? Every major system that gets replaced in the kernel obsoletes a bunch of code that now has to be rewritten to keep up. Worse, it makes whatever documentation there is (e.g., LDD) out of date. It was no fun reading parts of LDD only to look at the actual 2.6.24 source and see that parts of the book were already out-of-date (I can't remember what now).
If the kernel devs are only interested in itch-scratching then their numbers will continue to dwindle and Linux will fade in favor of a system where people actually do hard work: proper documentation.