Yes, the concern is real and common. The antidote is building stuff. A bunch of stuff. The more stuff you build, the more likely it is you'll have to get "dirty" with the underlying guts of it, the more you'll know, and the more valuable you'll be.
Of the many garbage collectors Java offers, have you ever used anything but the default? Do you have any idea what the trade-offs are and when you might want to use another one?
Have you ever profiled any Java code?
What do you know about Java byte code?
Ever SWIG wrap anything into Java?
Ever used Java serialization? Do you know what's wrong with it and why you wouldn't want to use it?
Custom class loaders?
My adivce: go learn everything there is to know about "something", it doesn't really matter what (maybe you want be the worlds foremost expert on malloc). In the process of gaining a very, very deep understanding of that 1 specific sliver; you're going to also learn about a ton of other stuff on the way.