I do remember struggling with compiler bugs - in particular the C++ compiler we were using on OS/2 way back in the day suffered a few - but I was primarily a machine-code programmer in those days so an inspection of the compiled object code would tell me what was wrong pretty quickly; plus the compiler guys (I think it was Watford) were very responsive and would usually fix things up pretty quickly.
So no, the bugs that have most challenged me have always been concurrency-related - deadlocks, race conditions and the like. My earliest experience of this was probably on the Atari ST. I was coding a game in 68000 assembler, one of the early 3D-rendered golf games. Whilst the golfer (a sprite) was taking his swing I needed to pre-render the first 3D frame, so hooked up a hardware interrupt to run the renderer whilst the golfer animation took place (or maybe it was the other way around; doesn't matter). Anyway it all worked nicely except, just occasionally, a rogue red pixel would appear in a random part of the screen. It took me about a month to figure out that there was one variable/memory location being read and written by both bits of code, with no mutex round it.
I guess it was good to learn a hard lesson about concurrency early in my programming career.