I'm a die-hard fluxbox user. Every once in awhile I try the others to see if they've gotten any better, but so far I've always gone back. What I need is simply a customizable menu when you right-click on the desktop, a simple way to map keystrokes to actions like launching apps, changing brightness etc., and a system tray with icons for power, network, and volume. Fluxbox doesn't come with applets for the system tray, but you can use the XFCE, Gnome or other stand-alone applets, and pick and choose the ones you want. Fluxbox is less likely to be suitable for you "out of the box" but I'd say if you spend some time with it, it's more likely you can get it working exactly how you want and eliminate all sources of irritation. And once you're done tweaking it, keep your configs forever and it's not going to change out from under you, so it's time well spent in my opinion. It surprises me how much of GUI design is spent making sure a user's first hour using it is easy, not much on making sure it's efficient for long-term use.
I don't use the panel so can't comment on that, but Fluxbox is also very modular, so you can use the panel that comes with it or use Tint2 or Cairo or whatever else you like in its place. I like that modular approach that the big desktops have gotten away from.
I suspect a tiling wm like awm would be more efficient for me after some getting used to, but for now flux just works and I love it.