I can speak to this. I focus a lot of my free time on the usability of free/open source software, and a few years ago, I looked into the usability of GIMP. I didn't do a full usability test, but conducted surveys of different people who used GIMP, versus Photoshop. What I found is that a person's perception of GIMP's usability depends on their familiarity with Photoshop:
People who used Photoshop all the time complained that GIMP had poor usability. This seemed to be because people knew they way around Photoshop very well, and were put off when the same functions were not accessed via the same menu path, or were called something slightly different. So they felt lost, like GIMP was broken even though they recognized it was very powerful.
People who used Photoshop occasionally, but not all the time commented that GIMP had good usability. These users understood the basic concepts behind Photoshop, such as layers and channels and plugins and tools, and could transfer that knowledge easily to GIMP. Because they didn't have a "muscle memory" of Photoshop, these users weren't put off by having the same functionality located elsewhere or with a slightly different name, because they probably didn't remember exactly what the feature was called in Photoshop, or in what exact menu it was located.
People who did not use Photoshop said that GIMP had poor usability. That seemed to be because these users didn't understand the basic concepts of Photoshop, about layers or tools or filters or what a "raster" image was, and felt overwhelmed by GIMP. If these users did any image manipulation at all, they used a simple "Paint" program like Microsoft Paint.
From your comment, it sounds like you use Photoshop quite often. So I'm not surprised you find GIMP has poor usability.