x264 does not use the GPU, the program just does not support it. I know that ATI once produced an Avivo H.264 encoder, but that one was of highly questionable quality. Of course, these days they might have made a better one. Have you tried comparing the speed and resulting video to x264 on veryfast or ultrafast presets?

This might not matter too much. Using the GPU to assist in video encoding might be less of a good idea than many people think. Many complex procedures during encoding are not all that suited for parallelization. Take entropy coding for example. You probably have most of a chance for doing anything useful with motion estimation, but that's still quite hard. A bunch of people have worked on adding GPU acceleration to x264, as part of their thesis. There wasn't any real success. Most of them failed to make it actually useful, since cache considerations and the like prevented them from using nicer algorithms than exhaustive search.

As for existing encoders, like Badaboom, they mostly aren't all that fast or good. You can probably beat them with x264 on fast settings and still get similar or even better quality.