Yes, but H.265 is a video compression format. I dont't think they are going to take videos with the OSIRIS camera (or any of the other instruments) nor transmit video data back to earth.
Anyway, after some googling one finds that the most advanced image compression formats are based on some form of wavelet transform.
Examples would be JPEG 2000, JPEG LS and ICER. All of them predate 2004 but have seen some improvement over the last years. ICER was used on the mars exploration rovers, so it could be they use something similar on the rosetta probe.
Their compression rates are all very similar and differ only in their compational efficiency. Some comparisons can be found in this paper and here (homepage of the MRP Format (Minimum-Rate Predictors), which marginally outperforms JPEG formats but only on 8bit grayscale images).
The latter also compares against JPEG XR or HD Photo from Microsoft which was released in 2009 but seems to perform worse than JPEG-LS.
I don't see any advances in still image compression since 2004 that I would describe as huge, but that view is just based on a quick internet research. I'd be happy to change my mind on this.
Maybe NASA or ESA have been working on something much better and I just missed it (I didn't check patent databases).