Exactly right. Top Gear once had a race between a Prius and a BMW M3. The Prius had to drive around the track as fast as it possibly could, and the BMW just had to keep up (which obviously took little or no effort). As you might expect, fuel consumption for the Prius was a lot higher than for the BMW.
Fuel economy figures are measured with very small accellerations. In every day driving, we tend to accelerate a bit more. For a big engine, this requires little or no extra effort from the car, and fuel consumption is only a little more. But for a small engine, even moderate accelerations consume a lot more fuel since it brings the engine a lot closer to its maximum power. Nothing unexpected here.