That is correct, and normal.
You do not have an inherent right of ownership over his code. His code is a derivative work of yours, and your merged codebase would be a derivative work of his.
By modifying your code, he has created a new work, which is permitted under the terms of the earlier-version GPL. He then releases his new complete work (which includes your parts). That's also permitted under the earlier-version GPL, as long as he releases the source code. When someone wants to use the new work, they're required to have a license to it. He offers the user a later version of the GPL for his new work. Your parts are now covered by the earlier version if the user gets them from you directly, or the later version if acquired from the modifier.
In your case of merging back changes, his changes are part of his new work, which is covered in its entirety by the later license, regardless of the fact that your code is included. It's no different from merging in a bit of MIT-licensed code or even a proprietary routine: you have to comply with the terms of the license. For a later-version GPL, that means your new combined derivative work would have to be compatible with the later-version GPL.
In short, the core idea is that your codebase is a new work every time you change and republish it. To "accept changes back into my codebase" is actually creating a new derivative work, and it must be properly licensed as any other derivative work must be, regardless of your involvement in the work's ancestry.