it would make absolute sense
Now back up a second there and consider proprietary software. Imagine if you bought Microsoft Office for your company and a year later Microsoft comes along and starts threatening you with patent infringement lawsuits over your use of their patents. I can't imagine that a court would stand for that at all without at least a fine print "requires additional patent license" and even then there's fitness requirements that the court would have at least a little discussion on.
Why would it be different for something given to you for free with explicit permission to use and give away? If the patent holder wants to provide a reference implementation to licensors, they can do so with a license tied to their patent, but it should not be the GPL.