Close. There are several different hairs to split here:
If you write a program that implements the patent, you have simply transformed an English description of the patent into an [insert programming language of choice here] description of the patent. That, in and of itself, is not illegal any more than translating the patent to Spanish, French, or Esperanto would be.
If you compile the program (assuming you used a language that isn't interpreted) and distribute it, you've not done any more than you have in the previous example. In this case you're just translating the code into assembly language. However at this point, you'll usually get sued for infringing on the patient even though, as the OP states, the patient is only valid when the algorithm is carried out using a generic programmable computer.
If the OP is correct, you would only be infringing at the point that the program is loaded into memory and executed by the processor. If that's the case, the onus would have to be on the plaintiff to prove that at some point I ran that particular algorithm.
I use the Linux kernel. It might be such that the code that implements the patented algorithm has never actually executed on my system. If that is true, I haven't infringed in any way. I'm sure there are binaries on my hard drive that I've never used. If any of them implements a patented algorithm, I haven't infringed.