(Trigger warning: Physics teacher here!)
Actually, momentum is NOT conserved for the *shooter*. He is bracing himself against something (his feet on the ground, usually), so there is an external force acting on him. Thus, this particular argument fails.
For the victim, if he is shot unawares (so he doesn't brace himself), then conservation of momentum *does* apply. I calculate that he will experience a force to his chest of about 35 newtons (~8 pounds). That's not much, and won't accelerate him much.
However--it may well knock him down. The reason is, the force to his chest will cause a torque on him, which will cause him to rotate down to the ground. If we assume he rotates about his feet, and treat him as a solid cylinder (reasonable approximation) then I get an angular acceleration of about 0.4 radians per second per second (22 degrees per second per second).
That only lasts while the bullet is in contact with him, of course; after that, the victim has a gravitational torque on him, with a corresponding angular acceleration (I estimate) of 9 rad/s/s.
If someone wants to check my work, I'll supply the numbers and things I used. I might well have made a mistake or missed something.