you just made two assumptions that directly contradict each other. Conservation of momentum follows from Newton's laws of motion.
Newton's laws also fall out of GR at non-relativistic velocities. The inventors say their device is consistent with GR.
If you assume momentum is not conserved
I'm not assuming it, if the device works as advertised, it meets the standard criteria for "reactionless", and thus it can be used to create a free energy machine.
For it not to be able to be used as a free energy machine, it must break some other tenet of physics (such as mass/inertia equivalence, non-locality, no universal frame of reference, etc.) The explanations by the inventors have mostly involved ad-hoc efforts to let it break some tenet, but somehow in a way that doesn't actually break it. "It's exchanging momentum with the inertial state of the background universe, but not by violating non-locality", "it's interacting with virtual particles, but in a way that doesn't violate CoE", "it can't be used to accelerate, only to hover, but totally doesn't violate equivalence," and so on.
There is no difference between "virtual" particles and "real" particles. They obey exactly the same physics.
Virtual particles work in a different way to regular particles, because at the beginning of a reaction the virtual particles always have net zero initial velocity (net zero momentum) WRT the device, regardless of the acceleration of the device; that means the device will have a constant force-per-watt equation regardless of velocity. With regular particles, if you accelerate, you change their velocity relative to you, that changes the force-per-watt equation. The two are not analogous.
(Read my explanation using your "swimmer" analogy.)
And this is the same regardless of what explanation you have for the "fabric" of space, virtual particles, quantum foam, M-theory. They all must have this zero-net-velocity component in order to be consistent with, not just theoretical, but observed general relativity.
Re: Virtual particles becoming "real".
However, they steal that energy from something else. For example, when a virtual particle pair is split by the event horizon, the lost partner reduces the mass of the black hole. The net energy must be zero. And it must be uniform, there's no propulsive effect.
[In the case of a "drive", if you push against one partner in a virtual pair, you will be pulled back by the equal-and-opposite virtual partner. The net is zero. Even with Casimir, both sides of the gap are pulled equally. No net force.]
A particle/antiparticle pair initially has zero momentum, but they absolutely can interact with other particles, and that interaction can impart momentum to them.
But not net momentum. Not a force in a single direction.
--
Aside: Doesn't it make you curious that the more careful (and skeptical) the experimenters are with their test rigs, the smaller the measured effect gets?
Doesn't it make you curious that the supposed "null" device, designed by the same people not to produce a force, somehow produced a force on the same test rig? Isn't that what a null hypothesis is for? Doesn't Occam's razor suggest that the experimental protocol is missing an unusual but mundane effect and not a revolutionary unidirectional reactionless propulsion system?
Frankly the whole thing screams of a very technical version of those old perpetual motion devices made from magnets and moving weights and levers arranged on a wheel, which the inventor insists allows one side to experience more force than the other. No matter how many people explain why it can't work, they concoct increasingly elaborate explanations for how it must work. Look! It's turning! Oh it stopped. Well that just means it needs stronger magnets and axles with less friction. (Or in the case of EmDrive, superconducting cavities and higher-Q designs.) Or, in mathematics, like someone encountering the Monty Hall three door problem for the first time.