Yes, it is only the traders that gain benefit--they are the ones participating in the transactions. Fast traders (HFTs), slow traders (e.g., retirement investors), and traders with all time frames in between assume the risks and accumulate the profits and losses.
No matter how slowly the transactions move, the participants with more current information have an advantage. This is true whether you're talking about making a fast profit using stocks or investing (speculating?) in real estate because you think it will be valuable decades in the future. Information always matters, and you can't eliminate the advantage of information and still have a free (or free-ish) market.
Of course, if you're interested in making use of the money owned by market participants to achieve social/economic/political goals, you may have a legitimate point. But at that point, we're not talking about investing, we're talking about taxation. I may support that, but let's be clear about what we're doing. Until then, the people who own the money get to use it for their own profit.