I disagree. It is of course not a matter of just changing the port and then be done with it. In addition to all the other security measures (fail2ban, iptables-woodo, key-based auth, etc) moving your sshd (or any service) to a non standard port is both an advisable and mildly effective security enhancing activity.
It might be that some bots are taking the time to scan the entire port space, but the number probes that bother to try your high hanging ports are negligible compared to the never ending avalanche of zombies that tries to squeeze through port 22 (or any other standard port for any other popular service for that matter.)
There is also the matter of bugs in the software. Though sshd is one of the most vetted, and hardened pieces of code out there it only takes one bug to cause a disaster (which has happened before). Attacks never get worse, they only get better! In the case of a 0-day in sshd that allows for arbitrary code execution I sure as hell don't want to be amongst the masses running on port 22.
In conclusion: If your only security measure is to move your services to non-standard ports, you might as well not bother. But your security level is a shade of gray, and using non-standard ports moves the slider a tad in the right direction.