Obviously the port change idea is unpopular because it's just obscurity, which != security.
However, most, of not all, of those attempts are probably from scripts rather than actual human attackers actively attempting to compromise your servers. The script kiddies rely on automation to rapidly find vulnerable hosts using easily-scripted vulnerabilities or brute force. The simple act of changing your ssh port will render those kinds of attacks useless against you.
I had a similar scenario where my logs were full of failed ssh attempts and by changing the ssh port, they all stopped. ALL OF THEM. I haven't had an unknown IP attempt to log in via SSH in around five years.
fail2ban, strong passwords, or whatever additional security you want to apply is great, and recommended, but a port change will solve more of the problem than you might think.