Why do you allow password logins for SSH? Why the hell don't you have port knocking enabled for SSH?
Because the odds of cracking a strong password via SSH on a server with root logins disabled and a 3 attempt before a long temporary ban are so infinitesimally small that I'm more worried about the sun not rising tomorrow.
There's nothing inherently insecure about a password based login providing the passwords are sent over an encrypted channel, the passwords are strong, and neither machine is compromised, and that even before any talk of login limits.
That belief is common, and incorrect.
Bear with me and I'll demonstrate why that's the case, and explain why it's a common belief.
It's complex, and most people won't invest the time to consider and understand the reasons why it's incorrect.
Many people believe it because many people believe it (a form of circular authority).
And because they over-invest in an emotionaly based opinion (which they won't test because they worry about where the results will lead - to further testing of their beliefs). It's foolish because they expend more time and energy defending their beliefs than it takes to test them, and change them when they prove wrong.
Note that those people get very angry when that emotional investment in "gut instinct" is challenged. Hence the title of this thread (it's a form of "la la la I can't hear you")
If it was correct there wouldn't be so many instances of attackers gaining access to password protected SSH.
The most common attacks (which your fail2ban logs will be full of) try a list of common passwords. That does have a low probability of success - but there are a lot of boxes allowing password SSH access. Sadly, most of those allow root logins.
fail2ban does limit attempts - by source. IPv4 only. Unless you enforce strong complex passwords dictionary attacks will succeed in a large number of cases.
Most admins do not enforce strong passwords. Key based authentication enforces the equivalent of very strong passwords - far stronger than the login buffer allows.
There's no need to guess - you can test the complexity of user passwords. Or simply enforce it by setting the default to key authentication only.
The default of encryption for a SSH connection is not relevant - it's the default. It doesn't render password access more secure than key authentication. In short it's a red herring. (there is no "providing" about it)
tl;dr your belief in the security of fail2ban is misplaced. An attacker can make unlimited attempts by just changing proxy every 3 attempts. There is a high-probability of them doing that. The more desirable access to your box, the higher the probabability.
The vast majority of unauthorised attempts you'll see in your logs are brain-dead bots looking for low-hanging fruit. Those attacks will only get smarter. Best to stay ahead of that curve.
Good security is about reducing risk as much as possible. Doing so is not about whether you "feel" it's secure, it's about whether it can be mathematically demonstrated it's safe. The point of the lead summary, and the article it references, is that it's been well demonstrated that poor passwords lead to poor security. The same big-brains that proved that also strongly recommend the banning of remote logins as root, and passwords for SSH access. There's a reason why the default SSH setting for all recent Linux and BSDs does just that - it's not to make your life hard, or the remote clients - it's to make life hard for attackers.
On the one hand as techs we want (proven) facts, on the other we are human and want to trust our intuitions, and we are lazy. It's that biased hand that is the cause of most security failures.