I use variations of the same passwords, so I have memorized many more passwords than phone numbers. Usually I can even remember some infrequently-used passwords - based on using variations. However, the phone numbers that I have memorized have not changed for years, for the most part. At our company, it's actually pretty rare for people to forget their frequently-used passwords. However, I have no idea how often people forget phone numbers, since it is trivial to just look them up.
If all passwords followed the same rules, without requiring frequent resets, it probably would be relatively easy for people to remember a few passwords. But, keep in mind that 3 of the 10 phone number digits (the area code) is relatively meaningless, in terms of the need to memorize it. So, for the most part, people are only remembering 7 numbers - not exactly a large number of possible values, compared to the possibilities for passwords.