It was definitely a big problem because if some of the kids have them, the ones that don't are excluded.
This was a huge issue for my oldest in middle school. She was in a very small minority that did not have a phone. It was made worse by the fact that we don't have a landline, so if friends wanted to call her, they were calling either my wife or my phone (and the kids calling were not a huge fan of that, so they didn't call much). In addition to the social aspects, it caused a few issues with extracurriculars where one of us wasn't present and things like pickup times were fuzzy rather than fixed.
We stuck to our guns and didn't let her have a phone until 14 but I'd be a liar if I said it didn't have negative aspects in addition to positive ones. Our youngest might end up with a phone closer to 12 as a result, though that remains to be seen.