One particle doesn't interfere with itself, and can't because the interference pattern is seen in the density of collisions over an area.
As many of these single dots build up, they tend to cluster around an interference pattern - as if some particles went through one slit, and some particles went through the other slit.
Not quite--and that's really the key element of this whole thing: the particle somehow DOES interfere with itself, because the interference pattern that builds up, just one particle / one dot at a time is DIFFERENT than what you'd get if each particle only went through one hole. Imagine you're up on a ladder, dropping beanbags through a plank with two slits in it (you can cover those slits if you want), and they form a pile on the ground below. If the beanbags can only go through one slit, the pile you get on the ground is a nice mound. If you open up BOTH slits, then what you expect is TWO mounds. If the slits are close enough together, you expect those mounds to overlap, with the height at each spot being AT LEAST AS HIGH as the height you'd see dropping the beanbags through just one hole.
But instead, what you see in the double-slit experiment is that, in between the two mounts, you get spots where there are FEWER beanbags than you'd get dropping them through just one hole. Somehow, instead of getting that 1+1=2, you're finding that 1+1=0. The beanbags are all still there--it's not like they're cancelling each other out.. they're just not all where you'd expect them.
The ONLY WAY to explain this (that we've found so far) is if each beanbag, which, again, you're dropping one at a time, somehow goes through BOTH slits and INTERFERES WITH ITSELF. This is where the idea of wave-particle duality comes in, because the patterns that you see (with valleys where there should be ridges) are similar to what you'd see with water waves or sound waves (sound waves can cancel each other out--that's the whole premise behind noise-cancelling headphones).
So then why don't we just say that photons (and beanbags) are waves and not particles at all? Well, because classical waves aren't "quantal," meaning you can't divide sound waves into discrete, indivisible components. You can have one "particle" of light (a photon). There's no corresponding discrete element of sound. So we say that they're particles after all, and simply adjust our thinking regarding just what a particle is and how one behaves.