First of all, Caller ID is trivially easy to fake, and the scammers all do it. For now, most of them pick random or fake numbers to avoid getting blacklisted, but if whitelists were common, they'd start forging real numbers to get through.
But many people (ok, me, at least) get lots of calls from numbers I don't recognize, and robocalls that I want that might not come from the number I recognize for somebody. Most of the robocalls are the pharmacy saying I've got something to pick up, or the dentist's office with a reminder about an appointment, or that kind of thing, and the calls from humans might be from some doctor my wife is going to or some business we were trying to reach that has different numbers for outgoing calls than incoming (like the painter calling from his cellphone instead of his office, or a big business calling from their call center or local office instead of their toll-free number.)
And yes, I could just let the answering machine pick up, and you can too. Some of the robocallers' robots do a better job of dealing with that than others.