Another way of looking at it is that predictability is an important factor. Almost regardless of speed, someone behaving unexpectedly can cause issues. Human drivers ironically tend to offer clues of their unpredictability, allowing other drivers to flag them for extra attention, while self driving cars probably does a lot less of that (stuff like weaving a bit, acting uncertain, blinking the wrong way etc.).
Of course, if all cars were self driving, the issue would vanish. I suppose it would also lessen as more and more self driving cars arrive on the roads, effectively training other drivers in how they behave.