Most speed limits are arbitrarily set and have no legitimate reason other than to generate revenue from speeding tickets.
Most speed limits are in residential areas, as most road miles are in residential areas - those speed limits are not set to generate speeding ticket revenue, or do you really think it would be safe to drive, say, 40-45 MPH down a neighborhood street?
At 3 A.M.? Probably. At 3 P.M.? Unlikely.
Most of what makes neighborhood streets dangerous is pedestrians. After dark, this concern goes way down. At some point, it becomes effectively zero, and the only thing increasing the risk is the number of driveway entrances, and in particular, blind driveway entrances.
School zones are another place where the speed limit is set for safety, not revenue generation - it has to do with reaction times, stopping distance, etc.
And, of course, the presence of small children who behave erratically. In general, you should drive those speeds whenever you see evidence that small children are playing or are likely to be playing anyway, e.g. when driving past parks before sunset, when you see small children walking down the sidewalk while tossing a ball back and forth, etc.
And when there's no evidence of children, it doesn't make sense to slow down nearly as much.
Cyclists and pedestrians are also a big risk. They often behave in unpredictable ways. Also, if you pull out in front of cyclists, this is a very bad thing. But all of those factors are also highly timing-dependent. When there are no cyclists nearby, a road can be 45 MPH, but when cyclists are nearby, you need to slow down. Drivers need to have the situational awareness to realize that driving at the speed limit is not always safe, because the alternative is for the speed limits to be set so low that they are always safe, which results in miserably slow roads.
I've heard of neighborhoods pushing for 5 MPH (8 KPH) speed limits. When cyclists and even some pedestrians would be ticketed for exceeding the speed limit, you're doing it wrong. Even at 15MPH, there's only a 9% chance of an accident seriously hurting a pedestrian even if you don't slow down at all, so the benefit would only come from drivers who are completely not paying attention, and would likely be cancelled out by a higher number of drivers zoning out and not paying attention, in which case the chances of pulling out in front of a cyclist (who realistically won't be going that slowly) goes up. No free lunch. But that doesn't keep people who don't understand statistics from saying "If 25 (residential default) is good, 5 is better."