We know that an asteroid of sufficient size is going to hit again. It's only a matter of time. Maybe that time is a million years from now, maybe it's a week from now.
Asteroids are a lot like sharks. They're scary and exciting, they're good antagonists for movies, and so we tend to overestimate the danger they pose. Yes, a shark can tear your arm off, and if you happen to run into one while swimming, you should probably head the other way. But the reality is that far more people are killed by dogs, bees, car accidents, choking on food, drug reactions, and soforth.
In terms of natural disasters, the big killers are earthquakes, cyclones, tsunamis and floods. Wikipedia has a list: http://en.wikipedia.org/wiki/List_of_natural_disasters_by_death_toll
. Since 1900, the most deadly natural disasters have included the 1931 China floods (150,000-4 million deaths), the 1971 earthquake in China (240,00-800,000 deaths), the 1970 cyclone in Bangladesh (500,000 deaths), and the 2004 tsunami (280,000 deaths). There are no well-documented instances of meteorites causing mass casualties, but even taking an ancient Chinese report of 10,000 deaths at face value, the worst impact event in the past 1000 years wouldn't even score as the worst natural disaster in the past two years.
Or look at it in geological terms. The human species has been around for about 200,000 years. There is just a single well-documented case of an asteroid causing an extinction in the past 500 million years, the Cretaceous-Paleogene Chicxulub impact that wiped out the dinosaurs. The odds of such an event happening in the next ten thousand years or so are vanishingly remote. And other major extinctions coincide with ice ages (Eocene-Oligocene event, Ordivician-Silurian event) or volcanic eruptions (Permo-Triassic, Triassic-Jurassic, Cenomanian-Turonian events). So if we really want to worry about existential threats to the survival of the species, we should worry about ice ages and volcanoes, not asteroids.