I think this is the most important point to note. If we look at the history of our species, it is only phenomenally recently that there has been mass expansion. Homo Sapiens Sapiens 15 thousand years ago was virtually identical to now, and lived mobile in very small groups (with a couple of exceptions). Imperialism didn't exist.
People in the West (and now elsewhere) have come to understand evolution, particularly of our species, in a very biased way. The interpretation is that evolution is directed, or is somehow moving species to "higher" or "more developed" forms. This is not the case. The essence of the theory of Natural Selection simply states that those individuals who are fittest *at the time they are living* will, on average, survive and reproduce more effectively. Nothing about becoming "more advanced" or "moving to perfection". These are cultural add-ons. Conditions favouring one adaptation can change and those organisms can become disadvantaged. Believing the particular set of circumstances that has led to agriculture, cities, pseudo free-market capitalism and imperialism are ineluctable is (scientifically) indefensible. It happened here and it might happen on other planets but there is nothing necessary about it.
There is another problem with the Fermi Paradox - that a species that has developed interstellar travel would be interested in us or our planet. If we look at the state of current biotech, nanotech and AI, it looks utterly certain that humans (obviously not Homo Sapiens Sapiens but what we become) will be able to survive without a biosphere long before we will be able to embark on interstellar travel. The later very probably requires the former anyway. If that situation is common, and there is no reason why it shouldn't be, then interest in colonising, or even visiting, earth would probably be extremely slight. Even without postulating some Prime Directive type practice, why immediately assume alien species will be Borg-like? Or like a benevolent or malevolent father-figure? We can't help but conceive of the question in our own current cultural terms, which is obviously why the "paradox" appears a paradox at all.
I personally think the idea of the "singularity" is a very useful one to consider here. Technology is advancing very quickly at the moment and the way we look at the world is changing very quickly too. If we can't imagine (or predict if you like) what the world will become in less than a century, then why on earth should we believe we know how we will *think* in a century or so? Maybe we'll all be immortal philosophers simply sitting around meditating absorbing energy where we sit. Who knows, but that's the point!