You'd need more than 50% to not return the form before it would make any practical difference at the statistical level.
Except that the census isn't designed solely for macro-level statistical information. One of the most important roles of a census is determining a city/county/state's population, which is used to allocate funding, and determine the number of representatives in the US House and state houses/senates, which does have a significant impact on the makeup of those bodies.
Return rates are not uniform across the board. Large cities are notoriously under-counted, because of the difficulty of counting the homeless population, renters, those who move during the course of the census, those who do not speak English (even though the Census prints in multiple languages, return rates are still lower among non-English speakers), and various other groups that tend to be much more prevalent in large cities than in smaller cities and more middle-class suburban neighborhoods. This map of Census forms returned county-by-county provides an interesting look at the issue. While the percentages can't be considered completely accurate due to issues like vacant apartments, etc, there's still significant variance. In New York state, for example, mail return rates per county range from 43% to 84%. That's a staggering variance, and when it comes to ensuring that residents have adequate funding and representation, having fairly accurate results is essential.
As an aside, statistical sampling for the census has been discussed in the past to avoid these issues. I'm not opposed to using a reasonable sampling technique, so long as it accounts for areas with statistically low return rates. However, Republicans oppose sampling because they feel it overcounts groups that tend to vote Democratic (and, Democrats tend to support sampling because they feel it's a more accurate count). In 1999, the Supreme Court ruled that sampling cannot be used to determine population for the purposes of apportioning US congressional seats, and while it could be used for drawing state/local lines and for allocating federal funds, it's such a political football that it probably won't happen in the foreseeable future.
What this country needs is a dime that will buy a good five-cent bagel.