Wouldn't it be more accurate to say that numbers might be innate to the human perception of reality? If our brains worked differently we might use an entirely other way of viewing quantities that, for example, relies entirely upon geometry or some other, completely alien faculty not just for expression but for actually thinking about them - like those of a mind (hypothetically, that of a little green man from alpha centauri) that sees reality as a continuum and doesn't think about discrete objects or groupings as such. Who's to say that such a mind wouldn't have a more accurate view of reality?
A friend of mine has dyscalculia, and he doesn't grasp arithmetic or more than very basic numeral manipulation - yet he obviously can judge quantities and can intuitively reason abstractly about logical relationships between things. He just can't do math like we others can.