Studies done on human babies under one year of age show that they have a concept of number up to three. (Source: ... I forget, but it's one of Keith Devlin's books.) These studies are based on the idea that babies stare longer at things that aren't expected -- so if you show them one object, then a second object, then hide them both and then reveal two objects, they are less surprised than if one object plus one object becomes three objects.
However, starting with four, the "innate math" of the brain fails. Everything after the number three is invented by human civilization. This is more remarkable when you consider that language IS innate to the human brain -- normal humans anywhere will develop a complete language, vocabulary and grammar with past and present tenses even if they aren't taught it by adults. (Source: my own head. I'd actually like to know if this is authoritative.)
The practical upshot of this is: yes, math is hard. The human brain isn't designed/evolved to do it at any level beyond "one, two, three, lots."