The book "Alex's Adventures in Numberland" by Alex Bellos talks about this. There is a difference in the way we treat numbers 1, 2, 3, possibly 4, compared to numbers 5 and above. The author suggests this is reflected in number systems. Roman numerals start I II III and there are two alternatives for four, IIII and IV. In Chinese too the first three numbers are groups of lines before it changes with four. Our Indio-Arabic glyphs for 1, 2, 3 also originate as one, two, or three lines joined up.
I would suggest that case endings in Slavonic languages might be another indicator. For two, three, or four of something you use the genitive singular (roughly: "three of apple") but then for five or more the genitive plural is used instead. (Things may get muddy when dealing with compound numbers like forty-two where you could use the same case ending as for two.)
What other languages or writing systems have a change between 2,3,4 and larger numbers?