Zero is not a number, it is a concept.
We could argue about whether or not numbers are anything other than concepts, but zero is a number. It is as much of a number as 1, 2, 3, ......
One of the big hangups in early mathematics was that they were confused about the very thing you are: namely there is a difference between 0 and non-existence. I can ask what speed an object at rest has, and the answer is 0 (in whichever units you want). The answer is not "it does not have one" or that the speed does not exist. If you try doing velocity addition, or momentum conservation etc on an object at rest you will see that you do need to stick in '0' for the velocity.
(You may object -- there is no difference between an object not having momentum and thus not counting it, or having zero momentum. The easiest way to see a difference is to go to a difference reference frame -- then the momentums all transform. It would not make sense for a non-existant momentum to transform).
The real problem with division by zero is that
0 x = 0
for any x. If we just have access to the RHS, we have no information about x, so asking "what do we need to multiply 0 by in order to get zero" the answer is "anything you want".
As to the question of whether or not infinity is a number, or simply a limiting process depends. There are not just one set of numbers, and you can actually define infinity to be a point, or simply a limit. (There are even multiple ways of "adding infinity" the most common of which in complex anaylsis is to add a "point at infinity" in which case -inf and +inf are the same thing.)
But it is true the most common way of dealing with numbers is to treat infinity as a limit.