Being able to quickly execute arithmetic is a basic skill that everyone should be pushed to master.
However it has jack all to do with algebra II.
You're mistaking correlation with causation. Inability to quickly add and subtract doesn't really prevent one from understanding algebra. It's just that your students who never truly mastered basic arithmetic are not really truly mastering any of the math they're encountering.
Speed has little to do with it. It's a red herring for most areas of math. Better grasp does typically result in faster execution for the individual, but that does not imply that the slower individuals you encounter do not have a good grasp.