It's hard to do almost any programming without understanding boolean operations (both logical and bitwise), and one will be really limited if one doesn't understand binary arithmetic and how hexadecimal works. I don't think this stuff is ordinarily taught in grades K to 4. One isn't going to understand how what integer types in many languages do unless one understands modulo-2^n arithmetic. Again, that's not ordinarily taught in grades K to 4. It may not even be taught in grades 5 to 12 (no doubt depends on school). None of this is *hard* mathematics, but it's mathematics nonetheless.
Generally speaking, all algorithms should probably be thought of as mathematical entities. So whenever one is trying to figure out an algorithm for a task, one is doing mathematics. It's not the sort of mathematics one typically does in K-12, but it's mathematics nonetheless. And it's not uncommon to have to do a little bit of traditional mathematics on the side to figure out if you're going to run out of memory or take too long.
And even if you're not trying to understand an algorithm yourself, at least you need to be able to understand statements like "Worst case performance of a merge sort is O(n log n) while the average case performance of a bubble sort is O(n^2)" in order to choose between off-the-shelf ones.