"Allowing patents on this means giving somebody the ownership of a piece of math."
No, this is only true for a particular patentable algorithm in the same way it is true for *all* patentable subject matter. This is another argument that is not consistently applied (any math argument that applies to a computer algorithm trivially generalizes to all physical machines and material processes).
To use your quicksort example, if someone patents the quicksort algorithm it does not prevent you from sorting data generally. Sure the alternative existing methods might be less efficient (something you could hopefully say about all types of patents), but there are an infinite number of alternative methods to achieve the end result. You are not patenting "sorting", you are patenting one method of doing that transformation. In the same way, you cannot patent a particular chemical, but you *can* patent a process for synthesizing that chemical. There are cases where the patented method is the only known (practical) method for achieving a particular result, but those are precisely the cases patents were intended for.
An argument that does not really follow is like your one above that someone might patent the basic building block algorithms of software. How can an algorithm that does not exist be a "basic building block"? What has everyone been using up until the point where it is invented? The algorithms we consider "basic" are so because they have been around forever and are therefore in the public domain. What were people doing before the existing "basic building block" algorithms were invented? An algorithm that becomes a future "basic building block", those are arguably the algorithms that more than any other deserve patent protection if you are going to bother at all. Pretty much non-obvious, novel, and eminently useful by definition.
There are open algorithm problems in computer science that, if solved, would generate new basic building blocks that don't exist now. This has been true in the chemical process patent field many times as well. But it seems odd to lay claim to them as "basic building blocks" before they exist. If someone invents an anti-gravity device it may become a basic building block of transportation, but does that suddenly make it non-patentable subject matter? I'm just looking for consistent treatment, not any particular outcome. If we exclude all the inventions that are good enough that they become basic building blocks, there is really no point to having patents at all. Arguably the goal of the patent system is to encourage the invention of basic building blocks that do not currently exist.