The problem is if a new, faster algorithm comes out. For sorting, it's pretty much done, so you're right. But for something else, if we have a graphical programming language, we'll have to alter the underlying structure some way. Let's say we have blocks/triangles/etc. How do we represent a quick sort vs a bubble sort vs a merge sort? And is that more efficient than writing text? If we think of a better solution, how are we to represent it? Do visual aids make our mind think of better solutions better than text in all scenarios? Will we have all of the visual objects that we need?
As a small example, at my current job, we had a SQL query running extremely slowly. I modified part of the query to be cached in Java, and it became 10x faster. If we abstract to visual objects, that SQL query and SQL query+Java will look identical. If we make it less abstract, then eventually we're using visual aids called characters (each letter in the alphabet and mathematical symbols are graphical images technically). Also, I'm able to manipulate everything in such a way that I'm not dependent on someone else to fix the underlying problem. I can debug the base code, and I can make it faster. If it's too abstract, I can't. Why did GameMaker not become the de facto standard for PS/XBOX/PC? Every year, they're pushing the envelope to make it faster and improved.
Which brings me to my next point, you'll still need some type of textual language to create this graphical language -- to create the underlying base structure.
Lastly, typing is pretty quick. With a graphical language, I'll have to move objects around with my mouse. With a very complex problem, that could take a lot longer than just typing up some text. For python, I just write: print 'hello' For a graphical language, I'll have to find the print section of objects (through I guess a large menu? or a toolbox?) and drag it onto the screen and then type in 'hello.' I'll need a damn large menu and/or toolbox of objects that I'll have to scan through. Now imagine "print str(2+5/70)" That took me 2 seconds to type. For a graphical language, I'll have to find and drag down print; find and drag down str cast; find and drag down the + image; and find and drag down the / image. Or something like that. It will suck.
For simple stuff (and for GUI programming), it works. Complex things will be way more inefficient.