Yes, and you could reasonably make a graphical interface for assembly language. Very simple tools with very simple connectors, mapping onto the real world (transistors, pins, registers, leads). The more abstract you get, the more explicit your instructions need to be to get reliable, reproducible, complex results. It's really hard to do that with predefined building blocks, unless part of your building block is a big chunk of very specific text, which sounds remarkably like code.
Another similar thing would be comparing movies to books. The richness, detail, and depth of a book just can't be done in film, in part because we don't want to sit there that long. But also, look at all the versions of a specific book that was done, each of which was done with the goal of being at least as good as the book, and how differently they turned out. This is something we don't want to have when we write code - we want the same results every time, with no limit to expression within the capabilities of the language/compiler/hardware/etc.
Another great example is language and writing itself. There's a reason the vast majority of the world went away from using pictures to describe concepts (words) - after a while you had a huge collection of pictures, and were still limited to doing more basic writing if you didn't have a clear idea of which picture referred to which concept. And so there was a decoupling between the concept (word) and the presentation of it (writing). C-A-T spells cat, but none of those parts have anything to do with the furry critter with pointy ears, whiskers, and a love for sleeping and chasing mice.
It's not that I think it would be impossible to make a visually mapped programming language, but the more abstract elements, particularly non-interface elements, would be just as difficult as they are today simply because they are abstract, difficult concepts.
