This is not a disparagement of terminals, because they're either good at certain tasks, or allow simpler development.
Terminal emulators have barely changed in 30 years. They're still just scrolling windows of unstructured text. Why is there so little innovation in an application we use every day?
On a highly technical level, modern graphic terminals are meant to emulate the ascii plain text of older devices, which in turn emulate the 80x25 character display, which in turn emulate the 80-column punch card system, and early ASCII tables. Even though terminals are useful, they're still anchored by significant historical design.
There's been a few attempts to at least break out of the raw text stream - mostly the Curses library in order to give colored text, or to at least allow presentation beyond a simple text stream, but it was still designed for terminals rather than interactive GUIs. Additionally, something like Telnet (a primitive remote console) didn't autodetect the client's display size.
Now, there's room to make adjustments if the official standard is willing to shed the last remnants of the oldest terminals; perhaps shedding the various ascii control characters that don't seem to be used even in special cases (anyone seen Shift Out or Shift In?), and replace it with new control characters that can help. However, this requires changing a standard and that's extremely slow.
- If I type ls to show a directory listing, I should be able to right-click on a filename and get a list of operations to perform on that file, just like a file browser.
So, that would be a rich-text or hypertext terminal. After fiddling with the standard, let's pretend this works, and implementing it isn't too complex.
The question is how it will work. Let's say I'm on Windows, and using a telnet to a Linux system - how will right-clicking to get the filename help when going across computers? There's no guaranteed file sharing, or it might be blocked for some reason. Maybe the file isn't in a shared folder, such as something in /etc/ or /usr/home/root/
Your example works fine on local terminals, but not for an SSH into a remote computer.
- Why can't I cat an image file and see the image right in the terminal window?
At that point, it's full hypertext. I'd want to be very cautious with that, considering the many ways some webpages can attempt to trick the user into doing something that shouldn't. And since terminals are designed for dealing with admin-type stuff, it can become rather dangerous.