we're stuck with a 26 year old web html / javascript / css / connectionless protocol technology stack
HTTP is not connectionless, it is based on TCP, which emulates telephone connections.
HTTP 1.1 has support for re-using an open TCP connection.
(HTTP/3 uses QUIC instead of TCP, which does connection multiplexing instead.)
HTTP is stateless, or at least it's supposed to be. It's a request-response protocol for documents. For that purpose, it has largely replaced Gopher and FTP.
It has the advantage of reporting the MIME type of the document.
HTTP and HTML date from 1989, which is a little more than 26 years old. Closer to 36. (I guess the naughties never happened?)
HTTPS dates from 1994, JavaScript dates from 1995, and CSS dates from 1996.
HTTP/1.1 dates from 1997.
none of the largest companies and stakeholders with a voice are willing to propose a significant replacement
There is the Gemini Protocol which has all the advantages of HTTP(S), but is much cleaner and simpler and faster.
Google does not seem to like it.
Anyway, neither HTTP nor Gopher nor Gemini are very good at being hypertext protocols. They don't do versioning, they don't do back-references, and what support there is for distributed caches in HTTP ("proxies") doesn't work very well - it is designed for reducing traffic, not for providing redundancy. (BitTorrent does that better.)
The optimization of compressing web content and combining HTTP requests is a band-aid
Support for gzip in connections is very useful. It means that compressed documents can keep their multipurpose internet mail exchange type, rather than the server reporting an application/gzip with no clue what to do with it. And text comresses very well.
And re-using existing connections reduces latency, because you don't need a three-way handshake for every document.
Those are solutions, not band-aids, but you are correct in that they don't solve problems other than the ones that they solve.
Connection multiplexing, what HTTP/2 and HTTP/3 do, only increases overhead. It belongs to the problem space, not to the solution space.
still are underlaid with the legacy JavaScript technology
The script tag in HTML allows specifying which scripting language to use, but in practice, only JavaScript is relevant, and that is not guaranteed to work. The venerable Lynx Browser/a> simply ignores it, as explicitly allowed by the W3 specification. In theory, you could implement a VM for any scripting language in your browser, like Dart, Coffescript, or Lua, but nobody bothers.
Probably for good reason. Injecting random executable code (like JavaScript) into documents (like HTML) is generally considered a severe security risk. Remember the grief that Microsoft got for executable macros in Word and WMF documents. Zero-click remote code execution.
HTML has a few severe short-comings when compared to more mature mark-up languages like ROFF or LaTEX.
For one, it does not distinguish between spaces between words and spaces between sentences, which makes text unnecessarily harder to read. How can you tell if a period at the end of a word makes the end of a sentence or an abbreviation?
Then there is the ambiguity of paragraphs. XHTML allows paragraphs to be nested, HTML4 and HTML5 do not. Now, you might think that nesting paragraphs doesn't make any sense, but HTML documents are full of div-Tags for structure. There is no standard way of grouping a picture with a caption, for example. Sections are separated by headers, but HTML layout engines generally don't treat the text between sections as one unit. But why should you need a tag specifically for for separating paragraphs anyway, except for nesting?
HTML does have the distinction of being media agnostic. Where DVI and PDF have strict page layouts, HTML doesn't care if it is rendered on an old black-and-white television screen, an RGB laser projector, a sheet of A4 paper, a sheet of US Letter paper, a text-to-speech system, or a Braille line display, or anything else. How an HTML document is rendered is inevitably and by necessity up to the user agent. And so it does not provide the typical 12 section layout from print media.
It is not particularly fit for the purpose of author-defined layouts. That's what PDF is for, which can also contain forms and random executable code.
And of course HTML is used for exactly that.
BBC.co.uk should not require 99+ http requests top get 7 MB of data for example.
Does it? The BCC is one those web sites that still work in Lynx, NetSurf, and Dillo. That is not a good example of what is going wrong on the web.
The main problem is the web frameworks that are desperatly trying to re-invent X11 with extra steps and more overhead.