But were those long address protocols designed to be routable in a worldwide network? Sure, Ethernet had a 48-bit address too, but it was only intended to be a unique hardware ID. There is no way to contact an arbitrary Ethernet MAC address outside of your LAN, even if you already know that it exists. Were they designed to work with the low-speed serial links that were common back in the day? Sure, you can spare a few extra bits when you've got over a million per second, but not when you've got a mere thousands of bits per second.
Back in those days communications were slow (56Kbps was about 6 characters per second, or 7cps synchronous). And CPUs weren't fast. People wouldn't have tolerated protocols that took up a significant percentage of CPU time. More importantly, fast routing depends on custom logic to handle headers without a CPU, and variable-length headers make this much harder. IPv6's optional headers are tricky enough, but variable address lengths would have been very hard to process with custom logic.
And encryption? It was literally a non-issue for network protocols back in those days because it is so compute-intensive. The point of a network protocol is to route data, you don't stick something as expensive as encryption on the lower layers without a good reason, such as wireless transmission. WiFi has link-layer encryption, but that disappears once the data goes onto a wire. And if you're not going to encrypt the headers anyhow (how do you use the options that specify encryption in an already encrypted header?), then why the fuck even bother? If the data needs to be encrypted, put that at layer 5 or 6 or 7 of the protocol.
Also, which algorithm? Any sufficiently fast algorithm from those days would be useless today. DES was brand new in the '70s, and eventually got chips, but you're going to require one of those in every network node? There are still unanswered questions about how its specific design was chosen. All specifying an algorithm would do is keep a bad algorithm alive forever. SSL is still trying to shake off bad algorithms. We've already thrown away at least two generations of encryption just for WiFi alone, that you can still use, and it's barely 20 years old. And yes, the munitions bullshit was another reason why they would have kept it completely out of the network protocol. It just isn't the business of a routing protocol to deal with encryption.
Hindsight is easy when you don't consider the limitations of what was knowable or possible back in the day. Very few things (other than perhaps the limitations of classful routing) could have been foreseen in what was still considered a mostly experimental system. There was no way they could have known that TCP/IP (which wasn't even their first protocol!) would have ended up the winner and persisted for decades until long after the point where it had run out of addresses.