8.3 came from Gary Kildall (who never worked for Microsoft) in 1974. Microsoft wasn't even founded until a year later and in 1980 chose to make the DOS file system compatible with Kildall's existing CP/M. None of this seems unwise or shows a lack of foresight on Microsoft's part.
Gary Kildall was the CEO of Digital Research. They created CP/M, a very popular operating system for (at the time) 8 bit microcomputers running the Z80 CPU.
Microsoft at best made a BASIC for these microcomputers - things like the FAT filesystem can trace their origins to microcomputer BASIC from Micro-Soft (as it was known then).
Now, IBM famously tried to arrange a hush-hush meeting with Digital Research to get CP/M ported to 16-bit processors (i.e., the 8086) for their new Personal Computer. Unfortunately, Gary was out flying, while his wife refused to sign the NDA IBM wanted. Now, depending on the version of the story, the widely publicized one was that Gary was flying for fun and not to be disturbed which irritated IBM as they mutually arranged for this meeting. In reality, Gary's wife was the one who did the business matters (Gary was the technical guy/programmer/etc) so Gary wasn't actually needed since it was thought that they would hammer out the legal agreements. But it was not only the failure of Gary's wife to sign the NDA, but that they could not hammer out a licensing arrangement. IBM wanted "free and clear" licensing - they'd pay Digital Research a sum of money for unlimited copies, and that was something Digital Research refused to do. So IBM left.
IBM then asked Micro-Soft (who was providing the BASIC and languages support) if they would supply the OS as well, which is where they bought QDOS from Seattle Computer Products, which was a clean room implementation of a CP/M like OS for the 8086. (It was by definition clean room - as CP/M for 16-bit processors didn't yet exist). Of course, Digital Research sued IBM and Microsoft, and IBM settled by offering CP/M, when it was available, with the IBM PC. Incidentally, this was one of the first "look and feel" lawsuits - it was felt that MS-DOS felt a little "too close" to CP/M
Of course, by then it was too late - MS-DOS and PC-DOS was well established for years, and were cheaper at $99 vs. DR's 16-bit CP/M at $250.
And yes, 8.3 was ridiculous in the 80s, when even contemporary PCs supported longer file names - I think the Commodore 64 supported 16 characters for the filename, while the Apple II supported 30 characters (!) in DOS and 15 characters (aww...) in ProDOS. And the Macintosh supported 32 characters initially.
And that we put up with this silliness until the mid 90s. When every other computer and OS on the market supported much longer filenames. Or *gasp*, spaces.
It should be noted that FAT12 and FAT16 are hardcoded to only support 8.3 filenames - the directory contains 8 characters for name, and 3 characters for extension (all filenames are padded to 8 with spaces). FAT32 was the first to support long filenames natively (support was added in Windows 98, so Windows 95 only did FAT12 and FAT16). So long filename support was a hack for FAT12 and FAT16 (and added to Linux as "vfat" but you could mount it as "msdos" to get back to 8.3. (and there was the umsdos extension to add UNIX permissions and ownership information).
Incidentally, long filename support was one of the patents Microsoft sued over initially, leading to vfat being slightly incompatible (I believe it stopped making the 8.3 compatible filename), but it was OK as Windows would auto-create them on mounting.