SGI's problem was that they had a good run at selling really expensive hardware which could do 3D graphics well; they only really had Sun for competition in those days. Once PC graphics got good enough, it was a death knell; you could either spend 10k+ on an SGI/Sun workstation or about 2k for a high-end PC which was just as good (or near enough). Given that most people had a PC to run their office apps (Outlook, Word, Excel etc), it also saved desk space.
Sun at least had a fairly solid server business which kept them going for a while; SGI servers were generally only used in places with a strong presence of SGIs on the desktop.
SGI also had some classics - "XFS is great, it'll never need to be fsck'd!" Few months later, the fsck for xfs was released... It was, in general, a good OS with some odd quirks as I recall.