Sun apparently didn't do their job *to the very end* at various points.
1. x86 - they entered the market, but not quite (no desktops, no laptops, no low-cost servers, only big machines). You can run Solaris on x86 but not quite. You can even run it on a laptop and have NVidia accelerator running, but for most people it's still a dream, urban legend as they can't do it at home with their own hardware. Maybe they shouldn't enter x86 at all?
2. Java cross-platform myth. Write once - run anywhere... not quite. It's very very popular as "enterprise" solution, but most people don't use any Java desktop apps, applets were disaster and JavaFX... later about that!
3. Open source and their products. We all know Java is open source now (finally, and obviously with large amount of work done by RedHat!), but Solaris? Binary blobs must be included in any build to make it work. Incompatible with GPL libenses, and also not a BSD model - what was that all about? It was like: "yeah we want to be part of open source movement. but you can't fork our code too much".
4. Failure on building community. This IS a big deal. Linux has got great large community of users/developers/fans. Apple has got it's army of zombie fanboys. Sun tried to build community around OpenSolaris and failed. "Project Kenai", "Zembly" look like half-finished sites. Just compare it with Github (I know it's a bit different usage but hey). The only successful one is Netbeans.org IMHO, but still - could be more successful if they didn't require signing agreement before submitting patches. Hell, I love Netbeans but I won't send them my code so they can use it in closed-source Sun Studio.
5. Not allowing interested users to use their innovative products. I am a software developer. I write software using Linux. I wanted to try out JavaFX... and you know what? It doesn't run on Linux. I wanted to write widgets on desktop using cutting-edge JVM drag-from-firefox-to-desktop feature, and expected my browser not to crash. I finally wanted x64 Java plugin for years, and once it got here - most people already use OpenJDK.
6. Desktop Java. Swing could be most popular GUI toolkit today if it integrated nicely with Gnome desktop for years now, if Java could be distributed easily with Debian, and people wrote software for it. No, let's keep Java close till it becomes obsolete on desktop and release it then. Crazy.
7. Trying to be service provider. OK, sun's hardware is great. Service providers buy Sun's hardware, say data centers. Now, one day, Sun becomes services provider, direct competitor of people who buy hardware from this company. Isn't there a conflict of interests?
8. No one mainline software. Yeah, sun has Solaris. But also had Linux distro. Bought MySQL, but also had flirted with PostgreSQL, Apache derby. It obviously confuses people, and look at IBM: "Go run Linux and DB2 on our servers".
9. Bunch of outdated, obsolete software that no one use. Some basic software like shells that come with Solaris were totally out of date till recently. And they still run "innovative" projects that failed many years ago: Project Looking Glass as best example.
10. Sparc failure. Maybe not exactly a failure. I know it's really great processor family. It has got potential. It's fast, multi-core, modern. Probably made them loose lots of money recently. What went wrong here? Maybe they should license chips for third-parties? Maybe they should build and push desktop/mobile versions? Maybe they should abandon it for PowerPC to provide better compatibility with IBM? I really don't know, but they did something wrong, and giving it's own customers alternative as AMD servers didn't help.
They did too much things wrong, and maybe too much things in general at the same time rather than concentrate on what brings them profit - their hardware.