Any software requiring documentation is broken.
I blame Bob Wallace.
Bob Wallace was one of the originators of the concept of "shareware", and he got paid not for his software. This made people wonder how Quicksoft was able to stay in business.
When questioned about this at one convention, he made circling motions with his hands on either side of his head, and said "Software is ... all up here ... it's not real, it's ephemeral. I don't sell software, I sell manuals". So Quicksoft made its money, and its livelihood in the margin between the cost of mass-producing a manual vs. printing it out from a floppy disk and using up a bunch of tractor feed paper and expensive ribbon.
Or, to put it another way, Quicksoft made their money by having a relatively feature-full product which was nearly impossible to use without documentation. And people have been mistakenly trying to copy his success by utilizing the same technique, ever since.
Why did WordPerfect lose out to Microsoft Word? It wasn't because WordPerfect didn't already own the market; it did. It wasn't because Microsoft Word had more features; it didn't. Was Word a lot better, intrinsically, than WordPerfect? It actually wasn't.
Frankly, it was because of the F1 key. By the time WordPerfect got around to deciding they needed a "Help!" key, some of the function keys were already assigned, and so they assigned the next available one to be the "Help!" key. It helped sell a hell of a lot of keyboard templates. And it hid the help from anyone who'd experimentally go looking for it by hitting unlabeled keys in order until they found it (in fact, this would totally screw you up in WordPerfect).
Microsoft hit on a UX innovation: when something goes wrong, make the "Help!" key the first key someone is likely to hit, before all other keys.
And then they did it one better: The F1 was assigned to be the "Help!" key in *all* their products. Instead of just being a great UX thing, locating the key where they did on the basis of probability, they turned it into a Schelling Point: anyone who wanted "Help!" in any Microsoft product knew where to go to find it, if they had ever used some other Microsoft product, and needed "Help!" there.
So back to the original question: should you invest in documentation? Well, yes... if your product has already failed to the point where it's nearly impossible to use without documentation, or because, like Bob Wallace, you intentionally made it nearly impossible to use without documentation because that's one of the premises of your business model.
Maybe you want to write books on your project, once it's used by enough people to make that profitable, and that's how you plan to turn your hobby into a vacation fund. Or maybe you want to get to be a published author about a product so you get hired as a tech writer somewhere, or you get a lot of speaking engagements, and monetize your efforts that way. But if making your product hard to use was one of your initial conditions, then I think your software is broken.