It's because they throw out a lot of POSIX features/requirements - e.g. nested directories, rename, links, durability/consistency guarantees. In other areas, such as permissions, they have their own POSIX-incompatible alternatives. These shortcuts do make implementation easier, allowing a stronger focus on pure scalability. The theory is that the combined complexity of POSIX semantics and dealing with high scale (including issues such as performance and fault handling) is just too much, and it becomes an either/or situation. As a member of the GlusterFS team, I strongly disagree. My colleagues, including those on the Ceph team, probably do as well. The semantics of object stores like S3 have been designed to make their own developers' lives easier, and to hell with the users.
Not all POSIX features are necessary. Some are outdated, poorly specified, or truly too cumbersome to live. On the other hand, the object-store feature set is *too* small. I've seen too many users start with an object store, then slowly reimplement much of what's missing themselves. The result is a horde of slow, buggy, incompatible implementations of functionality that should be natively provided by the underlying storage. That's a pretty lousy situation even before we start to talk about being able to share files/objects with any kind of sane semantics. You want to write a file on one machine, send a message to another machine, and be sure they'll read what you just wrote? Yeah, you can do that, but the techniques you'll have to use are the same ones that are already inside your distributed object store. Even if both their implementation and yours are done well, the duplication will be disastrous for both performance and fault handling. It would be *far* better to enhance object stores than to keep making those mistakes . . . or you could just deploy a distributed file system and use the appropriate subset of the functionality that's already built in.
A semantically-rich object store like Ceph's RADOS can be a wonderful thing, but the dumbed-down kind is a disgrace.