Comment Re:Pros & Cons of non-relational solutions (Score 1) 423
Large databases use a combination of range partitioning AND indexing. If you're lucky you've also got hash partitioning - to distribute your database across N servers. Partitioning is far more general and forgiving for purposes like this than indexing.
And creating trends in your application layer then storing them in logs can work, but:
1. you still rely on figuring out in advance what you're going to need
2. if you don't load those logs back into the relational database then you can't effectively join it to all that data. And you then either must log redundant data or have useless logs.
3. grep & sed or custom reporting code against your logs is a poor substitute for any standard reporting tool or custom code against your database.
4. database features like partitioning, indexing, parallelism result in in-database aggregates being faster than logs
5. did i mention automatic query rewrite? where the database automatically converts eligible queries against the base table to actually run against the summary tables...