Become a fan of Slashdot on Facebook


Forgot your password?
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. ×

Comment Re: Coming from an information security academic (Score 1) 88

Other folks here have provided insight and commentary that you likely have no clue as to what you are talking about, but who doesn't love a dogpile?

I have implemented MANY very large Splunk and ELK implementations. ELK will almost always ask for MORE hardware to get search performance. I agree that ELK scales out more quickly, but far less efficiently than Splunk does. If your sole criteria is search speed and you have unlimited hardware capacity then ELK is the way to go.

However, doing calculations on the logs, presenting the logs, transforming the data (geo IP lookups, changing the message so that it reads more easily), and doing multivariable comparison for either human or automated response is vastly superior in Splunk. In both the functions and toolkits available and the ability to front load a lot of your search work so that your performance is outstanding.

Cost wise... it's usually a wash. I have customers that have looked at the cost of installing and maintaining an ELK stack and replacing the lost features and ran away quickly. This is for >500GB/day infrastructures with a dedicated dev team of >3 people.

If your Splunk implementation is sucking wind that badly, then it is likely that whoever is paying for your implementation has expressed goals that are counter to your goals and thus you are ill served. If you are the payer, then you have done poorly at describing your desired outcome and approx 50% of the result is your fault.

Continuing on... You mention Flume and Solr. Solr, if you buy the production implementation (last time I looked) doesn't have a good flow control and message verification platform and is thus dependent on the messaging bus within Flume or the implementation of an outside message bus (Kafka, Redis). This results in another set of configurations to maintain, and a good place for logs to be lost in the ether. Flume itself is awesome, although the parsing recipes could use some work. If I were looking outside of Logstash/Beats (which is advisable as Logstash seems to still have some memory management issues) I would favor Fluent as the ingest process is less of a pain in the neck.

However, I've only done hundreds of implementations of log management infrastructures using logstash, ElasticSearch, Kibana, flume, kafka, redis, fluent, syslog-ng, and/or Splunk... so there are likely some options I haven't mentioned.

Slashdot Top Deals

Sentient plasmoids are a gas.