I also used to have doubt about it because I heard it was categorized as a "log analysis", which is very unsexy from a developer's point of view. Can we be tricked by names! After using it for a while just yesterday, I found it not bad at all.
In a nutshell, Splunk is valuable because it turns your unstructured untyped data into something much easier to analyze and Business Intelligence (BI) ready. IMHO, the strength of Splunk is two folds. First, it provides a central place to view and mine your logs across hundreds or thousands of machines. This is very useful in many organizations with large scale distributed computing because you don't need to ssh into your database server in one terminal and compare the results in another terminal connecting to your web server, if you could ever figure out which of the 100 database servers you should look at. Second, instead of just greping, it treats each log entry as a list of key-value pairs and provides a simple yet powerful query language for it. On top of that, it gives you a library of visualization components. I was able to build a fully-customized Google Analytics style dashboard for my application in 20 minutes. And I would say that is empowerment, and empowerment is the most valuable value.
And if you are very into technologies, they are a Python house, which is always a plus :)
That's mostly because their target customer is big enterprise who feeds on buzzwords they can feed to upper management. It's similar to the websites of SAP, Oracle and Intel, which all have the same qualities:
- No up-front pricing, you need to get a quote
- No screen-shots, you need to see a prepared demo
- Any contact what the company results in assignment of an "account manager", the used-car salesman of the software industry.
- Lots of testimonials from people you've never heard of talking about software you've never heard of using the very same buzzwords found on the website.
That's because they're a log analysis company trying to ride the wave of Big Data hype. It doesn't really fit but if you squint and look at it in the right light you can imagine that it does...
Splunk in a nutshell: it runs a daemon on all your boxes, that tails all your logfiles (you can configure which ones) and pipes them all to a central logging server. On the logging server, another daemon runs that does pattern matching/filtering on the incoming logs, and fires an alert when it gets a match. And umm, that's about it really.
The one useful thing it can do is give (say) devs access to logfiles from prod servers that they aren't allowed for whatever reason to log into themselves. But you could do this yourself with a periodic rsync to an internal webserver...
Not so - Splunk has a very sophisticated index for huge amounts of data, and a rather fancy DSL for analyzing it. I've looked at it in some depth as one of our products (http://zetetic.net/software-combine) does log aggregation (and does a much better job with Windows and SNMP than does Splunk), but Splunk is not at all a trivial piece of software.
Honestly, your competing with inhouse scripts people spend a few weeks / year maintaining. Which is not to say you can't make a lot of money doing so. Ideally, your saving people significant time writing these scripts, but tossing out buzzwords just alienates your users.
Splunk's query language is not like regex - it's somewhat more like SQL or LINQ, with groupings, aggregates, pivots, etc. It is definitely a DSL.
I am not a Splunk developer or serious user of it, but you greatly underestimate the problem space to suggest this class of products address the same problems solved by even a very skilled scripter's scripts.
It's much more than that. It lest you query logs, and queries can be very complex, which then gives you a real time result with charts and all. I've used it in production, and it's was the best damn tool to debug production problems. The licensing costs are insane though.
It really depends on the size of the logs, the number of users, and the deal you can negotiate. We paid around $60K/year, but we had quite a lot of logs.
As a guy who used to resell Splunk, it offers a lot more in comparison to it's open source alternatives. Being data agnostic with decent log compression, with an extremely rich searching syntax, some very good (HTML 5) reporting dashboards, decent reporting and a verbose API make it very attractive for users (companies) who have compliance requirements or require some form of performance monitoring. (I know it's a mega sentence)
I have since left working with Splunk directly but I would still advocate its use because it's one of the better commercial (albeit expensive) log management/SIEM products around.
* the largest data source in any enterprise
* increasingly must be stored for long-term analysis ...
* by corporate policy or ...
* by government regulation/demand
Regular open-source tool stacks can't do this without astronomical cost in storage hardware. Various log analysis players have managed to make this data storable and some have done well on the analysis side.
Splunk will do well. I hope this boosts acquisition interest in my old company, http://www.sensage.com, another log analysis company.
Splunk is one of the best pieces of software ever made. It's critical to almost everything Shopify does. It's pricy but easy to justify because it makes any ops and many dev tasks 10x easier and therefore there are some real savings in required manpower.
The push away from something very well understood and easily explained (syslog and SIEM aggregation) to calling themselves "big data" and "business analytics" is kind of lame, IMO. I assume they were pushed to do this by the investment market, since big data is hot.
Who designs these things? I gave up after two minutes.