I'm driving a 2017 Jeep Wrangler (JK) with a manual transmission. I don't drive much (less than 4k miles/yr).
I can't drive automatic transmissions without losing my mind. My guess is if you can find a car with a standard transmission, you can get it without "smart" features. If you can't drive stick, that's fine. I couldn't drive stick when I bought my first manual transmission. If I can learn, you can learn.
Put this write-up together over the course of the last 3 months when everyone started talking about getting VPN's. I know there's a lot of folks here building the "next best thing" on the Internet. Would be good if we started thinking about the privacy implications of our map-reduced, low-latency, high availability, machine learning algorithms.
+1 - Kind of the reason I started this blog post.. Besides reporting (which I know my setup lacks), from a techie perspective, what would this setup lack that Splunk provides.
Follow-up, is it _honestly_ worth the licensing to get those features?
We have a lot of in house expertise in ElasticSearch and choose it for a few reasons: 1) It's easier than Lucence/Sphinx to setup. 2) It's clustering support works out of the box and is so easy to configure it's not funny.
ES is basically a usability wrapper around Lucene. I've heard that Sphinx is better for a single node configuration, it's faster and uses less resources, but clustering with Sphinx is apparently tricky.
I'm not a Full-Text Search expert, but a number of really smart people at my company evaluated a number of them for one of the most critical pieces of our production site and they chose ElasticSearch.
Ok. The one part I don't like in ES is being java and the trouble that usually goes with it (yeah, judging stuff like this is bad, I know :)
For logging searching is the major item at large sites IMO (talking terabytes at least), when you're looking for all occurrences of "item x" over..
"the past week", it may take 1H
"the past month", it may take 10-30H
"the past year", uh, no, you don't do that.
So you gotta use ranges, but it's often hard to guess and you end up missing many log entries just because you don't have the time to search through them.
(obviously loading gigabytes of indexed data takes a while "physically speaking" anyway. I'm guessing ES can distribute the load tho, much like a web search engine does)
The vast majority of open source "big-data" infrastructure is in Java (Hadoop, HBase, Cassandra, Solr, Elastic Search etc). It works pretty well.
I'm not sure what your question is, but I've experimented with loading netflow data in Solr and I'm averaging sub-2 second query times. That's on a laptop, with a couple of minutes of netflow (around 10Gb).
With proper indexing your search response time shouldn't increase lineally with your data size.
loading 10gb on a traditional hdd takes more than 2s (that's 5gb/s read speed. Nice hard drive.). your data is either in ram and you've a lot of ram, either, it's just not 2s, or its not a 10gb index.
And i'm talking 100gb+ indexes ;-)
Obviously 2min of netflow data ain't much. I would want to see the result over 200h (or more) of netflow data, for example
No, querying the data takes less than 2 seconds. I can't remember the load time.
Obviously 2min of netflow data ain't much
Depends where you work...
I just checked, and it was 2Gb of netflow I tested on. That seemed small, so I looked a bit deeper and indeed I was only using a small fraction of our total netflow for that period.
Tt was adequate for what I was trying, though.
One of the follow-up posts to this is going to be on using OSSEC-HIDS which will give you logfile chained checksums. It's not perfect, but again it's about achieving the most value for the least amount of effort.
I can't drive automatic transmissions without losing my mind. My guess is if you can find a car with a standard transmission, you can get it without "smart" features. If you can't drive stick, that's fine. I couldn't drive stick when I bought my first manual transmission. If I can learn, you can learn.