Is this just a "monitoring" platform, or does this also intend to make use of Clickhouse as a "log analysis" platform ala Mozilla Hindsight?
Which is to say: is there some obvious hook-point to add watcher jobs (as Clickhouse stored procedures, maybe?) to process correlations in the inputs to the system? (Where by "watcher jobs", I mean things like "create a record in a table if a user makes requests across endpoints A, B, C, in that order — with other arbitrary requests from that user in between — in a five-minute sliding window, at some point within 12 hours of the user's registration.")
Using a DB as a log-analysis system would be pretty great, if it was practicable, as you'd be able to correlate present events with events from the distant past (or with statistical aggregations of all of history up to the present), rather than having to build your correlations only from what can be buffered in memory.
But most log-analysis platforms need extremely high event-processing-job performance to scale — Mozilla Hindsight, mentioned before, is a rewrite in C+Lua of a previous system (Mozilla Heka) where the fundamental bottleneck was the Golang runtime. I'd be curious to know whether Clickhouse sprocs/triggers/etc have been tuned to function at that sort of scale...
> Which is to say: is there some obvious hook-point to add watcher jobs (as Clickhouse stored procedures, maybe?) to process correlations in the inputs to the system? (Where by "watcher jobs", I mean things like "create a record in a table if a user makes requests across endpoints A, B, C, in that order — with other arbitrary requests from that user in between — in a five-minute sliding window, at some point within 12 hours of the user's registration.")
This is not something we have planned, nor have we heard yet. Is the use case for doing analytics on your web application? Or more of a complex tracing use case that creates new traces as time passes by?
Fwiw, Highlight is designed (in the current state) for basic log search and alerts.
We want to notice users who we ban for breaking the Terms of Use of our API SaaS, who then create new accounts and immediately resume doing the same thing they were doing before they were banned — where the metadata is all different (they're rotate VPNs, get a new [stolen] credit card, etc) but where there is a distinctive "activity-pattern fingerprint" to their ToU-breaking activity (different per violating user, but the same between the "incarnations" of the same user); and where having that fingerprint in the context of a brand new user is implausible, since nobody could learn to use our API to do such a complex thing so quickly.
Oh very interesting. I don't think we're going to build something for that use case specifically. But long term, we do plan to build a metrics product, which you could use to analyze these patterns. It's interesting to hear what people use these sorts of product for. Thanks for sharing.
Which is to say: is there some obvious hook-point to add watcher jobs (as Clickhouse stored procedures, maybe?) to process correlations in the inputs to the system? (Where by "watcher jobs", I mean things like "create a record in a table if a user makes requests across endpoints A, B, C, in that order — with other arbitrary requests from that user in between — in a five-minute sliding window, at some point within 12 hours of the user's registration.")
Using a DB as a log-analysis system would be pretty great, if it was practicable, as you'd be able to correlate present events with events from the distant past (or with statistical aggregations of all of history up to the present), rather than having to build your correlations only from what can be buffered in memory.
But most log-analysis platforms need extremely high event-processing-job performance to scale — Mozilla Hindsight, mentioned before, is a rewrite in C+Lua of a previous system (Mozilla Heka) where the fundamental bottleneck was the Golang runtime. I'd be curious to know whether Clickhouse sprocs/triggers/etc have been tuned to function at that sort of scale...