A lot of open-source projects by Nanobox look awesome, but most don't have any documentation or demos :-(.
I'm most interested in:
[Hooky D](https://github.com/nanopack/hookyd) - Remote hook execution layer
[Butter](https://github.com/nanopack/butter) - Git based deployment system
[Shuttle](https://github.com/nanopack/shuttle) - Transparent TCP proxy for web
As a point of clarification (I work at Pagoda Box/Nanobox). Pagoda Box uses an older suite of similar tools (https://github.com/stormpack-core) mostly written in Erlang. This newer project (written in Golang) is part of a group of tools (http://nanopack.io/) being written for https://nanobox.io/
The actual reason is much less exciting than you might have hoped and actually has nothing to do with erlang vs golang...
The erlang version has been running successfully for about 2 years and we haven't had any issues whatsoever (excepting a wrestling match with mnesia early on). We are erlang/elixir advocates and have used the erlang vm successfully on highly critical multi-million-concurrency services for over 6 years.
When we set out to build nanobox desktop (https://desktop.nanobox.io) our vision was to provide a single pre-compiled executable that could be run without any configuration. The design required a push layer and needed the same functionality that the erlang project was already providing. It wasn't feasible to package the erlang application into the nanobox binary, so we emulated the original project into a consumable golang package. As time went on we were porting more and more of the features into the golang port until all that was lacking was distribution and authentication. At that point we made the decision to consolidate our efforts into a single project, that could be a standalone service or composed within a golang binary.
I regret to inform you that there isn't a mass exodus within our company to ditch erlang for golang. While that certainly would make for a fun thread, in this case it was simply a matter of fit and effort consolidation.
This looks like a great project! I'm curious about the following caveat from the readme, however:
"Data flowing through mist is NOT touched in anyway. It is not verified in any way, but it MUST NOT contain a newline character as this will break the mist protocol."
I haven't read the code yet, but my gut reaction is that this suggests it would be possible to inject commands using malicious user input. Given the caveat, this may be allowed for by your threat model, but it may be worth some effort to mitigate.
EDIT: To clarify, this may not be the case at all, in which case I'd be curious to hear why this restriction is in place.
This is referring to messaging being sent through mist. Mist doesn't care what the data looks like, and won't transform it.
Currently the public-facing websocket client is not allowed to publish messages until this is mitigated, as you mentioned. Any feedback here would be appreciated.
Actually this proposal is quite bad compared to simple "<4 bytes size><size bytes contents>". If you use netstrings recursively like djb recommends, you will end up copying / reallocing the strings many times. Since your top-level length depends on the lower level length, you cannot send it without actually calculating the whole contents. Asn1 DER has the same issue because of the weird size encoding.
Mist isn't intended to be a message queue, its simply a message passer.
Similar to the old question, "if a tree falls in the woods and no one is there to hear it does it make a sound?" Yes; similarly Mist will send a message whether or not anyone is listening, and will keep sending them whether or not the client is prepared to handle them.
So to the first question, there's not a transactional layer to guarantee the client handled the message. Once a message is sent to the client, it is assumed the client has a mechanism for queuing messages if it can't keep up.
As to the second, the caveat is basically saying that if a message is broadcast and nobody cares about the message (ie: there aren't any subscriptions) then mist isn't going to store the message anywhere, it just gets dropped.
Err, I get that! but I am still wondering on where would I use this ? If I dont need transactional layer or buffering, I could use event listener and dispatcher pattern. Anything more than that, a message queue like ZMQ works. Where does this fit in ?
Oh I see. Sorry for the confusion. The initial need was for a rails app to send data model updates to dashboard clients. Later, as our infrastructure became more micro-service-oriented, mist has been used as a place for the dashboard javascript client to subscribe to a centralized event queue. Each micro-service will publish messages to the mist cluster, and the dashboard can respond accordingly.
Struggling to see when I'd use this over, say, rabbitmq. Feels a bit like "not invented here" syndrome, though glad to be proven wrong. Also curious that this replaces previous tooling written in erlang, be interested to here what prompted the move to golang, I would think a message broker was right in erlang's wheelhouse
Not the OP, but this looks like it's designed to be lossy, and prioritize availability over deliverability. If clients are unable to keep up with message volume, messages will be dropped.
That makes is okay for things like realtime dashboards (which this is apparently designed for) and other rapid events where transactionality is not required.
But it's obviously not suited for job-like scheduling like data processing or email delivery. Nor as an event bus for inter-service coordination ("on event X, do Y").
That's a pretty accurate assessment. Mist could be conceptualized as a self-hosted replacement for pusher (https://pusher.com/). It's designed specifically to address building realtime web apps.
Everything that mist does is "right in erlang's wheelhouse". Erlang is awesome and the actual reason we moved golang had nothing to do with erlang vs golang. I responded to that in detail at another spot on this thread (currently above, but who knows if that is still the case).
Two things come to mind: gnats uses wildcard topics, very useful to create filtered subscriptions. And gnats is pretty good at dealing with backpressure; the stated aim of Mist is to silently drop messages if clients can't keep up.
I would also say that gnats is mature and has been proven in production for several years, which might not be the case with this project.
Also to clarify (and yes I also work at Pagoda Box on the Nanobox project), this service was initially intended to provide push capabilities for rails and other web frameworks that can't natively provide a push socket layer. It has successfully filled that niche and has expanded to fill other needs as well.
Propagating events is really simple: When a client subscribes to a particular node, the node will replicate the subscription to all nodes in the cluster, and relay any messages from the 'proxied' subscriptions back to the client.
Also, if your network can't natively support multicast, you might want to check out red (https://github.com/nanopack/red) "A fast, in-kernel, ad-hoc point-to-point vxlan network."
I'm most interested in: