As a competitor (or "competitor," I don't think decentralized services actually compete with each other in the traditional sense), I just want to say:
Matrix is an amazing project with an incredibly hard-working team. If you can afford to, please stop what you're doing right now and give them money. Or code. Or both. We can win this thing!
One of the things that gives us (Matrix) enormous hope for the future is the shift we've seen in the Decentralised Web community over the last few years, where projects who in the past might have competed now support each other.
In the end, there is an incredibly small handful of developers working on decentralised infrastructure, who generally share very similar ideology and goals, pitted against an entire industry which defaults to holding users hostage in silos. We gotta stick together to prevail :)
So, Urbit: huge thanks for the support - we really appreciate it. And good luck on your side too!
I agree - I see this Benjamin Franklin ("if we don't all stand together, we shall all hang separately") attitude everywhere.
But I don't think this new attitude is an accident. It's not just because we're good people, or even good engineers. It's an inevitable consequence of what we're trying to do.
Fundamentally, all decentralized systems are one big federation. If two decentralized systems can't talk to each other, that's because no one needs to connect them yet. There is never a question of whether the connection can be made, as with centralized systems.
In short: for decentralized systems, the n in Metcalfe's law ("the value of a network is proportional to n^2, where n is its user count" - I think it's more like n log n, but whatever) is the number of users not in any decentralized network, but in all decentralized networks. (It also should add any centralized networks that you can gateway to.)
What this means in practice for Urbit and Matrix: if both Urbit and Matrix succeed, a gateway is inevitable. It's inevitable because (a) it will be demanded and (b) no one can prevent it. Then, Urbit will benefit from Matrix's network effect and vice versa.
So economically, we're not competitors. We're colleagues. So building a positive relationship is a professional no-brainer. In the new decentralized world, the only reason to be a jerk to someone else is that you're actually a jerk. You'll see this, but it isn't common -- at all.
(However, our domain is googleable and yours isn't. Owned! We'll just have to link to you :-)
Exactly-once messaging is not a hard problem so long as you change the problem a little. (Plug warning: this is the way Urbit does EOM, or EOM* if you prefer.)
TLDR, you don't need infinitely durable infinite memory. You just need (a) a single-level store in which every event is a transaction, (b) a message protocol with true end-to-end acks, and (c) a permanent session between every pair of nodes. We don't have single-level storage hardware (although XPoint comes close) but it's easy to simulate semantically.
Think about the problem intuitively. I want to send you a stream of messages, each of which you act on exactly once.
I do it like this: I put a sequence number in each message. I keep sending you message 1 until you send me an acknowledgment that you heard message 1. I can also send messages 2, 3, or whatever (lockstep isn't needed), but you process messages in order and ignore messages you've already heard. Never ack an ack, always ack a dup.
What does implementing this design require? It requires a persistent sequence number, on the sending endpoint, for every receiving endpoint in the world. (Edit: and of course another SN on the receiving end.) Of course, for every endpoint you haven't sent to yet, the number is 0 and doesn't need to be stored. This is not a terribly onerous amount of storage.
A sequence number is the piece of state generally known as a "session" in networking theory parlance. Of course a TCP connection is a session. We're simply saying that every two endpoints have a sort of implicit, persistent connection.
Moreover, every message must be a transaction. Acknowledging the message acknowledges that the application, as well as the sequence number, has been fully and persistently updated with the knowledge that the message contains. One way to think of this is that we're adding the latency of a transaction to persistent storage to our packet latency. SSDs are good here.
Furthermore, since in real life semantic messages need to be delivered in packet-sized fragments, a fragmentation model with end-to-end "piggybacked" acks is needed. There can't be a separate message-level ack (the "stack of acks" problem is the curse of the Internet stack) -- acknowledging all the fragment packets acknowledges the message.
You've embedded the idempotency into the protocol, which is nice, but doesn't get around the problems of not being able to do EOM. You also have the case where you sent a message and lost network connectivity before any ack could come back, resulting in you not knowing if your message arrived 0 or 1 times, which isn't surprising since that case is a constant on a network with anything less than 100% reliability.
That's not EOM, that's just the sensible workarounds to use in light of the fact EOM doesn't exist. Obviously a lot of protocols and applications use such things since the inability to have EOM in the real world has not rendered our entire network edifice impossible or useless in real life.
If you define EOM as magic, or as a solution to the Two Generals problem, EOM is certainly impossible.
If EOM means "the programmer doesn't have to think about idempotency," EOM is what I want. Happy to call this "EOM asterisk" if you and/or the OP like.
At any point in the conversation, you don't know whether your interlocutor has yet received the last message you sent. This is because you are talking over a network, rather than over a magic bus.
However, you know that before the endpoint processes each message, it has processed each previous message once and exactly once. I think this is what the programmer wants -- asterisk or no.
Network connectivity failure is best modeled as the limit case of network latency. Of course, when you send a message over a network in the real world, you can't know whether it has been received or not until you get an acknowledgment.
Also, I would say that the worst thing about the inability to do EOM on the Internet stack is that it puts the responsibility for idempotence on the application programmer, then tests that corner case 1 in 100 times, maybe 1 in 1000.
It is not impossible to handle a system that produces rare corner cases. It's just expensive and a pain in the butt.
The more I see from this project, the more I like it. The Pijul team seems just completely serious about building system software right. Also while I could wish for more doc, what's there is a really good overview.
Blatant plug: Urbit (urbit.org) is a general-purpose OS built on the event-sourced model. Urbit's state is a small, frozen function of its event log.
I don't know of any other generalized event-sourced OSes. But almost every principle Fowler describes in special-purpose event-sourced designs also seems to apply in the general-purpose case.
For example, the connection between CQRS and event-sourcing seems very deep and natural. The connection to patches in a revision-control system is also completely apropos.
When Fowler starts pointing at advanced problems like "we have to figure out how to deal with changes in the schema of events over time," you start to see a clear path from special-purpose event-sourced apps to general-purpose event-sourced system software.
Broadly speaking, one category of answer is: abstract the semantics of your app, until it becomes a general-purpose interpreter whose source code is delivered and updated in the event stream. Then the new schema is just a source patch. And your app is not just an app, but actually sort of an OS.
> Blatant plug: Urbit (urbit.org) is a general-purpose OS built on the event-sourced model.
I've looked through its documentation, and while I couldn't make heads or
tails of it, Urbit seems to be everything but an operating system. Espeically that
it needs unix system to run.
Urbit is an OS in that it defines the complete lifecycle of a general-purpose computer. It's not an OS in that it's the lowest layer above bare metal.
Sorry you had a bad experience with the docs. Urbit is actually much stupider than it looks.
Unfortunately, premature documentation is the second root of all evil, so some of the vagueness you sense is due to immature code. It's always a mistake to document a system into existence -- that is the path of vaporware. Better to run and have weak docs, than have good docs but not run.
I agree with you on the documentation thing but, what we see (and I experience also) is that most of the time, when things get serious enough (production), there isn't any available time to document the core of the software properly, let alone document it fully.... "lack of time" in this context meaning either (1) management won't let you stop implementing stuff they need "for yesterday" (everything is critical priority) and document for 10-20 hours or (2) your self run project needs more features and it's always cooler to study/code than to document puzzles you have already solved.
I'm not criticizing, I am exposing that I am part of the problem and lost, and I'd like to know how people handle this on their lives. I want to document more but there's always this rush to implement implement implement and not once, during some catastrophic breakdown of something, management wants everything solved quickly but they expect you to remember instantly of everything you did 2+ years ago :-)
PS: ... and they'll get mad if you say "you didn't let me document this damn thing"
I'd say this problem is best solved not by having no cryptographic identities, but by using disposable identities.
If you bound an identity cryptographically to your real name, that's permanent -- no getting around it. We all make mistakes. However, one of the benefits of Urbit is that you can build a reputation around a pseudonym that isn't linked to your real-life identity.
There's a place for the 4chans of the world, with totally disposable one-time identities. You can do this with Urbit as well -- use a 128-bit self-signing identity (comet). But most people tend to want a little more stability and permanence.
> I'd say this problem is best solved not by having no cryptographic identities, but by using disposable identities.
A problem with disposable identities as usually implemented is that they are too cheap and easy to make. This makes it easy to make a new identity every place you want to be an asshole. The end result is that places get tired of this, and ban such identities from their spaces.
I think what we might need are semi-disposable identities. You can make a new one whenever you want, but it costs something to make a new one, and that cost goes up with each new semi-disposable identity you make.
This is exactly the way Urbit works. (It's sincerely hard to tell from your post whether you've read the documentation, or you just had the same simple idea yourself.)
Anyone can create a 128-bit comet. But once the comet system starts being abused by assholes and bots, signs will go up all over Urbit. These signs will say "humans only" and deny admission to comets -- essentially blocking anyone who has not invested any real stake in their identity.
The normal quantum of reputation and of human identity is the planet (32 bits, like an IPv4 address). Ideally a virgin planet would cost about $10.
Basically, if this cost is far above the amount of money that a spammer can make before a reputation system catches up with the swine, there will be no commercial abuse. This leaves reputation systems to deal with the much easier job of wrangling random creeps and weirdos.
So if you invented this design yourself, yes, it's a very good one...
There are eight billion people in the world. There are 4 billion possible planets. What is your plan for when urbit takes off and reaches internet-scale?
A planet is like a car -- it's not a toy, it's a tool for a responsible grownup. There are about 4 billion adults in the world. And not all of them are even Internet users.
Of course, the population keeps growing. Eventually either people have to learn to share, or the price will go up, or humanity will learn how to stop expanding exponentially.
Think of a planet as a unit of autonomy and reputation, not of personality. The global supply of autonomy is 2^32 units. That's not perfect in any sense, but if the supply is infinite the network is ungovernable, and the value of each unit is zero.
It's also an improvement on a world in which there's one unit of autonomy, and its name is Mark Zuckerberg. Urbit does a lot for digital freedom, but we're not in the miracle business here. No one can do infinity for digital freedom.
> A planet is like a car -- it's not a toy, it's a tool for a responsible grownup. There are about 4 billion adults in the world. And not all of them are even Internet users.
There's a reason why the IETF decided to switch to 2^128 for IP6. Mainly, because there are more computers and devices that need addressing than IP4 can handle.
I have to question the design of Urbit why you chose to go with 2^32 addressing space?
> Of course, the population keeps growing. Eventually either people have to learn to share, or the price will go up, or humanity will learn how to stop expanding exponentially.
Again, the future is "Pay to Play", just like Ethereum? And, is that a population control argument? How would you propose to "stop expanding exponentially"? Chinese method? Ender's Game method, or something else unsavory?
> Think of a planet as a unit of autonomy and reputation, not of personality. The global supply of autonomy is 2^32 units. That's not perfect in any sense, but if the supply is infinite the network is ungovernable, and the value of each unit is zero.
Ah, so you are the "governor", hence why it must be governed? This is literally artificial scarcity on something that could easily have been near infinite
> It's also an improvement on a world in which there's one unit of autonomy, and its name is Mark Zuckerberg. Urbit does a lot for digital freedom, but we're not in the miracle business here. No one can do infinity for digital freedom.
So switching from one master to another makes better "freedom"? I get being against walled gardens. I'm against them as well. But you make no point why Urbit isn't just another walled garden (using funny language to hide behind it no less) with you at the helm.
> I have to question the design of Urbit why you chose to go with 2^32 addressing space?
They haven't. They went with base 256, for easier recognition of 32-bit (and 16-bit, and 8-bit) numbers. You can get a 2^128 identity or a 2^32 identity (or others.)
The fact that 64-bit numbers (like ~novfes-lodzod-sibfes-talzod) in Urbit's base 256 are twice as long as 32-bit numbers (~sibfes-talzod) which are twice as long as 16-bit numbers (~dalryp) and down to 8-bit numbers, which are the most memorable and highly coveted (disclosure: you're looking at a comment written by ~del, aka Hex number 0x25) is something between a feature of Urbit's networking stack, and an unavoidable consequence of doing math and dividing things up between persons that are remote from each other, therefore each needing addresses to be identifiable.
Identities that are made of 128-bit numbers are free, and essentially unlimited. They are basically public key hashes and the risk of a collision is sufficiently low that there is no need for them to be generated by a single authority and centrally assigned. (Note that this does not necessarily scale to the population of the earth and beyond as a solution to addressing graph nodes and efficiently routing traffic between them.) Identities that are made of shorter numbers are limited in number and assigned hierarchically; even if there are a boat-load of 64-bit numbers, it's not half as many, it's ½^64 times as many. Certainly there are more 64-bit numbers than humans on Earth today, or for the foreseeable future.
Whether you consider this to be more problem or solution will probably depend a lot on which side of the spam-wars you find yourself on. Currently there is no Urbit software or infrastructure that I am aware of that algorithmically discriminates against one kind of numbers or another.
You won't expect one authority to differentiate each of 8-billion humans and make sure that each one is granted a separate identity (but not more than one), will you? How about 256 such leaders? Still seems far-fetched, doesn't it... especially given the difficulty of coordinating 256 separate leaders. 65536 very smart individuals probably also cannot be expected to coordinate the identities of the rest of the humans in the world. It would be gargantuan undertaking just handling it when some of these leader-folks have forgotten or lost their passwords (and potentially disastrous for the 0.0015% of the world's population under their charge.)
64 bits of hierarchical ID space is enough for every 32-bit identity (planet) to individually dole out as many 64-bit numbers (moons) as there were 32-bit numbers to begin with. Urbit takes advantage of this roughly to make scaling the network's identity framework to rather immense proportions, happen at least a bit more naturally.
If your 32-bit leader gives a lot of 64-bit IDs to people that turn out to be spammers, you run the risk of being lumped in with those spammers by algorithmic processes. Imagine such a process that tries to discriminate against spammers and keep them from spamming everyone who is known on the whole network, at any scale. It's not a simple proposition, at all!
Do these things also get released if the person holding them dies?
(the reasonable assumption being that only a tiny fraction people will bother or care to actually make arrangements for this)
Or even without death, the idea being urbit "property" is all cryptographically locked tight, correct? So can a planet get lost? If someone loses all their keys or a server park gets droned and the dog ate their backups or something?
Planets can also issue moons. I believe somewhere down the line when the number of planets becomes an issue (a good problem to have, it must be said), it would be technically possible for moons to be given similar functionality to a planet. One could imagine a future in which a household, or even a group of close friends who share a planet that issues each of them a moon to hold their primary identity.
> There are eight billion people in the world. There are 4 billion possible planets. What is your plan for when urbit takes off and reaches internet-scale?
Don't worry! There aren't 4 billion people on this world that can spare $10 on a piece of virtual property. Just don't accidentally solve world poverty and it'll be fine.
That's a good point! One favorite idea of mine is that you have a trusted identity provider, that knows you, and hands out pseudo-anonymous identities. E.g. Facebook vouches User123 is female, 20-25 years old, has been active for 2 years, and is likely not a spammer. This gives you disposable identities and the benefits of proven identity.
The problem with any kind of long-lived pseudonymous identity though is that it is really hard to do proper opsec and keep it distinct from your real identity, but I'm not sure there is a technological solution for that. I mean even your grammar can give you away (stylometry).
Sorry, the server was struggling for a moment. We fixed it.
It's an OS admittedly in the metaphorical sense -- the same sense that your browser is an OS. That is, a platform that runs higher-level programs. Urbit doesn't run on bare metal, though it could, in the same sense that your browser could.
In fact, one way of describing Urbit is "the browser for the server side." In the same way that your browser replaces a bunch of individual client apps with one native client that's a platform for higher-level programing, Urbit s/client/server/. Does that help?
so it is a place that will centralize all my credentials so it can fetch and parse my mail, IMs, stock trades, files, etc to then inter-process that to my liking and possible talk to a client on my phone or such?
why a chain of specialized posix programs would fail for that?
Yes, Urbit is (or at least is designed to become) a personal server which solves this problem among others.
One: a chain of specialized Posix programs would work for that. You would have to be someone who can manage a chain of specialized Posix programs, though.
Two: serving as a general-purpose stateful HTTP client is a subset of what Urbit does, though an important subset. It's also a server on a public network.
The second problem can also be solved with Posix programs, but it pretty much seals the requirement for a trained professional. It is possible to imagine a high-usability personal Linux (Sandstorm, for instance), but not really a high-usability general-purpose Internet server (at least if it implements social apps via decentralized protocols).
(In fact, it's not even clear that trained professionals are comfortable with decentralized protocols -- see under, Google doesn't even want your XMPP traffic, etc.)
Urbit may or may not be the perfect solution, but I think it's clear that if ordinary humans are going to have their own personal servers, they're going to need some kind of new system software.
You still haven't made it clear who is running 'my' urbit programs and why they are doing that.
If they are doing it for free this won't scale. If I am paying them why don't I just pay someone to maintain standard software.
I find this very strange because I love all of the work on distributed and decentralized systems. I am normally excited to read more about a new approach, but I still don't understand what you are trying to do and feel like I am just being fed marketing terms like "The browser for the server side" wtf? Do you mean CGI?
When I say "the browser for the server side," what I mainly mean is the isolation layer between the browser and the OS.
For example, suppose someone designing the first JS environment at Netscape had suggested that since JS is so great, you should be able to make POSIX system calls from it. Or link to locally stored libraries. Or use a language someone had heard of before. I think you'll agree that if this decision had been made, most people would never have heard of JS.
The browser is a second-level OS which provides a service no first-level OS offers; it loads applications almost instantly and sandboxes them securely.
Now, in theory, you could modify a Unix to solve this problem. Arguably, it's a problem any OS ought to be able to solve. We are certainly much closer with containers. But still, imagine what it would take to replace webpages with Dockerfiles. (sandstorm.io is the closest to something like this; definitely check them out as well.)
Urbit's semantics are isolated from the platform in just the same way. The job of a general-purpose personal server is very different from the job of a general-purpose client -- they have almost nothing in common. But the isolation layer over the current systems platform is the crucial element.
form the site, i get the feeling that to use urbit i have to be a programmer and a theoretical mathematician... but i confess i did not waste much time on it yet.
Actually, Hoon is designed as a functional language to be used by people who DON'T have a PHD in category theory.
Although I must admit it does look foreign at first because of the symbols, I found it very easy to learn. And believe me, I am not a theoretical mathematician.
A planet, like ~tasfyn-partyv, has a parent chain going back to the galaxy: ~tasfyn-partyv (32-bit), ~doznec (16-bit), ~zod (8-bit). zod.urbit.org is directly DNS mapped.
Not using TURN and STUN specifically as the RFCs, but in effect it's the same. Routing will STUN (establish a direct peer-to-peer connection) so long as either side has full cone NAT.
As a planet, you need some star to route for you. Right now, the network is small and friendly enough that trust problems aren't an issue. As we grow, we'll put in an escape protocol so that you can switch stars if you have an issue -- there are 2^16, after all.
Urbit is not PGP. It's designed to feel more like the early Internet, ie, a wide-area world of nontrivial default trust. At present, it would be foolish to go full cypherpunk, because we're a long way from being worth attacking.
Hoon is unusual, but I wouldn't call it "esoteric" -- that term is reserved for languages which genuinely don't care about usability.
> Urbit is not PGP. It's designed to feel more like the early Internet, ie, a wide-area world of nontrivial default trust. At present, it would be foolish to go full cypherpunk, because we're a long way from being worth attacking.
It's not as much that. If I am going to be owning a service that I want to use among my peers generally someone has to run the thing and generally I am that person. If I am relying on urbit.org then the goal described in this overview of "Your urbit presents your whole digital life as a single web service. And since it's yours, open source and patent-free, it never shows you ads. Or loses your data. Or updates without your consent." is not met. If urbit.org is so entrenched and the galaxy lookup so well defined how do I use this when you are gone? The galaxy lookup is clearly one of the most critical pieces of the design yet seems the most weak. In the world of online games the lookup service is what has a community live or die, and I did not see a better answer after writing the comment.
It seems that the only way is to recompile and distribute an alternate version of the software.
> Hoon is unusual, but I wouldn't call it "esoteric" -- that term is reserved for languages which genuinely don't care about usability.
I would class it as such because there doesn't seem to be a description (going by the whitepaper) of modularity/code layout/best practice sort of thing. There is also the complexity of symbol management from bash/perl/php land that seems to be in Hoon too. I work with grammars all day and I guess I was just looking for a more formal description of the language.
Your question about the lookup service is an excellent one.
One: it costs 20 bucks a year to run urbit.org in its capacity as a DNS server for binding 256 names to IPs.
Two: at present, there are about 50 galaxy holders, so if between them they can't scrape up 20 bucks a year to keep urbit.org registered, Urbit has worse problems than centralization.
Three: the use of DNS as a root routing table is an implementation detail in the Unix process, completely isolated from Urbit. When Urbit wants to route a UDP packet to galaxy X, it routes to the reserved range X.1.0.0. If the DNS itself collapsed, we could probably find other ways of mapping this table.
I would not call bash/Perl/PHP "esoteric" either -- it has a very specific meaning:
Hoon is a little under-documented and should have its grammar specified somewhere, although the easiest way to do so would be just to clean up the Hoon parser (a parser combinator written in Hoon).
(The original plan was for the Hoon parser to look as pretty as a spec grammar, which many parsers in Hoon do. But looking at ++vast in hoon.hoon, you'll see we fall a little short of that goal.)
Given that the leap-second problem exists, this is the right technical way to deal with it. Unfortunately I've never seen anyone doing this, but they should.
(Of course it'd be ideal if humans were willing to abandon their silly timezones, and just plan schedules in absolute time that made sense for their geography. Alas, not sure we're ready as a species for this level of change.)
TLDR: if you define a bunch of complexity into your definition of time, you'll have a very complex model of time.
Eg, if you define the Gregorian calendar as a function of the position of the planets, rather than a function from atomic-clock seconds to a string, then the Gregorian calendar is not fixed. I suppose Pope Gregory designed it as a function of astronomy, so who are we to roll our own?
Matrix is an amazing project with an incredibly hard-working team. If you can afford to, please stop what you're doing right now and give them money. Or code. Or both. We can win this thing!
-- Urbit