Sure...if all the ilities weren't accounted for in the original design.
Virtually nobody is able to account for everything in the original design. Convincing yourself that you got it right is easy. Actually getting it right is HARD. It is possible. For example zookeeper seems to have. But your odds of success are very, very low.
See https://aphyr.com/tags/jepsen for many, many examples of competent people who thought they had it right, being proven wrong. Over and over again.
Failure is not only expected in distributed systems, it is mathematically impossible to avoid. The best that you can do is document your failure modes and what guarantees you will provide despite that.
Unfortunately, as aphyr proves over and over again, is that the guarantees we are given for how distributed software is supposed to work don't hold. Over and over again, across virtually every piece of well-known distributed piece of software. And my experience is that in house software is at least an order of magnitude worse.
However you think your software will work, it doesn't.
Virtually nobody is able to account for everything in the original design. Convincing yourself that you got it right is easy. Actually getting it right is HARD. It is possible. For example zookeeper seems to have. But your odds of success are very, very low.
See https://aphyr.com/tags/jepsen for many, many examples of competent people who thought they had it right, being proven wrong. Over and over again.