Uber reported that their Go code has quantitatively more concurrency bugs than code in other languages, and while to me it seems obvious from looking at Go's concurrency model, this is backed by actual data. Is there any quantitative data to back the claim that Go is better in an LLM based workflow than another popular language?
Matches my experience as well. Go fans have conflated "can easily make something concurrent" with "does concurrency well." Go's primitives for concurrency should almost never be used directly and Engineers below a certain skill level shouldn't be allowed to use them directly ever for long running production code.
As another example, Go still has not yielded a correct implementation of Raft or Paxos while there are dozens in Java, C++, and Rust. Antithesis found some more bugs in HashiCorp's Raft implementation recently[0]. I'm sure etcd still has some kicking around.
Maybe this is a "don't throw the baby out with the bath water' problem but the general evolution of Go has been lackluster. I reach for Rust, Zig, and modern Java instead depending on the specific needs and constraints.
"This Raft library is stable and feature complete. As of 2016, it is the most widely used Raft library in production, serving tens of thousands clusters each day. It powers distributed systems such as etcd, Kubernetes, Docker Swarm, Cloud Foundry Diego, CockroachDB, TiDB, Project Calico, Flannel, Hyperledger and more."
etcd has had numerous liveness and safety bugs, with one happening as recently as December of 2025. Would you consider that a correct implementation?
You may be interested in knowing that the largest managed Kubernetes service in the world (AWS EKS) ripped out etcd for in favor of their homegrown consensus service for large scale EKS clusters: https://aws.amazon.com/blogs/containers/under-the-hood-amazo...
etcd is some of the most amateur code I've ever seen, despite being one of the oldest and presumably most mature "infrastructure" projects written in Go.
goBGP is arguably even worse.
I don't have a third place in mind that's even worth mentioning relative to these two.
No they won't. They are generally faster in throughput than any non-gc application that isn't heavily hand optimized. Their problems are higher memory usage and unpredictable latency, not speed.
Just look at the code. Anywhere you like. There's nothing I can say that will be a substitute for seeing it yourself.
I've had to use it as a library sometimes, and it's also really poorly designed as a library. There's no consistent principle for what is exported and what remains internal, so almost anything non-trivial you do may require you to copy-paste parts of the library code because something happened to not be exported. So you must think, okay sure, but in return for that you get a minimal and stable API. Nope! It also has some of the most API churn of any Go module, already up to /v4 and that's only counting the semver-major breakage they bothered to acknowledge, not the many semver papercuts along the way.
Sorry to pile on, but yeah, I wanted to use etcd during 2021 and 2022, around v3.5, but etcd had serious issues including silent data corruption. If you are curious, ask gemini flash "there were a number of etcd releases years ago where it seems a new wave of developers came in and started breaking everything"
I'd like to know what you base your statement on that the Raft implementations in etcd or CockroachDB are incorrect. Your original paper does not mention those implementations, so where does that claim come from?
Having run a fleet of 100s of etcd clusters for 10000s of rps, and the fact that upstream runs tests similar to antithesis and recently partnered with antithesis [0], and jepsen has tested it long ago as well [1]. Etcd's raft algorithm is fine. Someone even did a TLA+ proof on it in the last couple years[2]. Yes there was a correctness issue a few years ago but otherwise the person you're replying to doesn't know what they're talking about. Also those bugs have nothing to do with the raft implementation, but instead the state machine implemented on top.
"there was a correction issue" is downplaying it. Etcd is truly the worst example of Raft.
Etcd corruption and loss of quorum is extremely common in practice and the GitHub issues sit for years. The design is simple, the performance is modest, yet it still has still never been reliable, despite being marketed as so. I can't speak to whether this is specifically due to their Raft implementation, but I'd argue the entire codebase is over-engineered and questionable.
Their lock, leader election, sessions, and leases are all awful and I'd never recommend anyone to use those. But as a strongly consistent kv store and if you need the watch mechanics, its useful. It has its place and that's mostly being used by kubernetes.
The raft algorithm works and if you implemented a less complex state machine (like using a simpler kv store that doesn't need global event ordering via revisions and watches) it would work. That's what antithesis said they did to test the raft algorithms in the other article linked
doesnt raft have a problem that it assumes no hysteresis? and that in general you can construct a latency graph that deterministically causes a permanent lock in the leadership election phase?
I'm not sure I fully understand your question, but the heartbeat time outs and leader election time outs are static. And yeah if you make votes and heartbeats time out in a way that nobody can be elected, then raft can't make progress.
I've never had this be a problem in reality but AWS has a pretty good backbone. Maybe if you were running it over a pretty unreliable network this would be an issue?
Surely the King is doing it, so that must be the correct way. Look, the King even wears clothes and is totally not naked at all.
That the world runs on Kubernetes is no qualitative statement about the correctness of its Raft implementation. You can say that it's clearly good enough to not matter most of the time, but that is a different statement.
No matter who you look at, they're just cooking with gas like you do, and they can make mistakes in just the same way.
Now; I'm only attacking your argument. I do neither know nor particularly care about the correctness of that implementation itself. There's been better refutations of the claim you replied to in other answers anyway.
I have a bunch of volumes that I'd like to get automatically attached and mounted to nodes on which their respective workload runs (which are automatically scheduled) who automatically fetch and mount their config files and secrets from a HA DB on demand. I also need some internal loadbalancing and integrations with something like certbot for all of my web workloads. Id also like to make sure that I get metrics and logs from every workload in some form.
Thats basically it for starters, what non-technical solution do you propose?
So you propose to just let Ops personnel handle attaching volumes and renewing certificates, that these arent technical problems but just one of team structure?
Please, do describe your solution for the concerns above. So far Ive only seen equivalents of Greenspun's tenth rule
Having maintained multiple etcd clusters for self-managed kubernetes the last few years I disagree with the "problematic" characterization.
Sure it may not be the best fit in a scenario where you want a cluster spanned over the entire globe (thats why GKE uses paxos-based Spanner instead of it) , but even spanned across an entire continent (in europe via glass fiber) it works quite well for me. Its one of the least problematic parts of the stack.
I don't care for Go myself (especially its concurrency model, which is a total dinosaur in a world where we have structured concurrency) so I'm not saying this to support my favourite language, but:
I hope my every competitor will take your advice to heart, as one of our competitors did when they read that "Go is not a memory safe language", so they wrote a blog about how they are porting to Rust. While our team was moving fast and using those "primitives that should almost never be used" around our long running production code base with success.
Some time has passed and now their company does not exist anymore and we have a lot of their clients.
Perhaps that company failed because it chose to port things to Rust and not because of Rust itself? Or any other number of reasons that survivorship bias might be mistaking.
I emphasised too much in that comment perhaps. I was going for because it chose to port things, meaning that maybe that company wasted time working on porting things instead of working on things needed to survive.
Where would one ever read that Go is not memory safe? That's just a false claim, and anyone believing it would have probably gone out of business regardless of choice of programming language.
> Go's internal data structures like interface values, slice headers, hash tables, and string headers are not immune to data races, so type and memory safety can be violated in multithreaded programs that modify shared instances of those types without synchronization.[113][114]
Nobody serious claims Go is fully memory safe. Here's Russ Cox telling you concurrency is a hole in the memory safety: https://research.swtch.com/gorace
By a strict definition of memory safety it isn't - you can tear two-pointer-wide values using data races and cause arbitrary memory issues if you try to using only normal code.
It's close enough for most purposes... but it isn't.
My ex-boss was a JS guy and then moved over to Rust. He loathed Go because it has pointers and it's possible to use a nil pointer if you are not competent.
JS is fine for what and where it is, Rust is fine too. I just appreciate the stupid simple nature of Go and it does the job just fine.
You seem to be implying, based on the rest of the thread, that Go has some sort of special defect that keeps it from implementing Raft correctly. But the "special defect" that Go has is that it in practice implements the same primitives in practice that almost every other mainstream language does, rather than implementing some sort of super-safe concurrency primitive like Erlang or Pony, or being immutable like Haskell. And even those things are of only marginal utility for Raft, preventing some local issues, but the hard part of Raft is more in the logic and the communication, for which none of these languages have any sort of special support or anything that will particularly help you get it right. Of the languages you listed only Rust provides any assistence over the standard mainstream languages, and like I said, in the context of Raft, it is not necessarily all that helpful.
If you want to see something that could potentially impact Raft's correctness, search the last couple of days of the HN front page for choreographic languages [1]. But none of these are even remotely mainstream enough to depend on for anything. Nor do I know if anyone in these languages has implemented Raft. A rather good test case for them, if any of them are looking. That's something that could actually help a Raft implementation's correctness, not just fiddle around the edges of local concurrency issues.
You're misreading what I said. I didn't say other languages don't have buggy Raft/Paxos implementations, just that Go is yet to yield a single correct one.
The intersection of the set of Raft libraries Antithesis tested and all Raft libraries in existence do not fully overlap. I personally have worked on multiple proprietary ones that Antithesis would not have access to.
But they are misreading what I said. My original post is clearly about Go. What they wrote is also ambiguous.
> The antithesis author states:
> "we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft"
What are they implying by citing that? That every language has a Raft implementation with bugs? Yes that's probably accurate because lots of people make Raft implementations for fun and learning. Again, Go does not have a single Raft/Paxos implementation that is rock solid. I have seen many in C++, Java, and Rust that are doing tens of millions of requests per second in production for over a decade.
Is their point that Go is not the only language with this problem? My post already points out the track record is that Go is the problem for writing correct code in highly critical domains.
If we rely on their evidence alone, it suggests nobody has ever made a correct implementation, so we learn nothing about Go. "Go has yet to yield a correct one" is an extremely misleading way to present evidence that says the same thing about every language.
The only way this becomes useful for comparing languages is if somebody gives evidence of correct implementations in other languages. You're claiming they exist but with no evidence and suggesting they're secret. How do you know those don't have bugs? Did any concurrency bug experts do extensive testing on them? And can we disprove secret Go implementations of the same quality?
> That every language has a Raft implementation with bugs? Yes that's probably accurate because lots of people make Raft implementations for fun and learning. Again, Go does not have a single Raft/Paxos implementation that is rock solid. I have seen many in C++, Java, and Rust that are doing tens of millions of requests per second in production for over a decade.
No, they are citing that every Raft implementation that Antithesis has tested has bugs. The etcd implementation you note in go that has bugs also does tens of millions of QPS and is over a decade old. How are you confident that the proprietary implementations that presumably haven't been fully tested don't have subtle bugs that don't show up in practice?
You're being very mealy-mouthed, even here, it reads as "The pre-eminent Go implementation can't even get it right" and yet the implementations Antithesis tested in Rust are apparently people's random "fun and learning" projects, nothing serious, and certainly not all the proprietary implementations that you've used that are all correct.
To combine both TFA with this comment: I find that LLMs are ~fine at generating/editing gocode, or at least as ~fine as they generate most mainstream languages.
But good god, the second it gets to anything concurrency-related, it just loses its mind. As much as it's gotten vaguely ok to try to let the agents loose on some bits of the codebase, they simply can't even do table stakes stuff with the kinds of concurrency you see in real life.
Correct concurrent code is mind-bogglingly hard even for seasoned veteran humans (and don't get me started on distributed programming...), so it's hardly surprising that LLMs with their limited context windows into the code have a hard time writing correct concurrent code
My experience as well. LLMs also struggle with Rust's many abstractions and offerings but you can know that if it compiles it is data race free and work with the LLM to use better abstractions over time.
Zig is also good at this but requires more up front design (thread-per-core, static allocation, etc.) and consistent checks to verify rules are followed.
C/C++ has "compiles but may have undefined behavior". Golang has numerous "compiles but has incorrect behavior" (normally known as footguns). Meanwhile with Rust, if you get past the compilation step, bugs become much much fewer. (You can still have memory leaks, but those are easily traceable).
It seems like claude code can code Rust pretty well with Opus, and I've started moving codebases away from Golang to Rust at work with Opus. Spin up an LLM and it cranks on it for a while, and as a benefit, I get easy apis to build on with other languages.
And that's the problem with Golang really, not that it's a bad language per se (all languages have footguns), but that the language interoperability story is terrible. Meanwhile Rust and Python/C/C++ go great together like peanut butter and chocolate. And I love it.
Logic bugs in fairness happens in all languages. My unproven theory is that the fewer the foot guns, the better it will be for humans and LLMs to generate.
If Rust's footguns are limited to getting past the borrow checker, and memory links with their reference counted memory structures, I'm all for it.
Once I've fought the language once I typically won't have to fight it again. This is my problem with C++.
I wouldn't be surprised if Java has a much better experience here. After all, java.util.concurrent has many great implementations, and Java's `record`s are immutable, as are it's upcoming value types.
Java has so many excellent concurrency containers, plus robust 3rd-party containers like JCTools. It puzzles me why Go communities do not offer such containers.
No thread/goroutine handles for fork/join handling from "outside", and no generics for many formative years that influenced tons of habits, then significantly weaker generics (improving very soon[1]), have all led to most concurrent code to be very "intrusive" - you create bare threads and add bare synchronization primitives (or nearly) by hand inside the threaded code to make it concurrent. `errgroup` is as far as a lot of code goes, in terms of sophistication.
Java leans heavily in the other direction: a lot of concurrency is added externally, without changing existing code, often in very declarative-flavored ways.
E.g. Future<T> serves as a foundation for a ridiculous amount of stuff, while Go forces channels for `select` whether they model your problem nicely or not, and they're very difficult (often impossible) to wrap without changing semantics.
There are very obviously lots of counter-examples for both langs (`synchronized`, rill in Go, etc), and I expect Go to become more Java-flavored in time (it already has moved this direction somewhat, and 1.27 will enable a lot more). But I think it's a fair summary of broad ecosystem habits.
It sounds like Go didn't follow the suggestion of pretty much every CS books on concurrency: favor containers over concurrency primitives.
> while Go forces channels for `select` whether they model your problem nicely or not, and they're very difficult (often impossible) to wrap without changing semantics.
I understand that Go's concurrency model is based on CSP and fork-joins and the primitives like locks, but they are not mutually exclusive with concurrency containers, right? It's okay if the Go team's core philosophy is that channels are the universal abstraction, but I don't get why the community didn't produce 3rd-party containers as robust as JCTools.
No, they're not mutually exclusive and tons of go code uses both to implement some kind of semantic. They serve moderately different purposes because their semantics are quite different.
Select, however, is encouraged very widely, and is used very widely, because it's very useful to be able to wait on any of multiple things efficiently. Select is great, you quickly grow to miss it in languages that don't have it. But select only works with channels, and often that means you're essentially forced to use channels where a mutex is much simpler and more natural. It also means tons of APIs are channel-centric because it's kinda the only non-blocking option (atomics exist, but that's a very different kind of non-blocking, and dramatically more error-prone for normal humans to write).
Once you go channel-centric for things like streams or concurrent RX style stuff, it does work, and some libraries do exactly that. But then you run into the historical limitations on generics, which makes for sometimes unnatural code, and channel performance is usually significantly worse than mutexes (it's still very fast, but you don't want to use it for extremely small things). And you are pretty much required to use those channels directly with select by hand because wrapping channels changes some semantics. And the semantics of those libraries are not mutex-y and are different from what most are already used to, because few other languages are channel-centric.
So you end up with high-level channel-oriented concurrency libraries that people only want to use for large expensive operations, thus are only designed for large expensive operations, which means it's not very common over all, and few develop the habit. E.g. there are quite a lot of goroutine-pool-helping libraries for ~seconds of work, but few functional-flavored generic parallelizing ones for opportunistic use.
It's part ecosystem, part language design, and part language history. You can do most of the Java stuff in Go with enough effort, but it just isn't done in practice very much, and it'll look and feel quite different.
> Go still has not yielded a correct implementation of Raft or Paxos while there are dozens in Java, C++, and Rust. Antithesis found some more bugs in HashiCorp's Raft implementation recently[0].
the source you link to contradicts your own claims.
they say:
> we’ve found bugs in every Raft implementation we’ve tested, including HashiCorp Raft, Aeron Cluster, OpenRaft, and MicroRaft
It's pretty easy to get yourself into trouble with channels: deadlocks, send on closed, channel leaks, deadlocks "fixed" thoughtlessly with arbitrarily-sized buffers, etc.
In my experience, shared memory instills the appropriate fear and caution, while the apparent simplicity of channels encourages novice Go programers to take on concurrency projects beyond their abilities and without due care. Been on both the submitter and reviewer side of that plenty of times in 10 years.
Rust's std::sync::mpsc contains most of the hazards of Go's channel, and arguably adds some because the Go runtime provides deadlock detection that Rust lacks.
The Raft bugs are the wrong kind of concurrency --- they're distsys bugs, not multithreading bugs. Not a good example, and a little telling that you'd cite it.
Just reading the abstract, it talks about finding a number of bugs (across millions of lines of code) but I didn't see any claims that there would be fewer bugs in a different language.
Go obviously does not stop you from writing buggy code. Neither does rust or zig or whatever. Does go make it more likely to have bugs? Or a specific class of bug? Like, the real world is about trade offs.
Uber has fairly large golang and java codebases so they have more of an apple to apples comparison here since they are both GC languages of a similar performance class. And if a large population tends to make more mistakes with 200hp sedan A vs 200hp sedan B, there is probably something up with the design of sedan A.
Which languages are they comparing with? From what I understood, Rust makes stronger correctness guarantees, including with regard to concurrency, but has a much higher learning curve and cognitive load.
The best way to write concurrency in any language is a single threaded polling loop. Goroutines and messages are just as bad as all the other alternatives, which is to say they are a miserable way to write code.
But Go is also perfectly good at single threaded polling loops.
Uber has a history of blaming the tool - in Facebook fashion - rather than admitting their “talent” sucks and they didn’t hire on merit.
They used to blame Python a lot too - Python is slow compared to others but not so slow to matter that much, and you can build other services around it to handle certain work.
Facebook - who chose PHP - used to blame iOS/Obj-c as the reason they couldn’t build a decent Facebook native app in the early days (anyone remember Fastbook?)
What concrete arguments are there to believe in your talent hypothesis instead of their tool hypothesis?
A couple more comments like this from you, and I'll be able to say, "cyanmoonx has a history of blaming the talent rather than bad tools". There being a history like that is neither an argument for nor against tools being bad. And also, don't forget that bad tools and bad talent don't rule each other out.
