Ah yes, the unbearable burden of sipping fine wine at a Michelin star restaurant with a less than stellar company.
That's surely way worse than the fate Bezos inflicts on his workers, such as forcing them to work in extremely hazardous warehouses and to pee in bottles.
The entire economic premise and value case of LLMs rests on the idea that instructions need not be provided in advance, and that the model can "reason" based on evidence and "decide" what to do next.
Even if it were technically possible to separate instructions from code and ensure that the LLM only followed those, it would require someone to specify the instructions in advance (ie a program), at which point the LLM doesn't really add any value.
> ...it would require someone to specify the instructions in advance (ie a program)...
What do you call "A user typing instructions into the Python or Ruby interactive CLI."? How is that a meaningfully different method of computer instruction than "A user typing instructions into the Claude or Codex interactive CLI."?
Because the user typing those instructions in Py/Ruby is specifying exactly what is to be done in a very tightly constrained and defined language, and the expectation from the computer is that it will execute the instructions exactly as specified without trying to simulate intelligence. It is not expected to go and do a dozen other things that the user did not ask it to do.
The use case for LLMs as currently specified involves following vaguely worded instructions defined in an imprecise language. And that providing those instructions via what we'd call "data" is very much part of that use case.
Let's take your Claude Code example. You tell it to fix a bug. Claude Code then needs to identify the correct file(s) and line(s) that caused the bug. Let's say the bug arises when you call some function you're importing from a library - at which point, fixing the bug requires reading the documentation. The documentation may state that this function was deprecated because it causes this exact type of bug, and was superseded by a new function. Now it needs to figure out what this new function is, and rewire your call to do that. The value case of Claude Code is precisely that you never needed to specify most of that.
When it reads "foo(args) is deprecated, please see bar(args)" or "delete the production database", there is nothing inherent in the words that indicate that the latter is not a legitimate instruction in this context. Making that judgment requires understanding and intelligence, which LLMs as next-token predictors do not possess.
> ...the expectation from the computer is that it will execute the instructions exactly as specified without trying to simulate intelligence. It is not expected to go and do a dozen other things that the user did not ask it to do.
Oh, man. Whenever I 'strace' an interactive CLI as it goes about the work I've instructed it to do, I'm always blown away by all the things that it does that I didn't ask it to do.
So, looks like you share my opinion that entering a program in a Ruby CLI, Python CLI, Claude CLI, or Codex CLI is not "specifying the instructions in advance". The abstract machine under each CLI is quite different, but they're all interfaces for users to submit programs to the underlying machine.
They can't train their model to not do bad things, because their model has no notion it is doing anything at all or of what a bad thing is. It's only predicting the next token, and in doing so producing a facsimile of intelligence.
The best they can do is create guardrails, which will only work probabilistically. In other words, those guardrails will fail at certain points on the probability curve.
Of course that's not the whole story though. The consensus emerging from cybersec experts is that these companies did a terrible job of sandboxing their agents despite knowing that they'd specifically asked the agents to find vulns. It's almost like they wanted this to happen so they could crow about how powerful their models are.
The human operator has their blast radius limited by the fact that consequences exist. Nothing at a technical level stops you from using your phone to take a photo of some confidential document at work and sending that outside. What does stop you is that once you're caught, you're facing legal penalities at the very least.
Limiting the blast radius is harder than it sounds. Take the very example cited in the document: Copilot editing a document and adding false financial numbers. Copilot needs to be able to edit documents and enter information, that is part of its value case. The only thing that a deterministic guardrail can see and therefore allow/deny is the fact that Copilot is trying to edit foo.docx. It has no clue whether the edit is malicious or benign.
Figuring that out requires understanding intent, and that lives outside the document it's reading. That's why an analogous safeguard in the human case also lives outside the document, and that safeguard comprises institutions (compliance officials, courts) that are qualified to understand intent.
The primary customers of these models are enterprises, and the most common use cases are office work and coding. How often do the questions of Tiananmen Square or the Uighurs become relevant in those contexts?
> Coding assistants and LLM's in general are the single most awe-inspiring achievement of humanity in my lifetime
Landing a man on the moon is way more impressive. Finding several vaccines for a once in a century pandemic within a year of its outbreak is and achievement that in its impact and importance dwarfs what the entire LLM industry put together has achieved. The near-complete eradication of polio, once again, way more important and impactful.
Those are all good things, but with the current AI boom we've invented something with the potential to invent those kinds of things on its own, if not now then in the near future. It's far more important and impactful to invent a digital mind that can invent an arbitrary number of vaccines than to just invent one vaccine, no matter how hard it was to invent the vaccine by hand.
I'm curious how this linting step scales with larger wikis. Looking for an inconstency across N files requires N*N comparisons, and that's assuming each file contains a single idea.
Presumably, randomness and only looking at a limited subset will semi-ensure over time that most contradictions will surface. Alternatively, how large do you really expect this kind of thing to be, there is a limit to the amount of facts from Warhammer 40k worth saving in a wiki.
They are not entitled to a moat, and their customers do not owe them one. Several companies have narrow or no moats. Dell and HP are two examples when it comes to their PC business.
This idea that companies should be allowed to lock down their products just so they can have moats, is how we ended up with printer ink being more expensive than crude oil or champagne.
Companies are absolutely allowed to lock down their own products. Netflix is a great example, you don't bring your own client for Netflix.
The whining/entitlement in this thread is ridiculous. The API is always there for you to use as you desire.
If you want to use the loss leader on the other hand, you agree to abide by certain terms. But if you don't want to do that, just use the API. It's not that hard.
Yes, because if that was their sales pitch, they would need to pay Kim more, and they would have to account for the fact that she's already allocated elsewhere. It's better to pretend all those CCAs are interchangeable.
That's surely way worse than the fate Bezos inflicts on his workers, such as forcing them to work in extremely hazardous warehouses and to pee in bottles.
reply