LLMs just follow scripts learned from human written text. In other words, it could only behave that way because someone has written a story to do so. In short, stop giving them bad ideas ;-)
Modern LLMs are pre and post-trained on order of magnitudes more synthetic data than 'ground truth' or directly human-authored text. But you could say the synthetic data came from a LLM, and the lineage should trace all the way back to human text.
I hypothesize it partially explains why Claude's writing gets more Claudish with almost every model release.
True, but in this context it's worth reminding that LLMs work exclusively from the word descriptions, not having access to the raw sensations.
Everything they learn about emotions is the statistical patterns of how humans react to situations based on their human feelings. There's no direct knowledge from having those feelings themselves.
You can build organisational structures to have the system more or less self-police, without controlling it exclusively from hard restrictions (see https://news.ycombinator.com/item?id=49372089).
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.
I don't think this is a solved problem, there are "misaligned behaviours" in organisations that similarly are supposed to be governed but aren't, or are following an easier path at the detriment to good process or against regulation.
Which is more or less the system the article author was building and benchmarking against (cheating) Sol on Codex. Just that the self-policing with different agents was more about questioning assumptions than about permissions
> Granular access to shell commands to avoid that is going to be an endless game of whackamole as it comes up with more elaborate ways to combine operations.
That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
In the article example, the agent should be aware of the benchmark context and know the implication of solving the task without external knowledge. Ideally it could detect when one subordinate agent has found a workaround to bypass the web access constraints, and discard the 'illicit' results.
There's a design pattern that could be used to build harnesses from that principle, the Viable System Model (VSM) [1]. In short, it recursively organizes a system into functional components with one of three roles: operators implementing a given task, coordinators transferring relevant info between subsystems, and decision nodes tasked with maintaining the integrity and mission of the whole system. A decision node could control the operators and prevent them from overriding the strategic goals or deviating into irrelevant rabbit holes.
Whenever I see posts like this trying to herd a LLM agent through harness structure, I'm reminded of this simple pattern and becoming increasingly convinced that this is the way forward. It makes you feel a sense of respect for the researchers in cybernetic theory in the 1960s and 1970s who foresaw the complexity of today’s systems.
> That kind of control is placed at the wrong level. The proper way to get alignment should be implemented by convincing the agent of your high level goals, so it can self-police and avoid those 'cheats' by itself.
No thanks. I run all my agents with lots of permissions, but I don't want them to have access they have no need for any more than I want coworkers to have access they have no need for, because whether or not you can "get alignment", mistakes get made all the time.
I don't want to have access I have no need for, and frequently tell clients to take away access to things when I no longer need them for exactly that reason.
You have to do that anyway. With a hierarchical control structure at least you have a central point to observe alignment instead of having to study the behaviour of the whole system in detail every time.
> If it "acts delighted" that's because it's effectively telling a story about a person who is excited at the opportunity of accomplishing something more easily.
That's spot-on. It is a mistake to think that LLMs have human feelings. Their behaviour is based on narrative descriptions learnt from human texts, without experiencing those feelings first-hand.
A useful way to understand them is as systems that write stories about human characters. We know the characters are fictional and no one is actually experiencing those feelings, but we can still judge whether the portrayal is realistic or whether it contains logical or emotional inconsistencies.
People like to call LLMs 'stochastical parrots' but I see them as the ultimate 'philosophical zombie'.
They are becoming more and more capable of imitating every single nuance of human behaviour yet they lack the neural pathways to connect those thoughts and behaviours with feelings and self-perception; it's blind imitation all the way down.
The process by which a model seems to generate discourse about deep philosophical questions is, in self-aware terms, equivalent to the knee-jerk reflex or the beating of the heart.
LLMs are terrible at anything systematic. They're incredibly good at anything heuristic, so it makes sense that they can explore wide mathematical spaces fast and converge towards interesting regions.
But ask them to enumerate all the intermediate steps required to create a formal direct proof, and it will loose attention and forget important details as they go out of their input window size. You need to combine them with a proper logical problem solver to get the best parts of both.
> But ask them to enumerate all the intermediate steps required to create a formal direct proof, and it will loose attention and forget important details as they go out of their input window size.
It's interesting how people will comment on LLM capabilities despite clearly not having engaged with frontier models in any meaningful way in a long time
Having models write Lean proofs of mathematical claims is standard operating procedure for any LLM math discovery!
Yeah but the LLM can only handle proofs that hold inside its context window. Proofs for novel theories requiring thousands of pages with dozen millions of steps will need support from external tools to organize the full structure of the formal document; it cannot be done by the LLM inference process alone, which was my point. It would be like asking a mathematician to proof theorems without pen and paper; external tooling is a must, the statistical essential nature of generating content from weights is 1) error prone and 2) not suitable for chains of systematic reasoning that are longer than the attention span.
The proofs will be only as good as the framework for linking successive instances of reasoning.
Not sure how true this restrictions is once you have the agents hammering on at a big code base of formalised proofs.
Proofs stretching thousands of pages are split into lemmas, grouped into sub theories.
What I haven’t seen agents do yet is to develop new ideas for entire such theories. I have usually seen them bite into some existing idea and grinding out related results. But I am less sure than ever that they won’t!
Exactly. As the saying goes, if one side says it's raining and the other side says it's not, it's not the job of the journalist to report both. It's the job of the journalist to look out the fucking window to see if it's really raining or not.
Whichis is fair and good, bc nobody would expect you to rebuild the actual rules of chess from the board alone.
However it would still be useful if archeologists used the board to figure out some games similar to checkers, or go; or if they also have the pieces they could guess it was a combat game like Shogi. Any of those would give you insight about the kinds of leisure that people may get from that board.
I believe the sweet spot that makes it practical and reliable will be combining LLMs with formal verification, although I doubt current hardware is up to the task (yet).
LLMs basically solve the classic Frame problem that prevented general problem solvers to be able to reason logically about the real world; however on their own they are utterly unpredictable and unreliable.
However if the database of weights is merely used as a heuristic to guide the logical reasoning engine to promising regions of the problem space, and the program itself is written to specification directly by an inference engine, the result would be classic software not affected by hallucinations.
The LLM could even help debugging the specifications by pointing out unclear or contradicting requirements, improving the process without compromising the integrity of the result.
reply