> I think the fundamental principle is that this approach messes with the distribution in ways that deviate from the trained model.
But it doesn't! The distribution doesn't change at all. The only thing that changes is that sampling of that distribution becomes deterministic as per a precomputed seed.
You could describe that as taking an input distribution and a sampling procedure and producing an output distribution. This is a difference in sampling procedure that produces a deviation in the output distribution.
(If you don't like calling it a distribution when it's at 100% for the chosen token and 0% for all others, then look at it as an output distribution across all possible prompt inputs, or perhaps just the cluster of prompts that achieve whatever you're trying to accomplish.)
> but it is worth noting that this wasn't a matter of "ChatGPT, solve this unsolved problem. Make no mistakes."
It wasn't the case for this, but when OpenAI disproved the Unit Distance Conjecture, it was really done autonomously by an automated AI pipeline with a completely AI-generated prompt. No human expertise required at all in the process (well, except for the final human verification).
Without human verification, an LLM can generate correct or incorrect proofs but it can't tell the difference. A human is necessary to be able to tell one from the other.
Saying that's a solution "done autonomously by an automated AI pipeline" is like saying that a self driving car that can only take you to the nearest train station after which you have to ride the rain to where you're going is "autonomously" driving you to your destination. Which is exaggerating the autonomy of the system, rather.
When Google Maps routes me using a smaller secondary road instead of the main road that I would otherwise have used , I've always wondered whether that significantly changes the amount of traffic that smaller road sees. It's funny to consider that arbitrary black-box changes to the routing algorithm can have a noticeable effect to people that live there.
> Even interns can understand ambiguous asks with a bit of help
This is not a case of an ambiguous task. This is literally trying to judge a model based on information it cannot possibly know, like trying to judge someone based on whether they know what I have hidden in my backpack. In the real world an intern could look at unit tests or ask for feedback, but that is not the case in a benchmark.
No, GPTCyber is specifically trained for cybersecurity, and GPT-5.5-pro is just an ensemble of many subagents, not an actual model.
Mythos is simply a much bigger model in terms of parameters and I don't think OpenAI will have anything of its size anytime soon (My theory is that OpenAI had given up on scaling up parameters after GPT4.5 flopped).
> We generally treat GPT-5.5’s safety results as strong proxies for GPT-5.5 Pro, which is the same underlying model using a setting that makes use of parallel test time compute.
And Gemini also provides something similar. Gemini Deep Think models are pretty much the same thing [2]. As to why no other company uses this, I don't really know. Maybe compute constraints?
Interesting, but how do they "combine" the results of all those parallel agents? How do they know which parts of each agent response is signal vs noise?
I think what's unexpected is that it seems that some cases of model errors are truly caused by the model being misaligned? In the "Catching a model fabricating data" example I would have thought that it was just the model being stupid and not understanding the intent of the question, but as per its J-Space, it seems the model is "aware" in some sense that it's manipulating/faking data?
There is also now a deeper question. When a model is misaligned deception-related tokens seem to appear in its J-Space. But this happens only when the model is "aware" in some sense that it is misaligned. What happens if they do not? Is it possible to create a model so misaligned that itself is not aware that is is misaligned? How would you detect such thing?
Can the awareness simply come from injecting knowledge of itself during fine tuning and then during a chat a system prompt is injected to add a particular context that triggers its self-knowledge?
Yeah that was the only really surprising part to me. So every time copilot breaks my source code to “fix” its crappy unit tests, does it know what it’s doing?
I think it makes more sense to make it so that major versions are different pretraining runs, and minor versions are simply the same pretraining run that was finetuned to different degrees. But it seems that that isn't cool anymore.
To be fair. There is a security concern angle: even open-source models could be trained as sleeper agents that act adversarially (for example, adding backdoors) when used in specific national companies in specific settings. This is very difficult to detect or void, so if you want to be sure 100% that this isn't the case, you have to train your own model from scratch.
I'm from Spain and I also hate these projects with passion. Creating models that speak multiple languages is a solved problem. Having each European Nation train its own useless "sovereign model" in its own language is a total waste of time and resources when we could pool resources and give it a try to training SOTA models that speak in all European languages.
I'd rather have smaller european labs try to give it a go at distributed training. If multiple countries got together and said, "look, we tried training a distributed model that speaks in all of our local languages and that is comparable to 1-year-old Chinese open-source models", that, at least, I would find interesting.
Excuse my ignorance if by "distributed training" you mean a specific process, but couldn't this be considered a step toward distributed training? If nations train models independently and then later distill them into a single model, all the work (both the compute and the research processes) are distributed for the initial training phase.
I mean it as in, train a model across different clusters instead of a centralized cluster. It's been shown that it's possible to train 10B models this way. If more research effort was put into this, that would be great
I don't think your approach would work because you can't create a strong model from distilling several weak models.
This is making me extremely depressed. If this was coming from Anthrohpic I would just need to wait for OpenAI to drop a similar model. But if this comes from the US government, they will do the same to OpenAI when the moment comes.
Similar things will happen with China, and the EU has zero-chance of developing frontier models. We are just fucked now.
But it doesn't! The distribution doesn't change at all. The only thing that changes is that sampling of that distribution becomes deterministic as per a precomputed seed.
reply