You know, it'd have been amazing if TFA has not opened with that video. So instead of clicking the link to view TFA, you went off and dug up the exact same link in TFA???
The main linked article actually does not have that video; the article linked from in the description does have it. Not surprising that someone missed it.
I made a nice little CLI tool for testing the subjunctive in Italian. Claude code spun off a bunch of Claude API calls to generate example sentences with fill-in-the-blank spots for the correctly conjugated verb. Having an AI generate a prompt for and call out to other AIs was a bit surreal!
Nice article! The generated images make me so nostalgic for the early days of AI image generation. DeepDream and others had such uncanny, interesting generations.
Yeah, generative AI used to be wild, alien creativity and not something that made art kids furious.
I wonder if models can be trained for "high-temperature" purposes. I'd rather have a model which can surprise me than one which can predicably produce generic mediocre results. I mean you can run them on high temperature of course, but it doesn't seem like it's optimized for that.
I believe D. A. Jimenez and C. Lin, "Dynamic branch prediction with perceptrons" is the paper which introduced the idea. It's been significantly refined since and I'm not too familiar with modern improvements, but B. Grayson et al., "Evolution of the Samsung Exynos CPU Microarchitecture" has a section on the branch predictor design which would talk about/reference some of those modern improvements.
> Several commenters suggested the original essay was written by an LLM. They were half right. Both that essay and this one were written with Claude as a drafting partner. I directed the argument; the LLM helped with prose. I mention this not as confession but as demonstration: the human brought the utility function, the machine brought the compute. If that division of labour bothers you, I’d suggest the discomfort says more about the Bitter Lesson than about my writing process.
Dammit. “Helping with prose” sounds like “getting a better grade from my English teacher”.
The quality of your prose is important because it increases the effective bandwidth between your thoughts and the reader.
Either the coherent thoughts are there or they’re not. Using an LLM to tune your prose is very much akin to those awful AI-assisted conversions of standard def television to 4K: Inventing details and nonsense structure to fill space.
Very cool idea. Interested to see how this progresses.
One question: how worried are you about over-training on this particular dataset? i.e. instead of generalizing you lean more toward memorization? Obviously you leave out a validation set but since you're meta-optimizing the model itself by its performance on the validation dataset you're still at risk of over-fitting.
yes, good point. right now, it's somewhat hard to overfit because the meta-optimization extracts tiny bits of information. but over time, we will switch the validation set to some other random subset of the FineWeb or even entirely OOD datasets!
The question is not if but when. I hope the project authors acknowledge the problem directly: it is not merely a risk; it is a statistical certainty given enough time. So, what's the plan?
At the very least, track it. How will the project maintainers instrument this?
https://youtu.be/Is8N7B9b0GQ