> How much of the identity of a mathematician is tied up in being the first to discovery?
Here is how I'd put it: math has an enormous focus on discovery. It is why we have Godel's incompleteness theorems, the Cantor set, Zorn's lemma, and so forth. We name things after their discoverers.
It is possible that, going forward, no more things will be named after human discoverers. The last such naming (of something truly significant) may already have occurred.
That is a massive culture change, at the very least.
This has already been shifting in science and engineering because typically large teams are needed to get things done. Math has held to it better, but even there they are almost always multi-named concepts, like Rivest-Shamir-Adleman etc. And even those often have to ignore names that could plausibly be added.
For example in classic image processing / computer vision we had things like the Hough transform, Canny edge detector, Sobel filter, Shi-Tomasi detector, Lucas-Kanade optical flow, Horn-Schunck optical flow, Kanade-Lucas-Tomasi tracker, Otsu thresholding, Viola-Jones detector, etc, but nowadays we no longer name things like that, but typically with some abbreviation or some name that relates to the concept more closely.
It's in part the larger teams thing, and also a cultural trend to de-emphasize "great men theories".
> We reuse WebAssembly linear memories under the covers to implement and sandbox the GC heap. A reference to a GC object is not a native pointer, it is a 32-bit index into the GC heap’s underlying linear memory [..] As far as being fast goes, it lets us use virtual-memory guard pages to elide explicit bounds checks, just like we do for linear memories
Array loads and stores still need an explicit bounds check, don't they? And struct loads and stores don't have one anyhow. Are there other bounds checks that Wasmtime is removing? I can't figure out what they mean here.
When WASM people say "linear memory", they mean the whole address space that the guest program uses resides contiguously in a relatively small chunk of the 64-bit address space. In typical native programs, each heap allocation gets an essentially-random 64-bit address, and the can spread over a much larger chunk of the address space, with no guarantees on where a pointer can point.
With the 4 GiB linear memory trick, WASM's 32-bit pointers can't point outside of the range of virtual memory they allocate for the guest, so from the point of view of host, the guest is unable to have an out-of-bounds reference.
The program running inside the WASM virtual machine can still corrupt its internal state, but that doesn't matter for the WASM engine's security model: the program can be assumed to be directly hostile.
They're saying they used a similar design for WasmGC -- concretely, they're 32-bit indexes to a second span of virtual memory, but opaque and with more rules about how they can be used. They're unforgeable by design, but even if you find a bug in the WASM engine, it's still only a 32-bit index.
The quote extends that 4 GiB thinking to the GC design: Even with a bug in the WASM engine, every index is "safe" to access at any time, eliding bounds checks (the explosion is instant, predictable, and contained).
Yes, formal mathematics has such limits. We can't expect machines to be perfect and provably perfect. But the industry isn't assuming that. Why would it? Natural intelligence is not perfect or provably perfect, either.
Rather than certainty, measurement is often enough. We can't prove a program will always halt, but we can check it halts in a specific execution.
Approximation is also often all we need. Even if we can't prove that we can train a network with more than 50% success, if we can get multiple shots at that (using different data, or initial random weights, or training techniques, or something else), then we can reduce that danger exponentially. (I don't know that we have a guarantee of succeeding there, but this would be the hope, and I am not aware of anything showing it is impossible, unlike perfect provability.)
Finally, it is possible that perfect provability does work on the problems we care about. Godel and Turing etc.'s proofs rely on finding rare situations where we can't prove things - cleverly-constructed pathological cases - but perhaps human behavior does not fall into that set. Human behavior may not be a pathological case for proofs or learnability.
It is. This happens to some people who discover undecidability is a thing. Then they get carried away and try to generalize to other areas. Branches of this line of thinking include the concept that intelligence must be analog with infinite precision,
(this is the same line of thinking as the argument against digital audio) or intelligence must be "quantum". (Penrose).
The formal answer to the halting problem is that deterministic systems with finite memory must repeat a previous state or halt. Undecidability requires unbounded memory. Finite memory systems may have so many states and be so complex that decidability is computationally infeasible. But that's not impossibility. I used to get this objection when I was working on proof of correctness systems. We now know that some programs are hard to decide, but many useful ones are not.
If your program is anywhere near undecidable, it's probably broken. Microsoft's position with their Static Driver Verifier was that if 45 minutes of symbolic execution hadn't found a way out, you failed driver verification. This is one of those problems, like linear programming, where part of the problem space is really hard in a computational sense, but most of it isn't.
(I sometimes wonder if Penrose was on to something with his idea that the brain uses quantum effects in microtubules.[1] Penrose is a first-rate physicist. His philosophy is not as good. It's become clear that we can do artificial intelligence with acres of transistors and gigawatts of power, but the efficiency is terrible. We need either a more efficient algorithm or better hardware elements. LLM training now is like a Newcomen steam engine, where something the size of a house had the power output of a modern motorcycle engine.)
> We can't prove a program will always halt, but we can check it halts in a specific execution.
The computer scientist says “The halting problem cannot be solved, I have proven it” and the software engineer goes “30 second timeout, if the answer’s not ready by then we don’t need it”
I think this comparison is interesting because it shows the difference between theory & practice, but it's a little inaccurate. The halting problem being undecidable just implies there is no single universal algorithm that can tell you whether any program at all will halt. In practice we are working in specific domains and can come up with heuristics which are "good enough" for a specific codebase / module, like the timeouts you mention. So the halting problem can be solved for some subsets of programs but not all of them, and the computer scientist would hopefully have realized this instead of treating the codebase generically as "any program".
I don't understand the point at all. Mathematical theorems like godel's incompleteness theorem or the halting problem are applicable to everything including the human brain. If they were a limitation to developing general intelligence then human beings wouldn't have existed.
> Godel's incompleteness theorem or the halting problem are applicable to everything including the human brain.
Maybe. That's if you assume that the universe is a conventional computer. Far from settled and the alternative isn't necessarily just "God does it" - it might be that the universe is a "computer-plus", with some processes that produce results not computable by Turing means. It would be strange, but so is quantum mechanics, so...
This line of thinking doesn’t really go anywhere. You can say “but what if <something outside of an axiomatic system>? Then <axiomatic system> doesn’t apply!”
Well great. Then where are you? You are precisely nowhere. You haven’t disproved the results in the system and you haven’t proved anything else.
The problem with arbitrary real numbers is not precision, it is computability. Most real numbers we actually work with in the sciences are computable - pi, e, 2, 0.(2), and so on - they are all computable by Turing Machines. However, Turing himself showed that not all real numbers are - you can construct real numbers like Chaitin's constant that are not actually computable by a TM. In fact, almost all real numbers are non-computable (the set of computable numbers is countable, so it has the same "size" as the Naturals, while the Reals are not countable - so they are much larger).
However, this doesn't mean that TMs can't operate on such numbers. You can compute the fact that pi + pi = 2 * pi even if you can't enumerate all of pi's decimals.
> However, this doesn't mean that TMs can't operate on such numbers.
Well, it kinda does. For example, you can't compute the sum of two arbitrary real values in a finite number of steps.
That being said, it's possible that the universe actually is not "running" on real numbers. This would also mean that our physics built on top of calculus is just an approximation.
This has always fascinated me. You can't really construct a meaningful amount of calculus without having real-valued variables. The intermediate value theorem depends on it, and it's one of the foundational parts.
I tried to construct something like "bounded error rational calculus", by trying to use rational ranges instead of real values and then constraining the range size. I was able to recover _some_ calculus, but I really have not studied it in enough details.
I find this topic of precision and of the computation that happens in the universe to be pretty interesting, but also very hard to pin down.
For example, let's take two arbitrary real numbers, call them x and y. They're almost certainly irrational uncomputable numbers, of course. Now, what is their sum? It's x + y, of course - I can tell you that without doing an infinity of computational work. Now, is this a useful representation? It very much depends on the use you have for it. For two unrelated uncomputable numbers, their sum is almost certainly uncomputable anyway, so there's not going to be any meaningful representation of it either. However, we can still derive useful properties about it, and use it in physical simulations as we need. For example, if both x and y are positive, I can tell you that x+y is greater than either of them, so I can also tell you that the electromagnetic force between two particles that are at distance x+y will be weaker than the force between two particles at distances of just x or just y.
So, what would it mean for the universe to "compute the exact value" of these two, or of their sum? What can physics do, assuming that some aspects are indeed continuous, that a simulation running on a TM actually can't?
But you have a problem when you want to compare two numbers (to see if the particles overlap). Or if you want to divide one number by another.
In theory, in classical physics if you start a simulation on a Turing machine, then all the initial states must be computable. And you can carry on using symbolic representation of numbers, even with transcendental functions (as long as you compute them to a precision necessary to carry out the next step of the simulation).
But with quantum mechanics, we might have true randomness, so precise classic simulation can become impossible after the first interaction.
Are they? Are there any actual Turing machines? Doesn’t that require infinite memory? I don’t know of a single thing which actually has that, nor one that comes close.
That just means even more limitations, then. (Though the grandparent comment is correct that literally everything is subject to the same Gödel-style limitation on its ability to make negative claims about its own behavior, whether computers, humans, or any other system at all.)
Not infinite memory, a Turing machine only requires "unbounded" memory, which is a way of saying that it will not run out of memory while running its program with the given input. In other words it just needs to have enough memory to run the program with whatever inputs it was given, which is much less than infinity. And this situation is quite common in the real world -- the programs I use on a daily basis have all the memory / RAM they need to do what I want (almost by definition). So in practice I might as well view them as Turing machines.
Finite machines such as FSMs (finite state machines) are a subset of TMs, so yes they are actual Turing Machines. What they aren't is UTMs ... universal Turing Machines that can emulate every TM.
The court system is the system built for dealing with this indeterminism in human affairs. The question is not whether imperfect systems are useful, or whether there are ways to construct a system to accommodate those imperfections - both of those have been answered again and again in human history (yes, and yes, with the same caveats as the first systems - turtles all the way down). The questions are how much effort does it take to reduce those imperfections to a level that renders the system a net improvement, what is the total cost of the system given those required efforts, and who pays the cost of the inevitable failures that still emerge from the system? As sits, the AI industry seems unwilling to put real effort into calculating the first, is certain the answer to the second is minimal, and is absolutely uninterested in being the answer to the third. The basic point here is, given that we know for a mathematical fact that the answer to the first is larger than we want it to be (that’s this whole article), how do we want to address the second and the third?
Their company does data projects. That plus context makes me think they’re talking about internal work process automation type of work, although it also seems like they’re talking about conversational interfaces (chatbots).
I completely buy the “emperor’s new clothes” argument for work process automation. I’m surprised they don’t address AI-assisted engineering, which seems to be going positively for a lot of folks (although I have doubts about its sustainability). I disagree about the success of chatbots, if the problem is narrowly-defined and chosen properly. My previous company built a conversational interface to a vector database and saw good results. (Although, arguably, the vector database was the real magic, and a traditional UI would have been faster and more accurate.)
In general, I think OP is more right than wrong, though, particularly about the AI mania and unrealistic expectations sweeping the C-suite.
Do yo have links handy for AI-assisted engineering going positively? The case I have on my mind of it going negatively is this recent Ford case [1] It's not that I believe it couldn't go positively, of course.
[1] Ford rehires human engineers after AI fails to match quality checks
That Ford story was really misleading. It wasn't about modern LLMs, and the way it was reported implied that Ford had fired and then hired people but if you read closer that wasn't necessarily the case at all - it sounded more like they were re-hiring people who had retired because they needed expertise that had left the company.
> Over the last three years, Ford says it has hired 350 veteran engineers, many of them former employees and others from suppliers, to help address seemingly intractable quality woes that have cost the automaker billions. [...]
> “We had been relying more and more on automated quality systems” and not getting the desired results, Galhotra said. “We brought back technical specialists” and “they hunt for failure points before a part ever reaches the plant floor.”
> That Ford story was really misleading. It wasn't about modern LLMs, and the way it was reported implied that Ford had fired and then hired people but if you read closer that wasn't necessarily the case at all - it sounded more like they were re-hiring people who had retired because they needed expertise that had left the company.
Oh come on, Simon. Are your glasses so rose coloured they are opaque? They fired, and then re-hired because the AI couldn't do the job. They had to re-hire the quality folks. So, no, it wasn't LLMs, it was quality control -- which is a much more well established domain for automated systems. LLMs are a complete shitshow compared to industrial process automation.
Side question -- do you worry about being so pro-LLM when the promises of LLMs are so clearly falling short?
Show me evidence that they had previously fired any of the people who were hired.
Your comment is exactly the misinterpretation I'm pushing back against here.
The time period covered by this story starts three years ago! What kind of LLMs do you think they were using back then?
> Side question -- do you worry about being so pro-LLM when the promises of LLMs are so clearly falling short
I think my record is looking pretty good here. I was early to the "LLMs are useful for writing code" thing, especially with the code interpreter pattern (both write and then execute code in a loop) which I now realize was our first hint at coding agents.
3 years ago is about when the BS "AI can do anything!" hype started. So, it is not a misinterpretation, but an accurate assessment of the hype cycle. Exactly the kind of things managers are bumbling into because "AGI any day now!" for the last 3-4 years. GPT 3.5 was released May 2022. I'm curious how many AI project in industry have crashed and burned during this hype-Summer before the Winter. Speaking of which, whatever happened to that "software dark factory" you were gushing over a while back?
When I said "misinterpretation" here I was talking about people assuming that the Ford story was a simple case of "Ford fired their employees because they thought AI could replace them, then had to hire them all back again". That's not what happened.
A TON of AI projects have crashed and burned in the past few years. This stuff is really hard to adopt at an organizational level, much more so than for individuals.
I've been pretty consistent in saying that I think the dark factory stuff is interesting more as an exploration of the edges of what this stuff can do. It's a great way to highlight the question of how we can get agents to demonstrate their work without reviewing every line of their code.
I think it's not clear from the article why they left (e.g. could be anything from retirement to going to work at another firm/contracting to being fired to switching careers), and likely it's going to be a mix, plus not all were previous Ford employees. Similarly the "AI" isn't clearly defined (but like you I would be surprised if it were LLMs). I suspect though why the article exists (and a possible source of your downvotes) is signalling against "AI", which if Ford wants more expert employees (given their issues), is something Ford wants to present.
As @simonw said, the Ford example isn’t a good one.
As for AI-assisted engineering going well, I think the jury is still out. Here on HN and with the engineers I know, you see people claiming multiples of productivity on coding tasks. But you also see people complaining about drowning in slop PRs.
I think there’s a lot of confounding factors to these reports. The type of work matters a lot: bug fixing good, prototyping good, big legacy codebases not so much, but maybe good for increased understanding. The type of automation matters: aggressive autocomplete good, vibe coding bad, dark factory (vibe coding with fancy harnesses and auto-“correcting” eval loops) questionable.
And then finally, the perennial mistake our industry makes, which is to value speed of creation over maintenance costs. Personally, I think this is where AI-assisted engineering is going to fall down really hard, but the jury’s still out on that one.
Anyway, there’s a really big spread in experiences with AI, that I think chalk up more to all this context rather than religion and belief. OP didn’t address it at all, which I think is a big gap in their essay, but I do think think they describe the executive-level mania pretty well.
> As for AI-assisted engineering going well, I think the jury is still out.
Anecdotally, AI-assisted engineering has helped me flesh out ideas or to learn extremely complicated APIs faster than trying to understand the docs (which usually are labyrinthine). MS COM ones, for example. I can go read the docs but it's easier to get a quick idea of what I need to do if I ask Claude to provide me an example of doing something specific with it, because MS's code samples (particularly their full ones in, say, the windows Desktop SDK repo) have always been annoying for me to wade through because I have to filter out a bunch of noise. I can't (and won't) try to guestimate "productivity" improvements though, but as an assistant AI has (somewhat) helped. I still do all the engineering work though. Along with it giving me tips on using more modern language features for languages like C++.
The Microsoft API docs are a special case where i'd say you pretty much need LLMs nowadays because after a bunch of document format conversions over the years they degraded massively.
If you can find some MSDN CDs/DVDs from the early 2000s, the content is much better (and you can clearly see that the current docs are often missing descriptions and names for method arguments or even entire paragraphs).
> I disagree about the success of chatbots, if the problem is narrowly-defined and chosen properly.
If you can narrow the problem down, then you could design a much better interface for it than a text box and free form text (unless that's the better solution).
As for as AI assisted engineering goes, the thing is that after some time with a project, you already have much of the workflows and routines nailed down as scripts and other various combinations of tooling. And unless it's spaghetti code, you will have various snippets you can copy from for new code. The one thing I've observed about AI projects is that there's often little technical design coherence about them. It's always a kitchen sink of technologies and practices.
> If you can narrow the problem down, then you could design a much better interface for it than a text box and free form text (unless that's the better solution).
Yes, I agree, in that the chatbot we built probably would have worked just as well with a traditional UI, and would have been done a lot faster. But it would have been a lot less sexy (actually important for the bottom line!) and there are future directions that could take advantage of the conversational interface that’s potentially better than a traditional UI.
On the down side, good chatbots are really frikkin difficult to write. These things (LLMs) are not reliable at scale. The basic functionality came together in weeks. Getting it to behave consistently and obey guardrails took months, and even then we had to accept a low level of failed conversations.
> But it would have been a lot less sexy (actually important for the bottom line!
That’s what the author have been saying. They do for nice demos which sell the illusion of having Jarvis in text format, but the usefulness is not really proven. And that may be important business wise. But as far as end users is concerned, there’s not a lot of productivity boost.
They run a consulting company called "Hermit Tech". Their websites has an olde style font.
They boast about "ancient techniques" from books written prior to the year 2000
> For non-executive management who might be struggling to deliver things that feel beyond their control, we have ancient techniques (see: books written between 1986 and 1999) to turn your team into the envy of the organisation, and we can drop in directly to get your team the resources it needs to save a struggling project.
So yeah, of course these people hate AI and everything about it.
No serious company is reaching out to these people for help with their AI project.
That has ever been the case. As soon as it works reliably, it's not "AI" any more. Take spellcheckers or collaborative filtering as examples, but there are lots more. Hofstadter in G.E.B. said it well:
> There is a related “Theorem” about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of “real thinking”. The ineluctable core of intelligence is always in that next thing which hasn’t yet been programmed. This “Theorem” was first proposed to me by Larry Tesler, so I call it Tesler’s Theorem: “AI is whatever hasn’t been done yet.”
In this moment, the opposite is happening. Everything is getting called "AI", whether it uses LLMs to prompt LLMs about how to prompt LLMs, uses "conventional" machine learning, or just looks mysterious enough that they can expect the market to not ask questions.
I am reminded of "game AI", which for the most part has historically been just giant decision trees, encoded one way or another, because if you hook up any sort of real AI to a game entity or collection of game entities that does any sort of learning or training, even simple 1980s-era reinforcement learning, it turns out the game entities will roflstomp the human players, and the human players aren't interested in paying for that experience. We've been calling those collections of if statements and for loops "AI" for a long time, though, because who wants to hear about how deliberately stupid their opponents are?
Sure, or maybe the actual applications of “AI” are small and unobtrusive, like the dictation of doctor’s notes example, and it’s not actually the massive revolution it’s claimed to be.
I struggle to reconcile this attitude with, among other things, the massive number of exploits being discovered and the pretty straightforward utility for coding. Have you used codex recently?
Perhaps, but I think what people mean by intelligence is something that learns and adapts. If LLMs couldn’t do in context learning I don’t think people would think of them as AI, more as a kind of queryable database via natural language. There are other algorithms that learn and adapt, but in much more narrow circumstances that most people won’t obviously interact with.
Right, and my point is that now that is the bar. If you'd told me in 2008 that markov chains would be smart enough to one-shot even a trivial video game from scratch I would certainly have called that AI even if you had to do some major rigamarole to get it to work.
Because AI meant "make computers smart like people".
We taught the computer to spell check, but the computer still didn't feel smart like a person, just smart like a machine so that obviously wasn't AI. We taught it to do algebra, same thing. With LLMs though, now it really does feel like an artificial human, so this time it really is AI.
Right, but I remember when spellcheck first started to be a thing, and people were like "Wow proofreaders are out of a job, ai is so cool", but then it became normal.
> The Zig standard library also has a general-purpose debug allocator. This is a safe allocator that can prevent double-free, use-after-free and can detect leaks.
>Never reuses memory addresses, making it easier for Zig to detect branch on undefined values in case of dangling pointers. This relies on the backing allocator to also not reuse addresses.
But it's not really clear what this means. "branch on undefined values" would I think indicate that maybe they're doing a fill pattern that the compiler can detect at runtime when dereferenced? But I don't see it in the `free` path. It's not clear if this is deterministic or not either.
I'm not sure what "branch on undefined values" means there, yeah, but never reusing memory addresses is enough to prevent use-after-free.
Or, rather, you can use a value after freeing it, but it will not be exploitable, because it will contain valid data of the right type. This is the same idea as Type-After-Type,
> Do we see signs of reasoning or is it anthropomorphism?
This is the part that so many folks just don't seem to understand (probably because it's been labeled as "thinking" or "reasoning" mode, and people assume that words have meaning). It's not reasoning or thought. It's spewing tokens pretending to "think", but it's actually just generating extra "context" to help the final answer be more coherent. The model isn't doing anything it doesn't already do. It's just doing more of it to improve the quality of the final answer displayed to the user.
You're describing a process by which a 'thinking' entity uses cognition to refine a solution to a stated problem. That's a lot of words so usually we shorten this to 'reasoning'.
Do LLMs 'think'? I 'think' they do in a way. I don't really know how I think myself but I know I do and therefore I am (thanks, Descartes). I have a somewhat better grasp of the way LLMs 'think'. They do so sequentially, building a chain of descriptors which best fit the problem and the preceding descriptors. I suspect I do something not entirely dissimilar- i.e. I imagine 'worlds' which are like the current one changed in some way so they the problem I'm working on is reduced, then refine those until it is resolved - but in a massively parallel way.
Fine. Whatever. I give up. LLMs think. Believe what you want. I literally no longer care, and this argument is beyond exhausting. Go ask the LLM to explain itself to you. It will happily spew out a pretty solid explanation of the details and math involved if you ask it the right questions in the right way. It'll also happily play along with you if you want to roleplay that it is an actual thinking machine. It's designed that way. But hey, whatever. It's a thinking intelligent machine and we're all doomed. I accept that my many decades of working with and learning about computers was wasted and I know nothing about them at all.
> "You're not making the point you think you are."
I don't care anymore. I'm not going to bother with this discussion anymore, not with anyone. I realize now that people want to believe what they want to believe and they don't care about facts or reality, so why should I care either? It's not worth my time or stress to give a damn anymore. I'm done. I'm not going to respond any further on any of these threads, and I'm probably done commenting in general. It's just not worth it anymore. I'm gonna go back to doin things I actually enjoy doing now. Y'all folks have fun. I genuinely wish you all the best.
I've read a bit through your comment history which gives me the impression of a mostly rational individual with whom I agree on some things while disagreeing on others. You don't seem to be a raving anti-LLM crusader nor come across as a starry-eyed LLM fanboi. I therefore conclude that these last rants were, to speak with Ebenezer, the result of an undigested bit of beef, a blot of mustard, a crumb of cheese, a fragment of an underdone potato rather than a farewell.
Assuming this to be the case my real question is: what makes you so sure these things don't "think"? This question can only be answered if we first know what "thinking" actually entails. Sure, LLMs are mechanistic and deterministic, feed them the same quote and seed and they'll produce the same output, token for token. If what they do is "thinking" - albeit mechanistically - then it seems to give lie to the concept of free will since the output for a given input only depends on the seed value. Surely humans don't 'think' like that? Well... who knows? The 'neural network' in human brains is far more complex than the ones used to run LLMs while LLMs can have access to more 'factoids' than the average human. What comprises 'thinking' as we do it? What would happen if you give, say, the neural circuitry in a rat brain access to enough storage to contain the training data used in current LLMs? Can a machine ever be made to 'think' or is that something which will always be limited to living organisms? If the answer is 'yes' we're back at the definitional question of what 'thinking' entails, if it is 'no' we're entering more in the realm of metaphysics and religion.
I don't know what 'thinking' entails, I just know I do it. I therefore can not definitely state whether LLMs 'think' or 'reason' but I can apply reason to what I observe and know about how these things work. Those observations and that knowledge lead me to conclude that, absent some metaphysical or religious veto these models can be made to 'think' and might already be doing so.
You phrased it better (and with a lot more patience) than I ever could muster.
So many people I meet are so deeply convinced LLMs absolutely cannot physically think, because they define "thinking" as "that thing you do with your human brain where neurons are involved", and they define "LLM thinking" as "that thing ChatGPT does where it says it's thinking but it's actually just detached inference".
The underlying assumption is usually two-fold:
1. That simulated thinking is not thinking.
2. That "LLM thinking" is always only defined as Chain of Thought.
Well, 1 is a pretty useless stance to have, because it removes space for any useful definition of what thinking is. And 2 is simply false, as presented by Anthropic here: https://www.anthropic.com/research/global-workspace
To be 100% honest, I really don't want to have this (or any other) conversation with you (or anyone else) right now (or possibly ever; I'm just sick and tired of people and their hateful bullshit), but purely out of respect for your fair and balanced reply I'll try to respond here in a civil and honest way to the best of my ability. This really is the last response I intend to make to anything anyone says on the topic, so I'll try to be thorough. If you want to respond to anything I say here, feel free, but don't expect a reply after this one (I may possibly, but probably not) . That having been said, here goes...
> ... "impression of a mostly rational individual with whom I agree on some things while disagreeing on others."
I try, really I do. It's gotten really hard these days. You're welcome to agree or disagree; Totally normal and expected. I just get tired of getting shut-down on every little thing I say by so many people who have sometimes less than zero experience in the topic they claim absolute certainty about, no matter if I can trot out a parade of facts proving my points. This inevitably leads to stress that is no longer as easy to just "brush off" as it used to be. Sorry for that.
> "You don't seem to be a raving anti-LLM crusader nor come across as a starry-eyed LLM fanboi."
You're right. I'm neither. I am actually quite impressed and amazed with what LLMs are capable of (especially in the hands of skilled and knowledgable users) but I also understand fully that there are tradeoffs involved and responsibilities involved in the usage of such tools. I do believe they (and other "AI" related technologies) have huge potential for both good and bad (largely dependent upon the user and their intent) and like any new tool, I genuinely do hope this one finds more of the good use than the bad, but more and more I'm feeling like it's just gonna get weaponized against society at large. Sad, but nothing I can say or do will change it. I'm fully convinced of that at this point.
> "what makes you so sure these things don't "think"?" ... <more stuff said here> ... "If the answer is 'yes' we're back at the definitional question of what 'thinking' entails, if it is 'no' we're entering more in the realm of metaphysics and religion."
So, in my mind, "thinking" is a much more "active" process than the "calculation" done by a machine just mechanistically working through a bunch of math. Does a desktop calculator "think"? Does a mechanical device like an Abacus or anything else that can "do math" without electronics? Calculation isn't necessarily "thinking", even though thinking can (and often does) result in calculation.
Now, where I'm coming from with my assertion that LLMs don't actually think is due to a few factors. First off, I've been learning the mathematics involved in how these things work for a very long time (decades now actually; as "neural network" technology and ideas is truly not a new thing), and while it's really amazing stuff, it's not magic. It's just math. Really fancy and complex math, but still just math. As soon as the math stops being done, the "thinking" stops. Does a brain ever stop thinking? I get the impression that until death it's kinda always active, even when you sleep. Not so with an LLM. You give it input, a buncha fancy math gets done by a really powerful "calculator" (computer), it responds with output, then it stops until it gets another "trigger" to start calculating some more.
There's some very real flaws in seeing that process as thinking however, even if you're only talking about that time during which the calculations are taking place. The problem I see there is that the LLM cannot "second guess" itself or worry about whether it might be incorrect about something. It just forges ahead with the calculations and gives the end result to the user, right or wrong, as it was designed to do. It has no "skin in the game" or reason to care (even if it had the ability to care) and it's got no real sense of "self" or the world or anything. It's just doing some really amazing math that results in an illusion of a thought process.
That having been said, I'm firmly convinced that even as these things stand now, they can absolutely assist humans in their thought processes if used properly and judiciously with full understanding of their limitations and weaknesses taken into account. I just don't believe that "more of the same" will somehow magically become "sentient" someday without a huge advance in both the hardware and software technologies it's built upon (on the level of the "positronic brain" or some kinda hand-wavy "quantum technology" science fiction concept). Pretty darn certain that more massive "AI data centers" aren't gonna lead to a "magical thinking machine" with the current forms of "AI" we're working with.
> "Those observations and that knowledge lead me to conclude that, absent some metaphysical or religious veto these models can be made to 'think' and might already be doing so."
Now, this here I can actually agree with, other than the "might already be doing so" part. They're not (yet). I'm really quite sure of that, knowing what I know about how these things work. They really are fantastic at faking it these days though, as evidenced by how many people truly are buying into the AI company CEO hype about AGI/ASI. I think that LLMs can absolutely be one part of a machine that's capable of a simulation of "thought" that could really be good enough to qualify as some form of "the real thing" on some level, and that may even someday (soon even?) surpass the capabilities of humans in that regard. It'll require some different ways of doing things though, and some combinations of classic traditional computing with a wide range of related "AI" technologies including LLMs, neural nets, vision models, etc, etc, and it'll have to be put in some sort of active state of operation where it's capable of doing the "thinking" and "learning" process continuously the way an actual brain does. I think it'll also help to give it access to a continuous input stream similar to how a brain has access to near constant input as well.
Anyone that wants to really know how this stuff works "under the hood" is welcome to ask an LLM about it. Many of 'em are actually quite good at explaining themselves, starting from "first principles" if you ask 'em to "keep it simple" all the way down through the deep mathematics involved. I encourage folks to have that discussion with several of their favorite LLMs if for no other reason than more knowledge about the topic is a good thing. Just be aware that they can at times say things that are actively incorrect and they will often say such things with great certainty (and sometimes even try to argue with you about it if you call them out on it). Always check your own (and the LLM's) knowledge against known verifiable provable facts. This stuff is all heavily documented and readily available "out there" on the Web with not too terribly much heavy searching required.
To summarize; I don't think it's impossible to create a "thinking machine" using these technologies. I just don't believe we're even remotely nearly as close to it as the AI mega-corporations would have us all believe. I might be wrong about everything I've said here, or I could be 100% correct. Dunno; No longer care either way really. I've said my piece and I'm done now. Bring on our AI overlords, for better or worse. I can't stop it either way.
I don't know why you're saying you're "tired of people's hateful bullshit". I originally responded that your point was off, and you're the one writing these comments: https://news.ycombinator.com/item?id=48886350
Take a breather, my guy. Stop saying in every comment that "you're done" and "this is your last response" and actually go touch a bit of grass. You do not owe anyone on this website a response, and the only thing your comments are for (when they're good and not hostile) is contributing to a discussion. Nobody here cares whether you respond or not.
Anyway, not to detract - this last comment was good. I just think maybe you're putting intent in people's mouths where there is none. For example, I certainly don't believe the bullshit Altman and Amodei are putting out. I just disagree the LLMs don't think.
And yes I also understand the math behind them. But it being math doesn't mean there cannot be emergent behaviour, just like there is emergent behaviour after the layers upon layers of biology in humans, resulting in thinking. "It's not magic, it's just [biology]" applies to us as well.
"But brains are the most complex machines in the universe" some might say -> Right, but who's to say the thinking we do requires machines as complex as our brains are? Our brains move muscles, help us breathe and manage millions of other invisible innate processes, many things LLMs do not need to do. And LLMs have a very, very deep access to and understanding of language, which is argued to be a significant contributor to how humans think (based on studies on nonverbal humans).
> As soon as the math stops being done, the "thinking" stops. Does a brain ever stop thinking?
Right, so I understand your point in this and there's something to that, but it enters in how thinking is defined in both ways.
First, the calculation not running is at best equivalent to time stopping. It's not like the LLM actually sits there waiting for input. As you said: we continuously receive input, and we do so because our biology is built this way and we are damn energy efficient, so we can afford to continuously and asynchronously process that input. But this is not really related to the process of thinking itself.
So it comes down to what happens during the calculation. And Anthropic's published research on the J-space is pretty damning evidence, IMO, that thinking does happen during the calculation.
Yeah, you can just fuck right off, too. My reply wasn't for you. It was for the person who wasn't actively trying to rub salt in the wounds and be a worthless little shit.
> I don't know why you're saying you're "tired of people's hateful bullshit". I originally responded that your point was off, and you're the one writing these comments: https://news.ycombinator.com/item?id=48886350
You really can't figure it out? Did I not make myself pretty clear in the comment I was responding to there that it's decades of shitty people like him (and apparently you) that have pushed me to this point? Funny how you point to my comment without noticing or acknowledging at all the fact that it was a response to another hostile comment that was all about just tryin' to be hateful and ignorant and push me even further into hatred.
> Take a breather, my guy.
I'm not "your guy". I actively hate you.
> ... and actually go touch a bit of grass.
And this sorta shit right here is why I hate you and everyone like you. You know what I did yesterday? I turned my computer off and went and "touched grass" (hung out in the park away from technology and people) and just chilled out. You make it sound like I don't love nature or some stupid shit. You don't know me, so just keep your unhelpful advice to yourself.
> I just think maybe you're putting intent in people's mouths where there is none.
So it wasn't your intent to make snide little insinuations and remarks in an attempt to purposely try to rub me the wrong way so as to piss me off even more after I'd finally managed to calm myself down? Yeah, sure. I believe that... It wasn't me who put those remarks in your mouth. You chose to try to rub salt in the wound. Congratulations. You're human garbage. Good for you.
Why? It might help me but it won't fix you and you're the problem here. Your whole intent is to aggravate me into hating you even more than I already do. Well, guess what. You're wasting your time at this point. I couldn't possibly hate you more than literally wishing you didn't exist at all. You've already succeeded at your goal, so you can just shut the fuck up now and go away. Congratulations. You win. Be happy and be gone.
If you really didn't care you could have just stopped replying instead of making sure you got in the last word.
A single reply and you folded, unable to provide any counter of any sort. Why even bother at all if you're going to throw a tantrum at the first sign of disagreement ?
Honestly, people need to get over this debate. It's pretty irrelevant in a lot of cases. When people ask "what is the model thinking?", they're really asking "what caused the model to produce this response (as opposed to a bunch of other plausible ones)?"
Whether it's thinking or word prediction or whatever you want to call it, people are trying to understand the causal chain.
It's not just a nominalistic debate though, as the people who are vocal against the idea that LLMs might "understand" or "think" also claim that because of this, they are fundamentally limited in what they can achieve, in contrast to human beings. Therefore any possibility of actual intelligence (or even superintelligence) is, according to them, just a fantasy.
The papers linked in the article discuss the mechanical operations that simulate reasoning. Intelligence is data efficiency and I don't see a strong argument that reasoning can exist if it requires a world's worth of data.
That doesn't mean that simulated reasoning isn't useful, it's wildly useful. But a thing is not its simulation.
"The King leaned over, looked and saw, yes, the Middle Ages simulated to a T, all digital, binary , and nonlinear, and there was the land of Dandelia, The Icicle Forest, the palace with the Helical Tower, the Aviary That Neighed, and the Treasury with a Hundred Eyes as well, and there was Ineffabelle herself, taking a slow, stochastic stroll through the simulated garden, and her circuits glowed red and gold as she picked simulated daisies, and hummed a simulated song."
"In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography.
"Suarez Miranda,Viajes de varones prudentes, Libro IV,Cap. XLV, Lerida, 1658"
My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
not sure your criticism makes sense though - they did this pre chatgpt. they are talking about the language models of that time. they did not make predictions about the future.
They made a claim about language models in general, not just ones that had been released so far.
The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.
With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.
They did use RLHF at the time, at which point it is not a pure probabilistic representation of the training corpora. Bizarrely, RLHF never came up in the paper.
We’ve know that since 1943 when McCulloch-Pitts came up with the first “artificial neuron” definition. And since LLMs are a descendant technology — our assumption should be they’re reasoning in some internal learned logic.
This is what the evidence supports — eg, the “stochastic parrot” crowd never can explain transfer learning. Whereas for the internal reasoning crowd that is easy: removing your top level judgments from a theory still leaves you with useful terms for describing a new theory — eg, removing your judgments about “which animal is this?” but preserving the underlying structure for representing an image in your new judgments, “is this cancer?”
There’s 80 years of reason to think DNNs reason and zero support other than “sTaTs R mAgIc!” to support the stochastic parrot interpretation.
Here is how I'd put it: math has an enormous focus on discovery. It is why we have Godel's incompleteness theorems, the Cantor set, Zorn's lemma, and so forth. We name things after their discoverers.
It is possible that, going forward, no more things will be named after human discoverers. The last such naming (of something truly significant) may already have occurred.
That is a massive culture change, at the very least.
reply