Even before LLMs, ML folks were already aware that you can use a model to teach another model. I doubt this is something AI companies put at the top of their investor materials, but it's been nice to see it play out.
That said, there are other moat factors like, a US company needing to use a US AI provider, sticky customers due to corporate onboarding friction, and others. Not nothing, but not as large a moat as some imagined.
Yes, but 10 or 15 years ago, I would have thought that there'd be more to it than just a slight modification on the ideas behind a CNN to get this level of AI.
There were somewhat good reasons to think it needed more than just this data-driven ML approach.
There's something startling about how (relatively) simple these networks are and yet how powerful they are. The main ingredient the AI darlings are using is vast amounts of compute and data. I don't want to take away anything from what the researchers came up with, but I suspect even they are surprised at how capable some of these models have become.
I'm unnerved by how alphazero is more complicated than the "intelligent llms"; it has at least multiple heads and MCTS, a search algorithm. The LLMs seem to just be monolithic (if complicated) architectures where tokens go in the bottom and tokens are spit out at the top.
early on there was a lot of talk about "emergent behaviors" in the models where they were good at things that were unexpected or did not align to the training data. IIRC doing arithmetic is one example from early on. I think this is where the AGI craze took off, the labs were throwing more and more data in the training to see what other behaviors would emerge. The thought was with enough data and enough parameters AGI would surface on its own.
Then i think tool use became a priority or at lest a sibling priority to more data/more params. Along with multiple specialized models communicating with each other which is sort of a special case of tool use. That pretty much brings us to today.
this comment is delusional. LLMs are awful at arithmetic and AGI is still very sci-fi, otherwise Claude would have told Anthropic how to cheaply generate energy for it to justify its existence by now. As long as the energy use debate persists you can be assured AGI has not arrived.
It is really an anti-war poem rather than anti-humanity or about nature in general. Sara Teasdale was forced to write her sentiments obliquely. From Wikipedia:
> "The Sedition Act of 1918 enacted two months before the original publication of "There Will Come Soft Rains" made it a criminal offense to "willfully utter, print, write, or publish any disloyal, profane, scurrilous, or abusive language about the form of the Government of the United States" and forced Teasdale to express her opposition to World War I "obliquely" in what might appear to be a pastoral poem."
If you think about it, Bradbury's story is also anti-war.
If you read the Influences section just under section you quoted from Wikipedia, it seems she was under the influence of the idea of an 'uncaring universe' from reading Darwin.
And yet there are humans that treat animals well (e.g. dogs, cats, crows and so on... ) These animals would feel the loss if humans disappeared. So i don't think it's right to say that nature (animals) wouldn't care and that the sentiment is a bit of implied put down of animals capacity for emotion and caring. Animals can care about their own and across species. Humanity would do well to feel less disjoint and separate from other humans and animals. That alone would reduce conflict a great amount and highlight the good that can be done between humans and animals.
As you probably noticed, I'm not fond of making a big distinction between humans and animals or nature. Hope that gives another perspective, and yeah, she probably was against the war and upset about the way humans were behaving at the time.
Thanks for the reply. I agree with you that humans are animals and also part of nature. I don't think there's an implied put down of animals or nature in the poem. I think the author was talking about how meaningless and tragic human endeavors are, such as war, when framed against the rest of nature. While (some) animals would briefly miss us -- and yes, some would struggle to live without mankind, having become dependent on us -- this would be brief, and nature would continue without us.
Is that really true? I was under the impression they needed to track the phase of the grid and thus would be able to detect if a fuse is blown or the grid goes down and shut themselves off.
I wouldn't know what to tell someone about what entering a (the?) void is like, but the closest i can imagine is a situation where you're 1000 nautical miles out to sea in a small sailboat on a pitch dark cloudy moonless night, far away from anyone. A storm is brewing and the seas are getting large, though you can't see them, it's too dark for your eyes, though you feel the force as it throws your small boat around. It's more than you bargained for, but there isn't anything you can do about it. You're in it now. You're losing control of the boat. You're afraid of what will happen if you lose control, the seas grow larger... The rains and the lightning and thunder are here and it's just too much too bear, so you go down below and lay ahull, surrendering to the sea and forces beyond your control. The sea will make of you what it will, you're fate rests with something larger than yourself. A part of you dies that night, and what emerges the next day is different from what was swallowed by the storm. Your old self at the helm, somewhat unrecognizable. And a part of you never made it out of the storm, remembering that you're always out to sea, and that you needed to come out here to remember that.
Thanks afro88. I was inspired by the storm in Terence's story, and my time on my sailboat when it felt like the sea was getting to be too much. Glad to share.
I can. An OBE feels utterly real, minus the "oh shit... this is happening!" aspect of it all. It is not like a dream, even a lucid dream. And like a cliched cinematic experience you lift out of your body intact, as if there were two of you, one there and one here. It is legit and strange and the 'reality' of it is a lot to handle, after the fact.
Remember the passage of the Argonauts between Scylla and Charibdis?
Scylla would devour the crew alive by its dog heads, leaving the ship intact but drifting on its own - that's a metaphor of madness: for the crew is the mind, and the ship is the body;
Charybdis would suck both the crew and the ship into its whirlpool - that's a metaphor of death.
Sounds very much like my breakdown 20 years ago. When I finally emerged 10 years later I was both the same and utterly changed. I liken it to having had my personality deconstructed and then reconstructed from the ground up. I wouldn't wish it on anyone though I'm now in a much better place.
I'm pretty sure these large models are run on Nvidia GPUs, not some unobtainable piece of secret kit. You could go down the street and buy from AMD or a number of other vendors to push out FLOPs if you wanted or needed, but you'll need a thick wallet to shell out for a cluster of GPUs to run these models. The reason people don't run the big Chinese models at home is that they can't afford the hardware, not that it isn't publicly available. This tech is essentially a large amount of matrix multiplications afterall.
I think the larger problem is that restricting US AI companies gives the Chinese a leg up because they now have a window open where they can become the source of the most powerful models available due to government restrictions rather than on technical merits. All Anthropic customers just got a downgrade last evening, for example. While the Chinese are able to serve the world or whoever, the US corporations will be limited to the US market, or whatever the powers that be will allow. This restrictiveness could turn out to be disadvantageous to American companies since people will migrate to wherever they can get the most powerful models.
I think you're over analyzing to some degree. The distribution and median outcome (1st through N order) was always negative for the course of action this administration has taken. The proponents try to sell people on the notion that this could all turn out great, which is way out on one end of the tail (e.g let's say 1% chance for sake of argument) for this action, and here we sit right around a median realized outcome for this kind of an intervention. I'd bundle all the N order effects up, then look how an aerial bombardment operation affects the liklihood of outcomes like the straight of hormuz being closed and/or controlled by iran, or iran surrendering the nuclear material and raising the white flag, etc...
You could probably do some simulations to see this was almost always going to be a losing strategy. Detailing the N order effects is good accounting, but the picture likely gets murkier the more you try to extrapolate N+1.
It appears that right now the administration is fighting desperately to achieve an international state like the one had under the prior nuclear deal with Iran... the one Trump tore up because it had Obama's name on it.
It's all petty BS and I really do hope the electorate gets it together.
It's a bit surprising many here are referencing the memory supply cycles without mentioning this revolutionary new application for memory people call AI. It'd be like talking about weather cycles and climate without mentioning global warming. Just like the planet is not going to get colder on average than it was for the foreseeable future, we're not going to need less memory for the foreseeable future.
China is trying to be vertically integrated, completely independent of outside influences and own the future supply/means of production. Memory and chips are piece of their larger plan. In any event, whatever China brings online will be absorbed - imagine a future where people's home computer has 100 to 1000GB of RAM in one variety or another. Folks are going to want better chips and more memory for years to come, supply will be absorbed.
> this revolutionary new application for memory people call AI.
Most analysts think LLMs will elevate the long-term base RAM demand level. I mentioned 15-25% above prior projections without AI, which I think everyone agrees is highly likely. That's actually a lot because it's an overall market number and RAM goes lots of places other than PCs, servers and high-end mobile (depending on how you segment, 25% overall could be in the neighborhood of doubling PC, server, high-end mobile demand).
Above that range analyst estimates diverge. Some are more bullish, and a few are much more bullish. But everyone's error bars get much wider when the numbers go over 30% overall. It's hard to tease out exactly how much of the current demand bubble will persist in the long-run. Clearly, the current market is distorted by short-term dynamics but which part is base demand and which distortion?
How much consumer AI compute will be on-device vs aggregated in load-balanced clouds? How much RAM will that kind of compute require? Will the market find it's more efficient to consolidate around two or three mega-datacenters or will each frontier lab (and geopolitical block) continue drag racing each other to tie-up future RAM (as much to keep it away from competitors as for their own needs). I don't know. I've been watching this game as an interested bystander for several decades and I wouldn't bet too much of my own money on the most bullish estimates.
I think there will continue to be latent demand from PC, server and laptop users who have put off purchases as hyperscalers skip to the front of the wafer line with large outlays of spending for chips and RAM. This would likely extend out beyond the datacenter demand which analyst are pretty confident will be in place for the next 2 to 3 years, looking at Nvidia estimates alone.
I'm concerned that the regular Joe/Jane won't be getting a GPU or a system upgrade for some time. And then you have people in Africa and elsewhere who can't afford new phones.
There probably is some distortion in that the top spenders aren't terribly against cornering the RAM market and forcing everyone to pay more for local AI due to higher RAM, GPU and CPU costs while squeezing people through their APIs to access AI services, but that's an expensive gambit that depends on local models not getting good enough and frontier models to continue to get better while requiring large swaths of memory and compute, keeping consumers demanding the latest models from the APIs. But if the supply increases and people have more access to RAM, GPUs and CPUs, we'll likely see more local model usage and people doing all sorts of creative work, even training their own models.
I'm bullish, not so much of an overzealousness as having witnessed so many of these computing explosions to not doubt it much anymore. We'll collectively find all sorts of creative uses for these wafers and tech. If I were to bet on it, I'd simply buy a basket of stocks and hold, riding out the noise - nothing too surprising there.
An industry doesn't need to increase supply when demand increases, they can absorb the demand as profits. More so in an industry that is hard to enter into. Consumers hate this and call it all sorts of things like market failure, gouging, etc... In this case, the suppliers are slow walking increases in production and enjoying a run up in profits. They raise concerns about oversupply, as if this deep learning thing were some passing fad and the market will have no use for the new supply if it passes. It's a dubious notion though, the demand is here to stay and new supply needs to come online to meet demand. We simply need more silicon in the market. High prices should eventually bring new supply online, but I'm a little disappointed by the rate of the ramp up.
>It's a dubious notion though, the demand is here to stay and new supply needs to come online to meet demand.
This is a big bet. Look at what happened in 2001 with the dot-com boom. We're still trading on their dark fiber over-build today. Meanwhile any overcapacity built in fabs will quickly be made obsolete by newer and better fab technology (or at least, that's been the pattern for the past 30 years).
I think you're missing the fact that building new supply takes time and sustained commitment, and there's simply nobody in a good position to make that commitment without losing big if your thesis turns out to be wrong.
If the demand for new AI builds is eventually satisfied, or worse, craters overnight, then who will be left holding the bag? It sure won't be Google or Apple, or even NVIDIA - it will be TSMC and Samsung.
But as you mention, this is all temporary. Either you're right, and demand will remain sustained long enough for some of the providers to decide to take that risk, or demand will crater and prices will fall.
The fact that the S&P 500 is near record highs at the same time as consumer confidence is at a 70 year low is not encouraging for continued all steam ahead in my mind... but then it's easy to predict a general future recession, and much harder to predict it to the day.
The idea that we're still trading dark fiber from 2001 is an old narrative right? I guess it is still floating around. But, we're in a second big fiber build out for not only residential, but also to connect all these new data centers. Could there be some demand oscillation for silicon? Sure, but overall deep learning is not some passing fad even if someone gets too far out over their skis and over buys. So far Big Tech has increased spend 3 years in a row and next year they'll spend more than this year. Demand has evidently not let up for AI services, we still need more silicon.
I doubt consumers will regret these compute purchases in 3 years, my 3 year old GPU is still holding value, actually increased in value, and I use it more now than ever.
I did predict or expect that if oil went to $150 we'd be in recession territory, currently hovering below that level and folks are feeling the squeeze and aren't happy. Things could get worse or better, tough to tell, but I think silicon demand is more or less secular and will do relatively well in a variety of macro conditions.
The dark fiber from the dotcom era is approaching the end of its life expectancy. Contracts for leasing dark strands reflected that. Most of it likely still has useful life left that the owners will want to monetize but it might change how it is used.
Bad metaphor. Not being forward enough is a common mistake. When learning, you may want to make yourself _feel_ like you're too far forward. We can commonly think we're too far forward when actually overall we're still too far back. Teaching it is very hard.
Although it is dependent on your style of skis (e.g. carving) and style of skiing (powder, racing, cruising).
For me, it is very rare that I go too far forward (I began with a leant back style and I haven't rectified that after many years of skiing). I try to prioritise fun over ability.
This seems like it would be easy to structurally solve - big tech could make strategic partnerships with 5 to 10 year horizons with the very fabs in question. If they promised to spend a flat or growing $X on silicon (without easy contract cancellation) over the next decade, then the risk would be entirely on the tech companies and not on the fab companies. Of course, there's always bankruptcy to worry about, but that's less of a threat for Google and Apple than it is for OpenAI or NVIDIA.
The fact that we _haven't_ seen such deals be made, and that ~50% of new datacenter builds have been quietly cancelled[1], suggests to me that we're dealing with paper demand more than true sustained demand.
> the suppliers are slow walking increases in production and enjoying a run up in profits
The major companies have been trying to build >$100B of new production capacity in the US for years now. All of these manufacturing facilities have been significantly delayed by NIMBYs using the same "environmental and community concerns" advocacy slop that hinders almost all productive industry in the US.
Blaming the suppliers is a lazy take. They aren't responsible for degrowth activists being able to dictate what we are allowed to build.
Apparently many people don't realize that Google is already invested, they are simply reupping their ante. Anthropic is a MicroAmaGooVidia amalgamation, a Frankenstein of more or less dead capital reborn as an AI corporation.
I suppose they could chose not to, but for 10B it's a simple choice to hedge a bet on the field.
Buy a cheap unlocked smartphone and run GrapheneOS[0]. I want my smartphone to be like my linux computers where I run them for as long as the hardware works and is still relevant. My iPhone 12 is getting close to its end of life support, yet it is still working well. We should expect better from trillion dollar companies. So I'm not supporting them with dollars wherever I can afford not to. That and I think it's more enjoyable to run something off the beaten path. I like to explore the space a little.
I swapped out my MBP for an Asus Pro Art running linux last year and that's been working out pretty well. Hopefully my cheap motorola phone will be supported by GrapheneOS soon and that will work out too.
GrapheneOS will support future Morotola phones that meet a subset of their requirements, rather than existing phones. Less likely to be budget lines for now.
The cheap Motorola phones won't support GrapheneOS because they are missing some of the security features that GrapheneOS requires. The Motorola partnership is for some new phones: hopefully at a lower price bracket, but likely to be flagships or 2nd tier.
That said, there are other moat factors like, a US company needing to use a US AI provider, sticky customers due to corporate onboarding friction, and others. Not nothing, but not as large a moat as some imagined.
reply