This doesn't "verify" anything. I can't see the code before, I can't see the code after. I can't verify they do the same thing. It won't be updated if there's a long churn of issues and breakage coming from this work plaguing the team for years. The only thing it verifies is that Asana did indeed make the claim, which I don't think anyone was doubting.
> Back in 2022, we set out to migrate Asana's frontend test suite off Enzyme, our aging testing library, and onto React Testing Library (RTL).
Your telling me they had a full team of engineers at Asana, doing nothing but rewriting tests for 4 years until AI came along and did the last year in a couple days? I'm extremely doubtful. I don't doubt for a minute they were one-track to take 5 years, but not because it was 5 years of engineering effort for humans.
Far more likely they finally cleared up some tech debt they had been plugging at off and on for 4 years, then a PR flack got ahold of it and it became a breathless "AI did 5 years of work in a couple days".
I'm not sure I agree with the extremely easy to verify part. How do you validate that your new test suite covers the exact same edge cases as your old?
But yes, LLMs are great at tests. I'm not doubting that an LLM migrated some legacy tests, or that it was a task taking a long time. I am doubting the way it was presented in the article, that this was taking a full team of engineers dedicated to nothing but this, 5 years to accomplish (and presumably already spent 4 million working on this, since the total estimate was 6 million, and they've been at it for 4 year).
I think some PR flack got ahold of the fact that an LLM wrapped up migrating some legacy tests that a team had been slowly chipping away at for 4 years, between their normal feature work, and were on track to finish in 5, then wrote it up like it was that teams entire focus instead of a piece of tech debt.
That makes far more sense to me than spending millions for a team of engineers dedicated to nothing but rewriting an existing test suite.
> Migrating a test suite is exactly the kind of work that LLM's excel at beause it's extremely easy to verify
I don't think that's true. All you see is all the test pass. You don't know that the tests still cover everything that they used to.
LLMs excel where there is an excellent test suite, and you ask them to modify the thing that the test suite tests, and forbid them from changing the tests.
How is it bad faith to question corporate marketing blog posts? That's not bad faith, that's table stakes for critical thinking.
We absolutely should be skeptical when an AI company makes big claims. The fact that the company the AI company is talking about also claims the same thing doesn't change that. OpenAI is getting awareness and marketing out of this, and I'm sure Asana is getting something out of it too.
And I'm not even saying OpenAI or Asana are necessarily lying. Asana might not find out for months or years that some tests had been rewritten poorly, and don't sufficiently test the thing they were supposed to test anymore. For example. If they truly had 5 years of work, then I find it hard to believe that in two weeks of the LLM churning, they had the time to review all the new tests. They spot-checked, at best.
Maybe everything is great. Maybe the LLM did a wonderful job, and this was awesome for Asana. But we have no idea, and we're unlikely to ever find out. Unless, of course, it's in Asana's interest from a marketing perspective to tell us.
(Not sure what the URLs you posted in your comment are supposed to prove. They're unrelated to the issue at hand.)
This is something that frustrates me a lot when I see people jumping on a piece of writing as AI generated, these AI-isms didn't show up in a vacuum. Yes, AI commonly overuses some patterns, but it got those patterns from it's training data because real people wrote (and continue to write) that way. A few matching sentences here or there is not the smoking gun people make it out to be.
The other thing that is characteristic of AI text is messing up the underlying meaning or fine details of what it's saying. It's not just that it's imitating people who write bad contrasting clauses, it also can't write good ones reliably.
edit: Ironically that last sentence is not super clear (but hey writing is hard and I'm constantly exposed to slop). My point is the nature of AI makes its writing worse than its training data, it's not all bad training data.
To be fair to committees, the above quote is explicitly from a single person and I expect that is true of most AD copy simply because having one person write it is cheaper.
Committees are more about stamping out flavour by making what exists more bland, rather than adding endlessly regurgitated idioms.
Catching a murderer is much, *much* easier than catching someone using a modern AI model to generate code, especially if they actually read it and fix occasional LLMisms, and not just vibecode from the hip. The societal damage is also infinitely more serious in the case of murder.
If your argument is "this is unenforceable" and "failure to enforce is a low-stakes problem" then I'm not sure what your concerns are, other than a desire to control how others choose to spend their time. Either of those might be a valid objection (although I do not consider them to be) but together they sort of cancel each other out, no?
Regardless, some efforts like this are explicitly performative. In the event some problems arise and it turns out someone used a specific tool in the process of causing those problems, administrators (and insurers) can say "look we forbade the use of that tool, so this is on them and not us." In other words, enforcement is not always even the point, but it always seems to be the first stop on the concern-trolling trolley.
Finally, yes, people can misbehave and break rules, and the more effort they put into doing so, the more successful they will be at the rule breaking. Nothing about that is justification for not having rules, even when they are hard to enforce.
It's worse than that! Even when the product managers inputs to the engineers are constrained for many problems no two engineers will produce consistent results! In fact recent studies suggest that even the same engineer may produce different results based on mood, if they've had coffee yet, and how near to EOD it is.
Far be it from me to doubt a stranger on the Internet, but can you point out any medical textbook or dictionary written pre-2020 that limits the definition of "vaccine" to "attenuated viruses"?
I mean, a company I worked at had a significant amount of money stolen after the attackers spent 6 months sitting on their access waiting for the right moment to fake an (expected) reply to an email exchange. The original breach (or at least the breach of this executives account) involved a very targeted phish. When the potential payout is millions it justifies a lot of effort.
That alone would be an improvement from the status quo, but the SKG movement does go one stop further in wanting publishers to be required to leave the game in a working (local) state of they design it to be dependent on central services they then shut down.
This could be as easy as releasing the tools they used for development when developing the game.
That's really tough. Marathon, by Bungie, is a good example of why this is tough - it's basically all the Destiny engine, network, and tooling code. When they shut down Destiny, it's not like they have a snapshot of their tooling. They've been continuously updating that tooling for ten years, and now they're using that investment for a new project.
Most game studios are similar, in reusing and improving a whole development architecture and systems across many titles. I would not agree with making them release that, even an older version. That's a big competitive moat for some studios.
I think if a community group wants to PAY to operate a server, that's quite reasonable. And then you still end up with a fight about how much to charge. But I don't think handing over server code is the right move.
Fortunately, Stop Killing Games isn't looking to mandate any one solution to this problem.
There are many possible ways to enable game preservation and SKG is pushing for game studios to pick one and implement it, instead of not doing anything and letting games die when they become unprofitable.
Practically speaking, I think the best way to do something like this is to enforce an auction when a studio shuts down an online game. That way the people that care get to vote with their dollars, rather than costing the studio.
I'm not sure I get this. A game costs about the same as watching a few movies - the cost per hour of entertainment is vanishingly low. For these long running games, the cost per hour of entertainment is down to pennies for these users. The same users who are still playing years later are also the ones who have gotten the most value for that same number of dollars already.
That's how copyright and patent law both work, yes. That's why we have expiry. And we all know the current expiry system is arbitrary (and capricious).
Copyright and patent law aren't relevant to this discussion. We're talking about goods that are purchased and then later rendered unusable by the seller.
I disagree. We're talking about goods purchased, and separately services without a paid support contract, and this is all about copyright law.
There's nothing illegal about selling a game where the servers only work for a week, if you say the servers are only going to work for a week unless you pay more.
The question is, what's reasonable if you don't say anything at all? Forever? A year? Five years? And that's not a trivial question.
If copyright law lasted two years, there would be nothing stopping someone from reverse engineering the service under one of these games after two years and running it themselves (barring serious cryptographic blockers).
Furthermore, the developer would have planned for that expiry of their IP protection. There's a loss to the game studio if they give away their intellectual property while it is still protected.
The only reason this is difficult at all is because of copyright law, and I suspect patent law as well but I think that's murkier.
If you start looking at this through that structure, I think it will make a lot more sense to you why each entity involved takes the actions they do.
> That's really tough. Marathon, by Bungie, is a good example of why this is tough - it's basically all the Destiny engine, network, and tooling code. When they shut down Destiny, it's not like they have a snapshot of their tooling. They've been continuously updating that tooling for ten years, and now they're using that investment for a new project.
I don't get why you think that makes it harder? Right up until the final version of the game their developers were somehow making changes and testing their changes. It's not like the game grew on its own. What were they using to test those changes? Release that. Job done.
I don't think that's what working state means here. It's not a full snapshot or even necessarily multiplayer support. It means minimum functional, which to me is just barely enough to see the assets used in the game. Not necessarily networking, matchmaking, online services, or the relevant tooling. Developers have already proposed making helpers that would introduce an effective switch to do this.
Also what you said about releasing the code and letting the community figure it out was explicitly an example SKG said was okay, from memory.
I think you're right from what I remember reading about them. But I think in practice it costs significant engineering time to package up what they are asking for. For a small game studio it may be incredibly prohibitive.
It just requires game engines to develop a tool to make this much easier to do. The rule won't apply retroactively, so it just means different design choices from the start.
> For a small game studio it may be incredibly prohibitive.
Alright, you lost me here. All of the games that do not follow what this law would suggest are AAA (or scams, essentially). The smaller studios always have some acceptable end-of-life plan from my experience.
Small like 30 people? I don't think so. Genuinely small studios are able to preserve their games for the long-term, so there is no excuse.
AAA gaming isn't something that needs to be protected. Anything of sufficient magnitude and budget should be made responsibly or not at all. If a game is unable to exist without screwing over the user in a very legal sense, it has no right to exist. The requests of SKG are not only sensible, they impose a serious legal ambiguity in the current system that needs to be corrected one way or the other.
AAA was always an untenable monster that brought obscene risk; this was clear even in the mid-2010s and the industry is rapidly moving away from that model for good reason. The user-base clearly does not look kindly on AAA anymore either. That's why they are not profitable anymore. The SKG requirements would make very little impact overall compared to this.
I got my start at a game studio in 2001, after they were bought by Microsoft (and their game went from Mac to Xbox, if you get my drift). They're "big" now, but they regularly lay off large portions of their staff, because game development is boom and bust.
I don't think what you say in your first paragraph follows. Are you really, really interested in learning more about why, with an open mind?
Absolutely! Happy to learn from an industry veteran; hard to argue with that pedigree (part of why I like HN). I figured you might have been, but wanted to push back a little because I think it is important.
Here is my impression. 2001 feels like it was still within a golden era of game development; "AAA" games of that time would have been made by smaller studios still, and budgets could be very large, but not catastrophically so. The industry was still expanding. Post-GFC, once graphics scaled, demands seemed to scale, and costs blew out. Games had to reduce risk as a consequence, become more consolidated, more live-service. But the model was never sustainable at that scale. The tech improved so that costs for basic games went down, but big-budget AAA live service costs went to the moon. Volatility skyrocketed, leading to rapid hiring-firing phases. Now it is at its most extreme and the AAA side of the industry is in crisis. Demands for long-term support could be the straw that breaks the camel's back, but it always seems like that back was going to break eventually anyway. What I hope is happening is that talent is falling into the hands of smaller publishers, but that might not be true. What I do think is that the nature of development may need to change so that studios are able to facilitate these requirements while remaining profitable. Some have shown it can be done, anyway.
That's my impression from a semi-outsider perspective. Happy to be corrected though.
First, thank you so much, I so appreciate when somebody cares what someone else has to say!
Second, I think you're right. The games industry is in really bad shape right now. That's part of why I'm so concerned about putting new requirements on studios. If they have to pay someone to package up software to preserve it like this, that's one less person to generate them revenue, and that means someone gets laid off.
Whether their back breaks or not is not binary. It's measured in careers, and time. Most studios aren't profitable at all, or they're very temporarily profitable after each release. That's why publishers buy them, to keep them afloat during the time when they are losing money, and that's an investment with an expected return from those good times.
If you tell a studio they suddenly have to keep something going when it was already a financial failure, there's no way they can plan for that. They did that planning years ago.
Thanks for continuing the discussion! It's not often an outsider would get to speak with someone with your experience.
I definitely agree that it is not tenable in any studio to have a full time staff member dedicated to packaging software to be in line with this legislation. The solution has to be similar to a toggle in the engine code itself at the very earliest design phases. That means there has to be sufficient advanced notice and can only apply to future titles. There is hope that an inexpensive industry of third parties may arise to easily handle this aspect upfront with new software, but it would be great if there was a tangible demo of this.
Good point about publishers providing the cash to get the studios through the bad times. I think where this falls apart now is in the modern big-budget live service model itself, since these projects are expected to be so long term, so expensive, and gain so much income, that their lack of success now seems to end up in a studio turning the lights out altogether. Concord comes to mind here. Bungie's reliance on the new Marathon is not something I would wish upon any studio either. These kinds of game development strategies do not seem to be healthy for the industry anymore, and some diversification is really needed. These comments here are not really an SKG thing, it just feels like a far bigger issue from the dev side right now. I hate that devs are constantly losing their jobs in the current market, and I think gamers do too. I just can't see the status quo as sustainable. I guess that's why I treated the "back breaking" as binary, but it is true that there is a whole range of suffering inbetween.
A studio should never be forced to keep something going when it is deemed a financial failure. Again, it has to be an upfront design decision during the early planning years so that the end of life version is mostly a compilation target. Latest version goes out, no more updates, no more servers to run. There were more specific solutions discussed by other devs in a video on Ross Scott's channel, but that's the general idea. Does this seem even remotely possible from your viewpoint? Maybe not on current projects, but for projects five to ten years in the future?
If it is too hard to implement in this way for the team, the legislation may force to accept that the nature of the project may be so inherently risky with the current staff resources that it should not come to fruition until new software developments have made it less risky. But you are right that it might just end up as one less person with an actual dev job at the studio, which isn't great in the current economy. Either way, I think many users feel their hands have been forced by the publishers as I understand.
I generally agree that you could require something like this for titles in the future. I think defining future is difficult, the engines everyone is using exist today, and often are just improving overtime, it's very rare for a studio to start from scratch.
But I'm still unsure that there's a good reason to require this. Basically we are saying that if a hobbyist really wants to operate and maintain something, they should be allowed to after some amount of time if the studio stops. It's a chink in the armor of copyright law, really - if a rights holder decides not to continue to license a film or TV show or book for publication, we let them.
I think it's a good idea to figure out who pays for this outside of the studio - I want studios to take risks, honestly the bigger the better, because that's how really interesting ideas get made. I recommended an auction in another thread about this, having a reserve and required auction at shutdown might be a good idea.
> Basically we are saying that if a hobbyist really wants to operate and maintain something, they should be allowed to after some amount of time if the studio stops.
I don't believe that is the problem as defined by SKG. I think the issue is less about empowering users to make use of the property afterwards (although that is nice), and more about the legal ambiguity in the process where products are removed from users without effective prior notice (from purchase date). Games without subscription fees are traded as effective goods by commerce law, but publishers operate as though they are services without a defined end-date through a EULA. IANAL, but my understanding is that whole concept is not legally tested. A one-time purchase should include a contract where the terms for revocation (ideally none, but we can't have nice things) are agreed upon and cannot change at the discretion of one party. You cannot have a license that essentially says "we can do whatever we want at any time". That has never been okay in the history of commerce. A subscription is different, since you know exactly how temporary it is at any particular time. Of course, both a defined lifetime or a subscription are tactics that have been tried and were not as popular with users, so companies resorted to effective trickery while users looked the other way for a time. The social contract is changing as users are watching games they loved die. So overall, the better solution for every party now is a minimal EOL. This isn't a precise threshold and it isn't an expectation that the game would continue to function as normal. It's a minimal effort taken to ensure that the customer still has something left to play around with which is in the spirit of the intended customer agreement; in the California bill, either patching for reasonable offline play (the command-line switch), providing server binaries (I can see this as more of a problem more often), or refunding (no-one wants this).
> if a rights holder decides not to continue to license a film or TV show or book for publication, we let them.
We let them, provided they don't rip the product we buy out of our hands. That's not new for games, but is only now happening for movies. It's unacceptable in every domain. It's like a user being told they can rent a movie, but it isn't clear how long for, so the movie can be requested back after only 5 minutes or could be after 5 years. It is fundamentally unfair. That comparison is quite real too, since some have bought a game, only to find it to be shutting down soon. That feels like theft. Sony removing movies from users libraries and from their computer feels like theft. I would argue it is theft, but that is part of the legal battle here.
> I want studios to take risks, honestly the bigger the better, because that's how really interesting ideas get made
I agree that creative risks should be taken, and also that bad economic strategies should not be rewarded. That's the natural course, and it seems that is happening right now just as one would expect. Many AAA studios are not taking many creative risks according to users (it's almost in the definition of AAA at this point), but are making poor economic decisions by not handling volatility with diversification. It's a recipe for disaster and the developers suffer most it seems.
> having a reserve and required auction at shutdown might be a good idea
I think that's fine, but I'm not sure whether it addresses the legal issues if the new rights holders do not uphold the original understanding of the purchase.
> Back in 2022, we set out to migrate Asana's frontend test suite off Enzyme, our aging testing library, and onto React Testing Library (RTL).
Your telling me they had a full team of engineers at Asana, doing nothing but rewriting tests for 4 years until AI came along and did the last year in a couple days? I'm extremely doubtful. I don't doubt for a minute they were one-track to take 5 years, but not because it was 5 years of engineering effort for humans.
Far more likely they finally cleared up some tech debt they had been plugging at off and on for 4 years, then a PR flack got ahold of it and it became a breathless "AI did 5 years of work in a couple days".
reply