If everyone starts applying cooldowns, won't it postpone the problem? So now there is a considerable amount of users who are affected and someone from the affected group discovers the infection and reports it.
But if everyone will be delaying updates, won't be there less chances to catch it in time? I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
>If everyone starts applying cooldowns, won't it postpone the problem?
There are still research firms who are actively and aggressively scanning new packages once they are pushed. For example socket.dev pulls new packages across ecosystems and performs automated analysis and runs it in a sandbox. We don't have to have them go boom in someone's production repos to find out there is a problem.
And if every malware developer worth their salt now introduces code to "wait out" that period of time, we're back to square one.
This assumes that they employ clandestine enough techniques that you have to actually install, wait and observe the behavior for longer than the cooldown period in order to detect this, because the code is "obfuscated" enough to evade static analysis of the code. It's anti-virus / anti-anti-virus 101 all over so to speak.
The good thing I suppose is that it raises the bar. Your regular "virus generator" script kid (sorry: supply chain attack generator script kid) can no longer pull this off.
Most automated analysis isn't dependent on just behavior, but rather suspicious things in the code itself. You have a popular open source package with files that exist on pypi but not github then that's a big flag, or if a similar package suddenly has some base64encoded garbage that runs through an obfuscated exec call. In other words the simple fact that the project has obfuscated code is enough to flag for further attention.
That said if the only issue is time, researchers will just run their automated analysis through machines with dates in the future alongside their normal tests.
Cat and mouse like the sibling says ;) Like if you start changing system time, I'll keep a log of system time to detect any "jumps" and then "behave normally" if I detect this. Of course I'll run the code that does this through "my obfuscator".
The thing with cat and mouse based on time is that this now became a default. I rather liked my odds when malware authors assumed that the defaults were that dependabot updates right away. If the general consensus online seems to be 7 days, then I'll set my dependabot to wait 10 days, so on average I'll catch even things people report over a weekend. Now that the default is a longer time period, I have to change my time period to be even longer, which actually increases my risk in another way: I'll stay vulnerable to _actual_ vulnerabilities vs. supply chain attacks for longer.
> I'll keep a log of system time to detect any "jumps" and then "behave normally" if I detect this.
This makes no sense, the system clock would be set before the suspect package is even pulled down. There isn't a "jump" just a reboot and system start at a "totally real" point in time.
And the premise is that this package can evade detection of its suspect code by using an ever-increasing amount of odd code? Yeah, that's a hard strategy.
A cat and mouse game at least raises the bar for the exploit. The status quo requires no attempt at obfuscation at all. It also makes it harder for such an exploit to turn into a worm, since developers have time to notice their credentials have been stolen and pull any malicious packages pushed using them. (and such worms hitting popular packages have so far been how most people have been exposed to this risk).
Fundamentally, this is a cat-and-mouse game. But I suspect that "time bomb" techniques aren't economically viable for attackers, at least not with current patterns: current attackers demonstrate "smash and grab" tendencies because they know their access is limited anyways. Attempting to wait out a cooldown exposes them to additional detection risk.
Of course, maybe the attacker profile changes over time. But that's the nature of the game.
That's a fair enough thing to say but then also setting the default to 7 days is no longer a necessity, because security researchers can automate running a new version through an accelerated time sandbox, can't they?
And no they can't and is probably why this change is happening to dependabot: A common pattern is sampling time before and after a sleep/timer, then aborting/delaying if the sandbox looks like it accelerated time by having a 1s sleep timer expire after 0.1s. Malware would also try to use different time sources and try to correlate them.
And yeah, like you say as well, of course its an arms race in the same way that back in the day state of the art viruses and worms were and anti-virus software was.
And personally I still liked it more when malware/supply chain attack actors assumed everyone was upgrading within a day or two while I was doing it after 7. You don't have to outrun the lion, just the guy slower than you and who's between you and the lion ;)
Publishing publicly then applying cooldowns in projects is much easier tgan establishing a new standard for pre-release security testing versions that works across ecosystems and gains zooling support.
Releasing secure software is the right thing to do. We only release insecure software because we need the eyeballs to make it secure.
If we believe the point made above that the many eyeballs are not that important then releasing before we have done everything to make the software as secure as possible is irresponsible.
"We only release insecure software because we need the eyeballs to make it secure." No one is doing this on purpose, they believe the software to be secure when they release it... You shouldn't gate the release of a package until every single offsec research shop in the world has had a chance to look at it
The cooldowns effectively make the initial push into the package repository into a pre-publishing step. It's pretty much isomorphic with what you're suggesting.
Also as an upstream, if your "coworker" releases a strange package without discussing the changes with the broader maintenance group, you might notice after 3-48 hours, but probably not within the hour unless you happened to be online.
> But if everyone will be delaying updates, won't be there less chances to catch it in time?
No: the security assumption behind cooldowns rests on security scanning parties, not on innocent users being victimized. Three days is a short cooldown, but it should be a good enough lead for scanning parties.
> I'm not fully sure if it's possible to preventively scan all NPM packages or how much compute it would require.
It’s not that much data, particularly for parties that are directly financially incentivized to be the first to report malware.
I generally try not to name the companies directly, because I don’t want to give them free advertising. But you can look up e.g. the recent Shai Hulud campaign.
> Just post one link of a "supply chain" problem that was prevented by any of these companies before it went into the wild and affected users.
This is not the claim being made, since cooldowns are not widely adopted at the moment.
Well, yeah. There’s no package police that’ll stop you from installing malware. The argument has never revolved around that; the argument is solely that cooldowns are effective if you use them, and timely detection by third parties is strong evidence of that.
I gave Shai Hulud as an example above. If you want precise timeline examples that demonstrate the efficacy of cooldowns, here’s some examples I collected last year[1].
(See the “Window of opportunity” column in the table.)
When you're running from a bear on a hiking trip you just have to be faster than your friend. So just set your cooldown slightly longer than everyone else's cooldowns. The cooldown will give security researchers some time to scan the packages so it's still good.
I agree, it’s just the wrong approach. As a user, there’s no way to know if a package has been audited during the cooldown by some generous cybersecurity firm before you pull it in, it’s just wishful thinking. Minimizing your dependencies is a more effective strategy against supply chain attacks.
The way to guarantee that is to pay one of those cybersecurity firms, that's basically their business.
And the most effective strategy is to audit and review your dependencies and any updates to them. That probably constrains how many you can have, but just minimizing them is reducing the size of the target, not protecting it per se.
Unless you are a big enough target, there is still potential value to this. Is your business worth enough for someone to spend a lot of money hunting bespoke vulnerabilities? If not, your main risk is being swept in a vulnerability affecting a common dependency, and eliminating the common dependencies removes that risk.
Most of the malicious ones just curl something in a postinstall script, scanners already catch that. The sneaky ones don't look malicious until they run, and three days may not help.
I don't think that HNers understand the recent supply chain attacks very well at all. I also don't think they realize the tests the SCA/package providers do to all the major packages.
Almost all these attacks try to reach out to external sites to steal your data. That is exceptionally hard to hide in any meaningful way.
You can make it much less suspicious. In particular, if you can compromise the package publishing process, and not just pushes to main, you can add your malicious code to binary artifacts, not to the source code.
I think the idea is that it gives a bit of time for the companies which run automated scans of new versions to run through and detect any issues with new versions before users install them en-mass.
That's true, although, if you look at them, you wouldn't notice. The only mention of JVM you can see in the IDEs is in the About dialog, and the IDEs install and run their own OpenJDK, so no JVM has to be installed globally. Almost as if they were a bit self-conscious about using such an "unsexy" architecture...
You mean a JRE, because the whole JDK contains a bunch of things you're never going to need.
Mind you, a default JRE redistribution makes your app at least 100+MB. Using jdeps to strip out unneeded things is a good idea if you want it to get down to 25 ish MBs.
Yeah, saying JRE is a bit of a shortcut since you're supposed to jlink & jpackage & jdance &jpray to get a slimmed down JVM released with your app, but it's closer to what would be a JRE than a full JDK
Similar - no (I'd be glad to be proven wrong), but you can find some small suppliers focused on specific niche like https://www.motedis.com if you need the profiles.
I found that a good tool helps a lot once I switched to Github Copilot app. It solved the friction and mental tax for me. I easily manage 4 sessions in parallel on same or different projects while 2 was max in the past. The bottleneck now is only in review and decision making.
reply