>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently.
No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packages to act both competently and in good faith ad infinitum.
Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there.
> Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently.
This is wildly circular logic!
"One person using these tools isn't bad security practice, the problem is that EVERYONE ELSE ["the ecosystem"] uses these tools and doesn't have higher standards!"
It should be no shock to anyone at this point that huge chunks of common developer tools have very poor security profiles. We've seen stories like this many times.
So do you actually agree or disagree that there's something wrong with npm? It reads as if you were playing both sides, just to land on blaming the individual each time.
Even if this was actually some weirdly written plea to shared responsibility, surely it makes sense that in a hierarchy, one would proritize trying to fix things upstream closer to the root, rather than downstream closer to the leaves, doesn't it?
> This is wildly circular logic!
They're very clearly implying a semantic disagreement there, not making a logical mistake.
I can’t speak for majormajor but I thought the language was kind of funny. “The problem is an ecosystem that allows packages to run arbitrary code silently” is an odd statement because for many people that’s kind of what a package manager does.
> one would proritize trying to fix things upstream closer to the root
One should prioritize fixing things one is responsible for. If you make a commitment to protect your user’s data, then you take responsibility for the tools you use, and how you use them.
Whether or not you – or someone else – should fix those tools upstream, is a separate issue to be solved later. First solve the problems that are your responsibility. Then worry about everyone else.
The npm ecosystem has many security issues but they are all mitigatable.
Same thing with IDE plugins. At least some are full-featured by the manufacturer, but I couldn't get on with VS Code as for every small feature I had to install some random plugin (even if popular, but still developed by who-knows-who).
The amount of browser extension authors who have talked openly about being approached to sell their extension or insert malicious code is many, and presumably many others have taken the money and not told us about it. It seems likely there are IDE extensions doing or going to do the same thing...
It's painful, but I've grown distrustful enough of the ecosystem that I disable updates on every IDE plugin not maintained by a company with known-adequate security controls and review the source code of plugin changes before installing updates, typically opting out unless something is broken.
It's unclear to me if the code linked on the plugin's description page is in amy way guaranteed to be the code that the IDE downloads.
The status quo in software distribution is simultaneously convenient, extraordinarily useful, and inescapably fucked.
>> The security failure is in an ecosystem that allows packages to run arbitrary code silently.
> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight.
How about both? It’s conceptually straightforward to build a language in which code cannot do anything other than read its inputs, consume resources, and produce correctly typed output.
This would not fully solve the supply chain problem — malicious code could produce maliciously incorrect output or exploit side channels, but the exposure would be much, much less than it is now.
> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight.
Could you explain how you'd design a package manager that does not allow that? As far as I understand the moment you use third party code you have to trust to some extent the code that you will run.
NPM setup similar dl_files_security_sigs.db .database for all downloaded files from npm in all offline install? List all versions, latest mod date, multiple latest crypto signatures (shar256, etc) and have been reviewed by multiple security org/researchers, auto flag if any contents are not pure clear/clean txt...
If it detects anything (file date, size, crypto sigs) < N days and have not been thru M="enough" security reviews, the npm system will automatically raise a security flag and stop the install and auto trigger security review on those files.
With proper (default secure) setup, any new version of npm downloads (code, config, scripts) will auto trigger stop download and flagged for global security review by multiple folks/orgs.
When/if this setup available as NPM default, would it stop similar compromise from happen to NPM again? Can anyone think of anyway to hack around this?
I'm speaking to the concept of automatic updates in general, which package managers either enable by default or implicitly allow through lack of security measures.
One obvious solution is to host your own repositories so that nothing gets updated without having been signed off by a trusted employee. Another is to check the cryptographic hash of all packages so it cannot change without the knowledge and consent of your employees.
You're right in that this does not completely eliminate the possibility of trojan horses being sneaked in through open-source dependencies but it would at the very least require some degree of finesse on the part of the person making the trojan horse so that they have to manipulate the system into doing something it was not designed to do.
One thing I really hate about the modern cybersecurity obsession is that there's a large contingent of people who aggressively advocate against anything which might present a problem if misused (rust, encryption on everything no matter how inconsequential, deprecating FTP, UEFI secure boot, timing side-channels, etc) yet at the same time there's a massive community of high-level software developers who appear to be under the impression that extremely basic vulnerabilities (trojan package managers, cross-site scripting, letting my cell phone provider steal my identity because my entire life is authenticated by a SIM card, literally just concatenating strings received over the internet into an SQL statement, etc) are unsolved problems which just has to be tolerated for now until somebody figures out a way to not download and execute non-vetted third-party code. Somehow the two groups never seem to cross swords.
TL;DR: Reading HN i feel like im constantly getting criticized for using C because I might fuck up and let a ROP through yet so many of the most severe modern security breaches are coming from people who think turning off automatic updates is like being asked to prove the rieman zeta hypothesis.
They can't explain, it's just victim blaming. The market currently doesn’t have a proper solution to this.
Everyone works with these package managers, I bet the commenter also has installed pip or npm packages without reading its full code, it just feels cool to tell other people they are dumb and it's their own fault for not reading all the code beforehand or for using a package manager, when every single person does the same. Some just are unlucky.
The whole ecosystem is broken, the expectations of trust are not compatible with the current amount of attacks.
It isn't victim blaming. People like you make it impossible to avoid attacks like these because you have no appetite for a better security model.
I run npm under bubblewrap because npm has a culture of high risk; of using too many dependencies from untrusted authors. But being scrupulous and responsible is a cost I pay with my time and attention. But it is important because if I run some untrusted code and am compromised it can affect others.
But that is challenging when every time some exploit rolls around people, like you, brush it off as "unlucky". As if to say it's inavoidable. That nobody can be expected to be responsible for the libraries they use because that is too hard or whatever. You simply lack the appetite for good hygene and it makes it harder for the minority of us who care about how our actions affect others.
> you have no appetite for a better security model
For what it's worth, there are some advancements. PNPM - the packager used in this case - doesn't automatically run postinstall scripts. In this case, either the engineer allowed it explicitly, or a transitive dependency was previously considered safe, and allowed by default, but stopped being safe.
PNPM also lets you specify a minimum package age, so you cannot install packages younger than X.
The combination of these would stop most attacks, but becomes less effective if everyone specifies a minimum package age, so no one would fall victim.
It's a bit grotesque because the system relies on either the package author noticing on time, or someone falling victim and reporting it.
NPM now supports publishing signed packages, and PNPM has a trustPolicy flag. This is a step in a good direction, but is still not enough, because it relies on publishers to know and care about signing packages, and it relies on consumers to require it.
There _is_ appetite for a better security model, but a lot of old, ubiquitous packages, are unmaintained and won't adopt it. The ecosystem is evolving, but very slowly, and breaking changes seem needed.
I had the chance to finish reading and it looks like Trigger were using an older version of PNPM which didn't do any of the above, and have since implemented everything I've mentioned in my post, plus some additional Git security.
So a slight amendment there on the human error side of things.
At some point you must be open to being compelled to read code you run or ship. Otherwise, if that's to hard, then I don't know what to tell you. We'll just never agree.
If you find a better solution than being responsible for what you do and who you trust, I'm all for it. Until then, that's part of the job.
When I was a junior, our company payed a commercial license for some of the larger libraries we used and it included support. Or manage risk by using fewer and more trustworthy projects like Django instead of reaching for a new dependency from some random person every time you need to solve a simple problem.
> What no appetite? I just don't like your solution.
When I say "appetite" I am being very deliberate. You are hungry but you won't eat your vegetables. When you say "I just don't like your vegetables", then you aren't that hungry. You don't have the appetite. You'd rather accept the risk. Which is fine but then don't complain when stuff like this happens and everyone is compromised.
I hope you've read every diff to every Linux kernel you've ever deployed... There's LOADS of code you've deployed I can bet a large amount of money you never read. So clearly there's solutions that solve the problem of having to read every line of every dependency you deploy. It's just that certain ecosystems are more easy to exploit so new solutions are needed. Read everything is not a solution, it's a bandaid that shows there's a problem of trust to be solved (or improved enough to discourage this wave of attacks) with a technical solution.
No, you are the problem because you have a higher expectation than reality. People shouldn't have to run npm in containers. You're over simplifying with one case where you have found one solution while ignoring the identical problems elsewhere. You are preventing us from looking at other solutions because you think the one you have is enough and works for everyone.
I agree with you that I shouldn't have to treat my libraries like untrusted code. I don't know what the rest of your comment means. I don't see how I'm preventing anybody from looking at other solutions to npm, they just don't want to do it because it's hard. And I have similar criticisms for cargo as it just copies npm and inherits all of its problems. I hate that.
npm has had a bad ecosystem since its inception. The left-pad thing being some of my earliest memories of it [1]. So none of this is new.
But all of this is still an issue because it's too convenient and that's the most important thing. Even cargo copies npm because they want to be seen as convenient and the risk is acknowledged. Nobody has the appetite to be held accountable for who they put their trust in.
> snickerbockers > No, your security failure is that you use a package manager
> you > It isn't victim blaming. People like you make it impossible to avoid attacks like these because you have no appetite for a better security model.
I'd wager a large portion of people with `npm` don't actually realize they have `npm`. I'd also wager that most people that know they have `npm` aren't aware of the security issues.
Under those conditions, people are not in fact making choices. These are not people "that have no appetite for a better security model". These are people who don't even know they are unsafe!
Yes, this is victim blaming. Just in the same way people blame a rape victim for what they wear. Does what you wear modify the situation? Yes. Does it cause the situation? No. We only really blame a victim if they are putting themselves directly, and knowingly, in harms way. This is not that case! This is a case where people are uninformed, both in the dangers present as well as the existence of danger.
FFS, on more than one occasion I've installed a package only to see that it bundles `npm` along with it. And I'm more diligent than most people, so I know tons of people don't know it's happening. Especially because you can't always run `which npm` to find if it is installed. But the fact is that you can do something like `brew install foo` and foo has a dependency that has a dependency that has node as a dependency.
Dependency hell is integral to the problem here! So you can go ahead and choose a package manager that doesn't allow 3rd parties to push arbitrary code and end up with a package manager that allows 3rd parties to push arbitrary code! That's even what made left-pad a thing (and don't get me started on the absurdity of using a module for this functionality!).
> Nobody has the appetite to be held accountable for who they put their trust in
That is jut not the reality of things. In the real world nobody can read all the lines of code. It just simply isn't possible. You aren't reading everything that you're running, let alone all the dependencies and all the way down to the fucking kernel. There just isn't enough time in the day to do this within your lifetime, even if you are running a very cut down system. There's just too many lines of code!
So stop this bullshit rhetoric of "know what you're running" because it is ignoring the reality of the situation. Yes, people should do due diligence and inspect, but the reality is that this is not possible to do. Nor is it bulletproof, as it requires the reader to be omniscient themselves, or at least a security expert with years of training to even be able to spot security mistakes. Hell, if everyone (or just programmers) already had that kind of training then I'd wager 90+% of issues wouldn't even exist in the code in the first place.
So stop oversimplifying the situation because we can't even begin to talk about what needs to be done to solve things if we can't even discuss the reality of the problem.
>it's their own fault for not reading all the code beforehand or for using a package manager, when every single person does the same.
But like, isn't that actually the core of the problem? People choose to blindly trust some random 3rd parties - isn't exploiting this trust seems to be inevitable and predictable outcome?
Victim-blaming is when a girl gets raped and you tell her that it's her fault for dressing like a skank and getting drunk at a college fraternity party. Telling the bank they should have put the money in a vault instead of leaving it in an unlocked drawer next to the cash register is not victim-blaming. Telling the CIA that they shouldn't have given Osama Bin-Laden guns and money to fight the soviets in afghanistan is not victim-blaming. Telling president Roosevelt it was a poor decision to park the entire Pacific fleet in a poorly-defended naval base adjacent to an expansionist empire which is already at war with most of America's allies is not victim-blaming. *Telling a well-funded corporation to not download and execute third-party code with privileges is not victim blaming, especially as their customers are often the ones who are actually being targeted.*
>I bet the commenter also has installed pip or npm packages without reading its full code
I think i did use pip at some point about a decade ago but i can't remember what for. In general though you lose that bet because I don't use either of these programs.
> it just feels cool to tell other people they are dumb
it does, yes.
>and it's their own fault for not reading all the code beforehand or for using a package manager, when every single person does the same.
I don't suppose you've ever played an old video game called "Lemmings"?
>Some just are unlucky.
Lol.
>The whole ecosystem is broken, the expectations of trust are not compatible with the current amount of attacks.
that's kind of my point, except it doesn't mitigate responsibility for participating in that ecosystem.
This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user. That said, it requires careful design on the part of the application (and its install process) to not allow a non-elevated application to overwrite some part of the trusted application and get the keys anyway. macOS has the best system here in principle with its bundle signing, but most developer tools are not in bundles so its of limited utility in this circumstance.
> This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user.
Isn't that a smartphone-and-app-store-only thing?
As I understand it, no mainstream desktop OS provides the capabilities to, for example, protect a user's browser cookies from a malicious tool launched by that user.
That's why e.g. PC games ship with anti-cheat mechanisms - because PCs don't have a comprehensive attested-signed-code-only mechanism to prevent nefarious modifications by the device owner.
> As I understand it, no mainstream desktop OS provides the capabilities to, for example, protect a user's browser cookies from a malicious tool launched by that user.
macOS sandboxing has been used for this kind of thing for years. Open a terminal window on a new Mac and trying to open the user’s photo library, Desktop, iCloud documents, etc. will trigger a permissions prompt.
Interesting, it's a few years since I've used a Mac.
Descriptions of this stuff online are pretty confusing. Apparently there's an "App Sandbox" and also "Transparency Consent and Control" - I assume from your mention of the photo library describing the latter?
How does this protection interact with IDEs? For some operations conducted in an IDE, like checking out code and collecting dependencies the user grants the software access to SSH keys, artifact repo credentials and suchlike. But unsigned code can also be run as a child process of the IDE - such as when the user compiles and runs their code.
How does the sandboxing protection interact with the IDE and its subprocesses, to ensure only the right subprocesses can access credentials?
They added sandboxing in the 2000s, which does mandatory access control (e.g. you can write a rule that Firefox.app can’t access ~/Library/Keychains) and expanded it with containers (not OCI) which standardize the layout starting with the App Store so they all follow common restrictions for what they can access and where they store different classes of data. Those policies are inherited by child processes (e.g. your Terminal.app permissions apply to CLI tools you run in its windows but not something you start by logging in via SSH) so much of the effort has been standardizing the UX – don’t access photos directly, use the system picker which allows the user to select subsets, etc.
So the answer to that question depends on what permissions the IDE has asked for and been granted. It’s likely that the first time you opened a shell inside the IDE you’d get promoted for permission to access protected locations the first time you ran a command which did something protected, but they could ask for something like full disk access at install time to avoid many prompts.
macOS and Windows’s native keychains both support this - they encrypt the secrets with a key that is not accessible to apps that run with user permissions without sudo (macOS) or elevation (Windows). The actual user can still access them, but a normal app (other than the one that stored the secret in the keychain originally) running as that user cannot do so directly.
Serious question: what tools only support netrc for authentication? I'm aware of lots of tools that (unfortunately IMO) support netrc as a source of credentials, but I can't think of a single one that requires it.
I think the OP is aware of that and I agree with them that it’s bad practice despite how common it is.
For example with AWS, you can use the AWS CLI to sign you in and that goes through the HTTPS auth flow to provide you with temporary access keys. Which means:
1. You don’t have any access keys in plain text
2. Even if your env vars are also stolen, those AWS keys expire within a few hours anyway.
If the cloud service you’re using doesn’t support OIDC or any other ephemeral access keys, then you should store them encrypted. There’s numerous ways you can do this, from password managers to just using PGP/GPG directly. Just make sure you aren’t pasting them into your shell otherwise you’ll then have those keys in plain text in your .history file.
I will agree that It does take effort to get your cloud credentials set up in a convenient way (easy to access, but without those access keys in plain text). But if you’re doing cloud stuff professionally, like the devs in the article, then you really should learn how to use these tools.
> If the cloud service you’re using doesn’t support OIDC or any other ephemeral access keys, then you should store them encrypted. There’s numerous ways you can do this, from password managers to just using PGP/GPG directly. Just make sure you aren’t pasting them into your shell otherwise you’ll then have those keys in plain text in your .history file.
This doesn't really help though, for a supply chain attack, because you're still going to need to decrypt those keys for your code to read at some point, and the attacker has visibility on that, right?
Like the shell isn't the only thing the attacker has access to, they also have access to variables set in your code.
I agree it doesn’t keep you completely safe. However scanning the file system for plain text secrets is significantly easier than the alternatives.
For example, for vars to be read, you’d need the compromised code to be part of your the same project. But if you scan the file system, you can pick up secrets for any project written in any language, even those which differ from the code base that pulled the compromised module.
This example applies directly to the article; it wasn’t their core code base that ran the compromised code but instead an experimental repository.
Furthermore, we can see from these supply chain attacks that they do scan the file system. So we do know that encrypting secrets adds a layer of protection against the attacks happening in the wild.
In an ideal world, we’d use OIDC everywhere and not need hardcoded access keys. But in instances where we can’t, encrypting them is better than not.
It's certainly a smaller surface that could help. For instance, a compromised dev dependency that isn't used in the production build would not be able to get to secrets for prod environments at that point. If your local tooling for interacting with prod stuff (for debugging, etc) is set up in a more secure way that doesn't mean long-lived high-value secrets staying on the filesystem, then other compromised things have less access to them. Add good, phishing-resistant 2FA on top, and even with a keylogger to grab your web login creds for that AWS browser-based auth flow, an attacker couldn't re-use it remotely.
(And that sort of ephemeral-login-for-aws-tooling-from-local-env is a standard part of compliance processes that I've gone through.)
That's not correct. The (ephemeral) keys are still available. Just do `aws configure export-credentials --profile <YOUR_OIDC_PROFILE>`
Sure, they'll likely expire in 1-24 hours, but that can be more than enough for the attacker.
You also can try to limit the impact of the credentials by adding IP restrictions to the assumed role, but then the attacker can just proxy their requests through your machine.
> That's not correct. The (ephemeral) keys are still available. Just do `aws configure export-credentials --profile <YOUR_OIDC_PROFILE>`
That’s not on the file system though. Which is the point I’m directly addressing.
I did also say there are other ways to pull those keys and how this isn’t completely solution. But it’s still vastly better than having those keys in clear text on the file system.
Arguing that there are other ways to circumvent security policies is a lousy excuse to remove security policies that directly protect you against known attacks seen in the wild.
> Sure, they'll likely expire in 1-24 hours, but that can be more than enough for the attacker.
It depends on the attacker, but yes, in some situations that might be more than long enough. Which is while I would strongly recommend people don’t set their OIDC creds to 24 hours. 8 hours is usually long enough, shorter should be required if you’re working on sensitive/high profile systems. And in the case of this specific attack, 8 hours would have been sufficient given the attacker probed AWS while the German team were asleep.
But again, i do agree it’s not a complete solution. However it’s still better than hardcoded access keys in plain text saved in the file system.
> You also can try to limit the impact of the credentials by adding IP restrictions to the assumed role, but then the attacker can just proxy their requests through your machine.
In practice this never happens (attacks proxying) in the wild. But you’re right that might be another countermeasure they employ one day.
Security is definitely a game of ”cat and mouse”. But I wouldn’t suggest people use hardcoded access keys just because there are counter attacks to the OIDC approach. That would be like “throwing the baby out with the bath water.”
They are. In `~/.aws/cli/cache` and `~/.aws/sso/cache`. AWS doesn't do anything particularly secure with its keys. And none of the AWS client libraries are designed for the separation of the key material and the application code.
I also don't think it's even possible to use the commonly available TPMs or Apple's Secure Enclave for hardware-assisted signatures.
> 8 hours is usually long enough. And in the case of this specific attack, 8 hours would have been sufficient given the attacker probed AWS while the German team were asleep.
They could have just waited a bit. 8 hours does not materially change anything, the credential is still long-lived enough.
I love SSO and OIDC but the AWS tooling for them is... not great. In particular, they have poor support for observability. A user can legitimately have multiple parallel sessions, and it's more difficult to parse the CloudTrail. And revocation is done by essentially pushing the policy to prohibit all the keys that are older than some timestamp. Static credentials are easier to manage.
> In practice this never happens (attacks proxying) in the wild. But you’re right that might be another countermeasure they employ one day.
If I remember correctly, LastPass (or was it Okta?) was hacked by an attacker spying on the RAM of the process that had credentials.
And if you look at the timeline, the attack took only minutes to do. It clearly was automated.
I tried to wargame some scenarios for hardware-based security, but I don't think it's feasible at all. If you (as a developer) have access to some AWS system, then the attacker running code on your behalf can also trivially get it.
You can use keyring/keychain with credential_process although it's only a minor shift in security from "being able to read from the fs" to "being able to execute a binary"
> They are. In `~/.aws/cli/cache` and `~/.aws/sso/cache`. AWS doesn't do anything particularly secure with its keys.
Thanks for the correction. That’s disappointing to read. I’d have hoped they’d have done something more secure than that.
> And none of the AWS client libraries are designed for the separation of the key material and the application code.
The client libraries can read from env vars too. Which isn’t perfect either, but on some OSs, can be more secure than reading from the FS.
> If I remember correctly, LastPass (or was it Okta?) was hacked by an attacker spying on the RAM of the process that had credentials.
That was a targeted attack.
But again, I’m not suggesting OIDC solves everything. But it’s still more secure than not using it.
> And if you look at the timeline, the attack took only minutes to do. It clearly was automated.
Automated doesn’t mean it happens the moment the host is compromised. If you look at the timeline, you see that the attack happened over night; hours after the system was compromised.
> They could have just waited a bit. 8 hours does not materially change anything, the credential is still long-lived enough.
Except when you look at the timeline of those specific attack, they probed AWS more than 8 hours after the start of the working day.
A shorter TTL reduces the window of attack. That is a material change for the better. Yes I agree on its own it’s not a complete solution. But saying “it has no material benefit so why bother” is clearly ridiculous. By the same logic, you could argue “why bother rotating keys at all, we might as well keep the same credentials for years”….
Security isn’t a Boolean state. It’s incremental improvements that leave the system, as a whole, more of a challenge.
Yes there will always be ways to circumvent security policies. But the harder you make it, the more you reduce your risk. And having ephemeral access tokens reduces your risk because an attacker then has a shorter window for attack.
> I tried to wargame some scenarios for hardware-based security, but I don't think it's feasible at all. If you (as a developer) have access to some AWS system, then the attacker running code on your behalf can also trivially get it.
The “trivial” part depends entirely on how you access AWS and what security policies are in place.
It can range anywhere from “forced to proxy from the hosts machine from inside their code base while they are actively working” to “has indefinite access from any location at any time of day”.
A sufficiently advanced attack can gain access but that doesn’t mean we shouldn’t be hardening against less sophisticated attacks.
To use an analogy, a burglar can break a window to gain access to your house, but that doesn’t mean there isn’t any benefit in locking your windows and doors.
> A sufficiently advanced attack can gain access but that doesn’t mean we shouldn’t be hardening against less sophisticated attacks.
I'm a bit worried that with the advent of AI, there won't be any real difference between these two. And AI can do recon, choose the tools, and perform the attack all within a couple of minutes. It doesn't have to be perfect, after all.
I've been thinking about it, and I'm just going to give up on trying to secure the dev environments. I think it's a done deal that developers' machines are going to be compromised at some point.
For production access, I'm going to gate it behind hardware-backed 2FA with a separate git repository and build infrastructure for deployments. Read-write access will be available only via RDP/VNC through a cloud host with mandatory 2FA.
And this still won't protect against more sophisticated attackers that can just insert a sneaky code snippet that introduces a deliberate vulnerability.
> """
I'm strongly in favor of blocking post-install scripts by default. :+1:
This is a change that will have a painful adjustment period for our users, but I believe in ~1 year everyone will look back and be thankful we made it. It's nuts that a [pnpm|yarn|npm] install can run arbitrary code in the first place.
"""
> stored in our database which was not compromised
Personally I don't really agree with "was not compromised"
You say yourself that the guy had access to your secrets and AWS, I'd definitely consider that compromised even if the guy (to your knowledge) didn't read anything from the database. Assume breach if access was possible.
It depends on what kind of access we're talking about. If we're talking about AWS resource mutations, one can trust CloudTrail to accurately log those actions. CloudTrail can also log data plane events, though you have to turn it on, and it costs extra. Similarly, RDS access logging is pretty trustworthy, though functionality varies by engine.
Most non-trivial security investigations involve building chains of events. If SSM Session Manager was used to access the EC2 instance (as is best practice) using stolen credentials, then the investigation would connect access to the instance to the use of instance credentials to access the S3 bucket, as both events would be recorded by CloudTrail.
CloudTrail has what it has. It's not going to record accesses to EC2 instances via SSH because AWS service APIs aren't used. (That's one of the reasons why using Session Manager is recommended over SSH.) But that doesn't mean CloudTrail isn't trustworthy; it just means it's not omniscient.
Ideally you should have a clear audit log of all developer actions that access production resources, and clear records of custody over any shared production credentials (e.g. you should be able to show the database password used by service A is not available outside of it, and that no malicious code was deployed to service A). A lot of places don't do this, of course, but often you can come up with a pretty good circumstantial case that it was unlikely that exfiltration occurred over the time range in question.
That definitely helps, but I don't think it solves the compromised machine scenario.
If the attacker has shell access to the dev's laptop, they are likely just running commands directly from that machine (or proxying through it). So to GitHub, the traffic still looks like it's coming from the allowed IP.
Allowlists are mostly for stopping usage of a token that got stolen and taken off-device.
> This is one of the frustrating realities of these attacks: once the malware runs, identifying the source becomes extremely difficult. The package doesn't announce itself. The pnpm install completes successfully. Everything looks normal.
Sounds like there’s no EDR running on the dev machines? You should have more to investigate if Sentinel One/CrowdStrike/etc were running.
My org is currently at 7 people and we have 365 repositories associated with our github org. We've been around for a number of years and I'd guess that impacts the number of repos more than the number of team members.
A core SRE principle is that "machines/servers are cattle, not pets". They shouldn't be special or bespoke in a way that makes replacement painful or difficult.
I've heard the term used for servers before but not version control repositories. I just don't understand what it would mean for a git repo to be a cattle vs a pet. Like what is an example of a cattle repo vs a pet repo. The metaphore just sounds like gibberish to me idk.
Unless all it means is that that you can have more than a few like the other commenter said but I didn't think that was what the metaphore meant with respect to servers so again I have no idea lol
To me it would mean that a git repo should not have scripts, runners, etc. configured that we don't have the means to easily and readily replace. It should all be documented and understood well enough that we could kill the repo and init another at will.
That’s weird, pnpm no longer automatically runs lifecycle scripts like preinstall [1], so unless they were running a very old version of pnpm, shouldn’t they have been protected from Shai-Hulud?
Let me understand it fully. That means they updated dependencies using old, out of date package manager. If pnpm was up to date, this would no have happened? Sounds totally like their fault then
I have been thinking about this. How do I make my git setup on my laptop secure? Currently, I have my ssh key on the laptop, so if I want to push, I just use git push. And I have admin credentials for the org. How do I make it more secure?
1) Get 1Password, 2) use 1Password to hold all your SSH keys and authorize SSH access [1], 3) use 1Password to sign your Git commits and set up your remote VCS to validate them [2], 4) use GitHub OAuth [3] or the GitHub CLI's Login with HTTPS [4] to do repository push/pull. If you don't like 1Password, use BitWarden.
With this setup there are two different SSH keys, one for access to GitHub, one is a commit signing key, but you don't use either to push/pull to GitHub, you use OAuth (over HTTPS). This combination provides the most security (without hardware tokens) and 1Password and the OAuth apps make it seamless.
Do not use a user with admin credentials for day to day tasks, make that a separate user in 1Password. This way if your regular account gets compromised the attacker will not have admin credentials.
Okay great advice, thanks. I'm already using Bitwarden and found out they have an SSH Agent feature too [1]. I've tried lastpass, Bitwarden, 1password and I prefer Bitwarden (good UX, very affordable)
One approach I started using a could of years ago was storing SSH private keys in the TPM, and using it via PKCS11 in SSH agent.
One benefit of Microsoft requiring them for Windows 11 support is that nearly every recent computer has a TPM, either hardware or emulated by the CPU firmware.
It guarantees that the private key can never be exfiltrated or copied. But it doesn't stop malicious software on your machine from doing bad things from your machine.
So I'm not certain how much protection it really offers on this scenario.
That's what I do. For those of us too lazy to read the article, tl;dr:
ssh-keygen -t ed25519-sk
or, if your FIDO token doesn't support edwards curves:
ssh-keygen -t ecdsa-sk
tap the token when ssh asks for it, done.
Use the ssh key as usual. OpenSSH will ask you to tap the token every time you use it: silent git pushes without you confirming it by tapping the token become impossible. Extracting the key from your machine does nothing — it's useless without the hardware token.
Looks like on the server side this can be mitigated somewhat by the MaxStartups¹ setting for OpenSSH or equivalent behavior for other services that support SSH auth (e.g., Git forges like GitHub):
MaxStartups
Specifies the maximum number of concurrent unauthenticated
connections to the SSH daemon. Additional connections
will be dropped until authentication succeeds or the
LoginGraceTime expires for a connection. The default is
10:30:100.
Alternatively, random early drop can be enabled by
specifying the three colon separated values
start:rate:full (e.g. "10:30:60"). sshd(8) will refuse
connection attempts with a probability of rate/100 (30%)
if there are currently start (10) unauthenticated
connections. The probability increases linearly and all
connection attempts are refused if the number of
unauthenticated connections reaches full (60).
So it looks like it's possible to support ControlMaster while still somewhat hampering mass-cloning thousands of repos via SSH key without reauthenticating.
Admittedly I'd put this more in the category of making endpoint compromise easier to detect than that of actually preventing any particular theft of data or manipulation of systems. But it might still be worth doing! If it means only a few dozen or only a hundred repos get compromised before detection instead of a few thousand, that's a good thing.
Besides all that (or MaxSessions, as another user mentions), if an attacker compromises a developer laptop and can only open those connections as long as the developer is online, that's one thing. But a plaintext key that they can grab and reuse from their own box is obviously an even sweeter prize!
"The SSH key on my YubiKey is useless to attackers" is obviously the wrong way to think about this, but using a smartcard for SSH keys is still a way to avoid storing plaintext secrets. It's good hygiene.
There is no defense against a compromised laptop. You should prevent this at all cost.
You can make it a bit more challenging for the attacker by using secure enclaves (like TPM or Yubikey), enforce signed commits, etc. but if someone compromised your machine, they can do whatever you can.
Enforcing signing off on commits by multiple people is probably your only bet. But if you have admin creds, an attacker can turn that off, too. So depending on your paranoia level and risk appetite, you need a dedicated machine for admin actions.
It's more nuanced than that. Modern OSes and applications can, and often do, require re-authentication before proceeding with sensitive actions. I can't just run `sudo` without re-authenticating myself; and my ssh agent will reauthenticate me as well. See, e.g., https://developer.1password.com/docs/ssh/agent/security
The malware can wait until you authenticate and perform its actions then in the context of your user session. The malware can also hijack your PATH variable and replace sudo with a wrapper that includes malicious commands.
It can also just get lucky and perform a 'git push' while your SSH agent happens to be unlocked. We don't want to rely on luck here.
Really, it's pointless. Unless you are signing specific actions from an independent piece of hardware [1], the malware can do what you can do. We can talk about the details all day long, and you can make it a bit harder for autonomously acting malware, but at the end of the day it's just a finger exercise to do what they want to do after they compromised your machine.
Do you have evidence or a reproducible test case of a successful malware hijack of an ssh session using a Mac and the 1Password agent, or the sudo replacement you suggested? I assume you fully read the link I sent?
I don't think you're necessarily wrong in theory -- but on the other hand you seem to discount taking reasonable (if imperfect) precautionary and defensive measures in favor of an "impossible, therefore don't bother" attitude. Taken to its logical extreme, people with such attitudes would never take risks like driving, or let their children out of the house.
You get the idea. It can do something similar to the git binary and hijack "git commit" such that it will amend whatever it wants and you will happily sign it and push it using your hardened SSH agent.
You say it's unlikely, fine, so your risk appetite is sufficiently high. I just want to highlight the risk.
It could have created a bash alias then. And I don't think a dev wants to be restricted in creating executables. Again, if a dev can do it, so can the malware.
A compromised laptop should always be treated as a fully compromised. However, you can take steps that drastically reduce the likelihood of bad things happening before you can react (e.g. disable accounts/rotate keys).
Further, you can take actions that inherently limit the ability for a compromise to actually cause impact. Not needing to actually store certain things on the machine is a great start.
You can add a gpg key and subkeys to a yubikey and use gpg-agent instead of ssh-agent for ssh auth. When you commit or push, it asks you for a pin for the yubikey to unlock it.
1 store my ssh key in 1Password and use the 1Password ssh agent. This agents asks for access to the key(s) with Touch ID. Either for each access or for each session etc. one can also whitelist programs but I think this all reduces the security.
There is the FIDO feature which means you don’t need to hackle with gpg at all. You can even use an ssh key as signing key to add another layer of security on the GitHub side by only allowing signed commits.
You can set up your repo to disable pushing directly to branches like main and require MFA to use the org admin account, so something malicious would need to push to a benign branch and separately be merged into one that deploys come from.
There's nothing wrong with pushing to main, as long as you don't blindly treat the head of the main branch as production-ready. It's a branch like any other; Git doesn't care what its name is.
They can't with git by itself, but if you're also signed in to GitHub or BitBucket's CLI with an account able to approve merges they could use those tools.
I’ve started to get more and more paranoid about this. It’s tough when you’re running untrusted code, but I think I’ve improved this by:
not storing SSH keys on the filesystem, and instead using an agent (like 1Password) to mediate access
Stop storing dev secrets/credentials on the filesystem, injecting them into processes with env vars or other mechanisms. Your password manager could have a way to do this.
Develop in a VM separate from your regular computer usage. On windows this is essential anyway through using WSL, but similar things exist for other OSs
This is what agents are for. You load your private key into an agent so you don't have to enter your passphrase every time you use it. Agents are supposed to be hardened so that your private key can't be easily exfiltrated from them. You can then configure `ssh` to pass requests through the agent.
There are lots of agents out there, from the basic `ssh-agent`, to `ssh-agent` integrated with the MacOS keychain (which automatically unlocks when you log in), to 1Password (which is quite nice!).
This is a good defense for malware that only has read access to the filesystem or a stolen hard drive scenario without disk encryption, but does nothing against the compromised dev machine scenario.
This seems to be the standard thing people miss. All the things that make security more convenient also make it weaker. They boast about how "doing thing X" makes them super secure, pat on the back and done. Completely ignoring other avenues they left open.
A case like this brings this out a lot. Compromised dev machine means that anything that doesn't require a separate piece of hardware that asks for your interaction is not going to help. And the more interactions you require for tightening security again the more tedious it becomes and you're likely going to just instinctively press the fob whenever it asks.
Sure, it raises the bar a bit because malware has to take it into account and if there are enough softer targets they may not have bothered. This time.
Classic: you only have to outrun the other guy. Not the lion.
Like, I see the comment about the Keychain integration and all that. But in the end I fail to see (without further explanation but I'm eager to learn if there's something I am unaware of) where this isn't different from what I am saying.
Like yes, my ssh key has a passphrase of course. Which is different from my system one actually. As soon as I log into the system I add the key, which means entering the passphrase once, so I don't have to enter it all the time. That would get old real fast. But now ssh can just use my key to do stuff and the agent doesn't know if it's me or I got compromised by npm installing something. And if you add a hardware token you "just have to tap" each time that's a step back into more security but does add tedium. Depending on how often my workflow uses ssh (or something that uses the key) in the background this will become something most people just blindly "tap" on. And then we are back towards less security but with more setup steps, complications and tedium.
I saw the "or allow for a session", which is a step towards security again, because I may be able to allow a script that does several things with ssh with a single tap, which is great of course. Hopefully that cuts the taps down so much that I don't just blindly tap on every request for it. Like the 1password thing you mentioned. If I do lots of things that make it "ask again" often enough I get pushed into "yeah yeah, I know the drill, just tap" security hole.
Keep in mind that not every agent is so naive as to allow a local client to connect to it without reauthenticating somehow.
1Password, for example, will, for each new application, pop up a fingerprint request on my Mac before handling the connection request and allow additional requests for a configurable period of time -- and, by default, it will lock the agent when you lock your machine. It will also request authentication before allowing any new process to make the first connection. See e.g. https://developer.1password.com/docs/ssh/agent/security
You memorize it, or keep it in 1Password. 1Password can manage your SSH keys, and 1Password can/does require a password, so it's still protected with something you know + something you have.
Passphrases, when strong enough, are fine when they are not traversing a medium that can be observed by a third party. They're not recommended for authenticating a secure connection over a network, but they’re fine for unlocking a much longer secret that cannot be cracked via guessing, rainbow tables, or other well known means. Hell, most people unlock their phones with a 4 digit passcode, and their computers with a passphrase.
> when they are not traversing a medium that can be observed by a third party
Isn't that why all those security experts are pushing for SSL everywhere and 30 second certificate expiration? To make the medium unobservable by a third party?
If you believe them, passphrases should be okay over fiber you don't control too.
One thing I forgot to mention is what the trust relationship looks like. Passphrases used for authentication are known by both parties and could be leaked by the other side or stolen from them, while private keys remain only available to you. With public key authentication, the other party only has your public key, which is freely shareable.
And yes, we all know that 2FA, passkeys, etc. are all better than passphrases, and that layer 3 wire encryption is important.
I’m merely responding to your blanket assertion that passphrases aren’t “secure enough,” but sometimes they are.
My SSH keys aren't on my computer: they're safely hidden on a hardware token, behind a secure element, like a Yubikey.
Devices like the Yubikey do precisely exist because computers aren't things to be trusted. So their reason for being is to offer a minimal attack surface.
When I git fetch/pull/push I just do it. But it requires me to physically use my Yubikey. It's not 100% foolproof but it's way better than having SSH keys only protected by a password.
So Git over SSH, on a Git/SSH server that supports Yubikeys.
Not a perfect defense, but sufficient to make your key much harder to exploit: Use a Yubikey (or similar) resident SSH key, with the Yubikey configured to require a touch for each authentication request.
I wouldn't say that's better. Now your .config directory contains a github token that can do more than just repo pull/push, and it is trivially exfiltrated. Though similar thing could be said for browser cookies.
password-protect your key (preferably with a good password that is not the same password you use to log in to your account). If you use a password it's encrypted; otherwise its stored on plaintext and anybody who manages to get a hold of your laptop can steal the private key.
It was a really noisy worm though, and it looked like a few actors also jumped on the exposed credentials making private repos public and modifying readmes promoting a startup/discord.
Got any pointers on how to configure this for yarn? I'm not turning anything up in the yarn documentation or in my random google searches.
npm still seems to be debating whether they even want to do it. One of many reasons I ditched npm for yarn years ago (though the initial impetus was npm's confused and constantly changing behaviors around peer dependencies)
Yarn is unfortunately a dead-end security-wise under current maintainership.
If you are still on yarn v1 I suggest being consistent with '--ignore-scripts --frozen-lockfile' and run any necessary lifecycle scripts for dependencies yourself. There is @lavamoat/allow-scripts to manage this if your project warrants it.
If you are on newer yarn versions I strongly encourage to migrate off to either pnpm or npm.
Obviously blocking install scripts is a good thing, but this is just a false sense of security. If you install a package you will likely execute some code from it too, so the malware can just run then. And that is what the next attack will do as everyone starts using pnpm (or if npm blocks it too).
It's not a false sense of security imo. Code often runs in its own environment, for example a container. We're "used to" sandboxing/ isolating runtime code. It's the package installation process that gets less attention.
The approach the attacker took makes little sense to me, perhaps someone else has an explanation for it? At first they monitored what's going on and then silently exfiltrated credentials and private repos. Makes sense so far. But then why make so much noise with trying to force push repositories? It's Git, surely there's a clone of nearly everything on most dev machines etc.
It's most likely two or more separate attackers operating. The first malware, Shai Hulud 2, exfiltrates credentials from the infected dev machine to new public GitHub repositories. As the repositories are public and searchable via GitHub's interfaces, any malicious attacker aware of the attack can easily grab the credentials and launch any attack, whether it's a noisy destructive script or some sophisticated ransomware.
Given that all the stolen credentials were made public, I was hoping that someone would build a haveibeenpwned style site. We know we were compromised on at least a few tokens, but it would be nice to be able to search using a compromised token to find out what else leaked. We’ve rotated everything we could think of but not knowing if we’ve missed something sucks.
We don't have a clear explanation of the destructive behavior, right? It looks like it had no real purpose, and there were much more effective ways of destroying their repos. Very script kiddie-like, which does not really fit the main complexity of the virus. Very surprising.
"The simultaneous activity from US and India confirmed we were dealing with a single attacker using multiple VPNs or servers, not separate actors."
Did it really? It's not clear to me why the possibility that the exfiltrated credentials were shared with other actors, each acting independently, is ruled out.
> This incident involved one of our engineers installing a compromised package on their development machine, which led to credential theft and unauthorized access to our GitHub organization.
The org only has 4-5 engineers. So you can imagine the impact a large org will have.
I'm wondering why storing creds in env variables as plain text is acceptable - e.g. they better be dynamically fetched from a secret manager with 2FA in the way
It’s almost like Microsoft sells security products and runs the most insecure JavaScript package manager to build those security products and couldn’t switch off of it even if the engineers in the org recommended a more secure JavaScript execution context— and that’s realistically why anthropic bought an engine.
It has nothing to do with interpreters or JIT, it has nothing to do with npm at all. All package managers have the insane security model of "arbitrary code execution with no constraints".
It just so happens that all of those languages share the worst design points, such as the need for a package manager at all and the classic "eval and equivalents run arbitrary code".
>All package managers have the insane security model of "arbitrary code execution with no constraints".
Not all of them, just the most popular ones for these highly sophisticated, well thought-out bunch of absolute languages.
I tend to agree but think npms post install hook is a degree worse. Triggering during install, silently because npm didn't like someone using the feature to ask for donations, is worse than requiring you to load and run the package code.
It was on development branches. The threat actor was trying to delete development work.
Their main branch was already protected. I don't think it makes sense to protect every single branch in a repo? Since not all devs will have the ability to turn this off
>Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently.
No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight. You only have "secutity" to the extent that you can trust the people who control those packages to act both competently and in good faith ad infinitum.
Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there.
> Running npm install is not negligence. Installing dependencies is not a security failure. The security failure is in an ecosystem that allows packages to run arbitrary code silently.
This is wildly circular logic!
"One person using these tools isn't bad security practice, the problem is that EVERYONE ELSE ["the ecosystem"] uses these tools and doesn't have higher standards!"
It should be no shock to anyone at this point that huge chunks of common developer tools have very poor security profiles. We've seen stories like this many times.
If you care, you need to actually care!
So do you actually agree or disagree that there's something wrong with npm? It reads as if you were playing both sides, just to land on blaming the individual each time.
Even if this was actually some weirdly written plea to shared responsibility, surely it makes sense that in a hierarchy, one would proritize trying to fix things upstream closer to the root, rather than downstream closer to the leaves, doesn't it?
> This is wildly circular logic!
They're very clearly implying a semantic disagreement there, not making a logical mistake.
I can’t speak for majormajor but I thought the language was kind of funny. “The problem is an ecosystem that allows packages to run arbitrary code silently” is an odd statement because for many people that’s kind of what a package manager does.
[dead]
> one would proritize trying to fix things upstream closer to the root
One should prioritize fixing things one is responsible for. If you make a commitment to protect your user’s data, then you take responsibility for the tools you use, and how you use them.
Whether or not you – or someone else – should fix those tools upstream, is a separate issue to be solved later. First solve the problems that are your responsibility. Then worry about everyone else.
The npm ecosystem has many security issues but they are all mitigatable.
Same thing with IDE plugins. At least some are full-featured by the manufacturer, but I couldn't get on with VS Code as for every small feature I had to install some random plugin (even if popular, but still developed by who-knows-who).
The amount of browser extension authors who have talked openly about being approached to sell their extension or insert malicious code is many, and presumably many others have taken the money and not told us about it. It seems likely there are IDE extensions doing or going to do the same thing...
It's painful, but I've grown distrustful enough of the ecosystem that I disable updates on every IDE plugin not maintained by a company with known-adequate security controls and review the source code of plugin changes before installing updates, typically opting out unless something is broken.
It's unclear to me if the code linked on the plugin's description page is in amy way guaranteed to be the code that the IDE downloads.
The status quo in software distribution is simultaneously convenient, extraordinarily useful, and inescapably fucked.
>> The security failure is in an ecosystem that allows packages to run arbitrary code silently.
> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight.
How about both? It’s conceptually straightforward to build a language in which code cannot do anything other than read its inputs, consume resources, and produce correctly typed output.
This would not fully solve the supply chain problem — malicious code could produce maliciously incorrect output or exploit side channels, but the exposure would be much, much less than it is now.
> No, your security failure is that you use a package manager that allows third-parties push arbitrary code into your product with no oversight.
Could you explain how you'd design a package manager that does not allow that? As far as I understand the moment you use third party code you have to trust to some extent the code that you will run.
Can we design something like virustotal setup? (https://en.wikipedia.org/wiki/VirusTotal)
NPM setup similar dl_files_security_sigs.db .database for all downloaded files from npm in all offline install? List all versions, latest mod date, multiple latest crypto signatures (shar256, etc) and have been reviewed by multiple security org/researchers, auto flag if any contents are not pure clear/clean txt...
If it detects anything (file date, size, crypto sigs) < N days and have not been thru M="enough" security reviews, the npm system will automatically raise a security flag and stop the install and auto trigger security review on those files.
With proper (default secure) setup, any new version of npm downloads (code, config, scripts) will auto trigger stop download and flagged for global security review by multiple folks/orgs.
When/if this setup available as NPM default, would it stop similar compromise from happen to NPM again? Can anyone think of anyway to hack around this?
How would you identify "security researchers" and tell them apart from the attacker in a trench coat?
After you've done that, why would these supposedly expert security researchers review random code in your package manager?
> have been reviewed by multiple security org/researchers
I imagine reviewing all the code for all the packages for all the published versions gets really expensive. Who's paying for this?
Microsoft has a 3.5 trillion dollar market cap. I guess they can pay for it?
I'm speaking to the concept of automatic updates in general, which package managers either enable by default or implicitly allow through lack of security measures.
One obvious solution is to host your own repositories so that nothing gets updated without having been signed off by a trusted employee. Another is to check the cryptographic hash of all packages so it cannot change without the knowledge and consent of your employees.
You're right in that this does not completely eliminate the possibility of trojan horses being sneaked in through open-source dependencies but it would at the very least require some degree of finesse on the part of the person making the trojan horse so that they have to manipulate the system into doing something it was not designed to do.
One thing I really hate about the modern cybersecurity obsession is that there's a large contingent of people who aggressively advocate against anything which might present a problem if misused (rust, encryption on everything no matter how inconsequential, deprecating FTP, UEFI secure boot, timing side-channels, etc) yet at the same time there's a massive community of high-level software developers who appear to be under the impression that extremely basic vulnerabilities (trojan package managers, cross-site scripting, letting my cell phone provider steal my identity because my entire life is authenticated by a SIM card, literally just concatenating strings received over the internet into an SQL statement, etc) are unsolved problems which just has to be tolerated for now until somebody figures out a way to not download and execute non-vetted third-party code. Somehow the two groups never seem to cross swords.
TL;DR: Reading HN i feel like im constantly getting criticized for using C because I might fuck up and let a ROP through yet so many of the most severe modern security breaches are coming from people who think turning off automatic updates is like being asked to prove the rieman zeta hypothesis.
They can't explain, it's just victim blaming. The market currently doesn’t have a proper solution to this.
Everyone works with these package managers, I bet the commenter also has installed pip or npm packages without reading its full code, it just feels cool to tell other people they are dumb and it's their own fault for not reading all the code beforehand or for using a package manager, when every single person does the same. Some just are unlucky.
The whole ecosystem is broken, the expectations of trust are not compatible with the current amount of attacks.
It isn't victim blaming. People like you make it impossible to avoid attacks like these because you have no appetite for a better security model.
I run npm under bubblewrap because npm has a culture of high risk; of using too many dependencies from untrusted authors. But being scrupulous and responsible is a cost I pay with my time and attention. But it is important because if I run some untrusted code and am compromised it can affect others.
But that is challenging when every time some exploit rolls around people, like you, brush it off as "unlucky". As if to say it's inavoidable. That nobody can be expected to be responsible for the libraries they use because that is too hard or whatever. You simply lack the appetite for good hygene and it makes it harder for the minority of us who care about how our actions affect others.
> you have no appetite for a better security model
For what it's worth, there are some advancements. PNPM - the packager used in this case - doesn't automatically run postinstall scripts. In this case, either the engineer allowed it explicitly, or a transitive dependency was previously considered safe, and allowed by default, but stopped being safe.
PNPM also lets you specify a minimum package age, so you cannot install packages younger than X. The combination of these would stop most attacks, but becomes less effective if everyone specifies a minimum package age, so no one would fall victim.
It's a bit grotesque because the system relies on either the package author noticing on time, or someone falling victim and reporting it.
NPM now supports publishing signed packages, and PNPM has a trustPolicy flag. This is a step in a good direction, but is still not enough, because it relies on publishers to know and care about signing packages, and it relies on consumers to require it.
There _is_ appetite for a better security model, but a lot of old, ubiquitous packages, are unmaintained and won't adopt it. The ecosystem is evolving, but very slowly, and breaking changes seem needed.
I had the chance to finish reading and it looks like Trigger were using an older version of PNPM which didn't do any of the above, and have since implemented everything I've mentioned in my post, plus some additional Git security.
So a slight amendment there on the human error side of things.
What no appetite? I just don't like your solution. The industry needs an answer to this problem stat, and it can't be "just read the code before".
At some point you must be open to being compelled to read code you run or ship. Otherwise, if that's to hard, then I don't know what to tell you. We'll just never agree.
If you find a better solution than being responsible for what you do and who you trust, I'm all for it. Until then, that's part of the job.
When I was a junior, our company payed a commercial license for some of the larger libraries we used and it included support. Or manage risk by using fewer and more trustworthy projects like Django instead of reaching for a new dependency from some random person every time you need to solve a simple problem.
> What no appetite? I just don't like your solution.
When I say "appetite" I am being very deliberate. You are hungry but you won't eat your vegetables. When you say "I just don't like your vegetables", then you aren't that hungry. You don't have the appetite. You'd rather accept the risk. Which is fine but then don't complain when stuff like this happens and everyone is compromised.
I hope you've read every diff to every Linux kernel you've ever deployed... There's LOADS of code you've deployed I can bet a large amount of money you never read. So clearly there's solutions that solve the problem of having to read every line of every dependency you deploy. It's just that certain ecosystems are more easy to exploit so new solutions are needed. Read everything is not a solution, it's a bandaid that shows there's a problem of trust to be solved (or improved enough to discourage this wave of attacks) with a technical solution.
No, you are the problem because you have a higher expectation than reality. People shouldn't have to run npm in containers. You're over simplifying with one case where you have found one solution while ignoring the identical problems elsewhere. You are preventing us from looking at other solutions because you think the one you have is enough and works for everyone.
I agree with you that I shouldn't have to treat my libraries like untrusted code. I don't know what the rest of your comment means. I don't see how I'm preventing anybody from looking at other solutions to npm, they just don't want to do it because it's hard. And I have similar criticisms for cargo as it just copies npm and inherits all of its problems. I hate that.
npm has had a bad ecosystem since its inception. The left-pad thing being some of my earliest memories of it [1]. So none of this is new.
But all of this is still an issue because it's too convenient and that's the most important thing. Even cargo copies npm because they want to be seen as convenient and the risk is acknowledged. Nobody has the appetite to be held accountable for who they put their trust in.
[1] https://en.wikipedia.org/wiki/Npm_left-pad_incident
The problem is you're victim blaming.
I'd wager a large portion of people with `npm` don't actually realize they have `npm`. I'd also wager that most people that know they have `npm` aren't aware of the security issues.Under those conditions, people are not in fact making choices. These are not people "that have no appetite for a better security model". These are people who don't even know they are unsafe!
Yes, this is victim blaming. Just in the same way people blame a rape victim for what they wear. Does what you wear modify the situation? Yes. Does it cause the situation? No. We only really blame a victim if they are putting themselves directly, and knowingly, in harms way. This is not that case! This is a case where people are uninformed, both in the dangers present as well as the existence of danger.
FFS, on more than one occasion I've installed a package only to see that it bundles `npm` along with it. And I'm more diligent than most people, so I know tons of people don't know it's happening. Especially because you can't always run `which npm` to find if it is installed. But the fact is that you can do something like `brew install foo` and foo has a dependency that has a dependency that has node as a dependency.
Dependency hell is integral to the problem here! So you can go ahead and choose a package manager that doesn't allow 3rd parties to push arbitrary code and end up with a package manager that allows 3rd parties to push arbitrary code! That's even what made left-pad a thing (and don't get me started on the absurdity of using a module for this functionality!).
That is jut not the reality of things. In the real world nobody can read all the lines of code. It just simply isn't possible. You aren't reading everything that you're running, let alone all the dependencies and all the way down to the fucking kernel. There just isn't enough time in the day to do this within your lifetime, even if you are running a very cut down system. There's just too many lines of code!So stop this bullshit rhetoric of "know what you're running" because it is ignoring the reality of the situation. Yes, people should do due diligence and inspect, but the reality is that this is not possible to do. Nor is it bulletproof, as it requires the reader to be omniscient themselves, or at least a security expert with years of training to even be able to spot security mistakes. Hell, if everyone (or just programmers) already had that kind of training then I'd wager 90+% of issues wouldn't even exist in the code in the first place.
So stop oversimplifying the situation because we can't even begin to talk about what needs to be done to solve things if we can't even discuss the reality of the problem.
>it's their own fault for not reading all the code beforehand or for using a package manager, when every single person does the same.
But like, isn't that actually the core of the problem? People choose to blindly trust some random 3rd parties - isn't exploiting this trust seems to be inevitable and predictable outcome?
>it's just victim blaming
Victim-blaming is when a girl gets raped and you tell her that it's her fault for dressing like a skank and getting drunk at a college fraternity party. Telling the bank they should have put the money in a vault instead of leaving it in an unlocked drawer next to the cash register is not victim-blaming. Telling the CIA that they shouldn't have given Osama Bin-Laden guns and money to fight the soviets in afghanistan is not victim-blaming. Telling president Roosevelt it was a poor decision to park the entire Pacific fleet in a poorly-defended naval base adjacent to an expansionist empire which is already at war with most of America's allies is not victim-blaming. *Telling a well-funded corporation to not download and execute third-party code with privileges is not victim blaming, especially as their customers are often the ones who are actually being targeted.*
>I bet the commenter also has installed pip or npm packages without reading its full code
I think i did use pip at some point about a decade ago but i can't remember what for. In general though you lose that bet because I don't use either of these programs.
> it just feels cool to tell other people they are dumb
it does, yes.
>and it's their own fault for not reading all the code beforehand or for using a package manager, when every single person does the same.
I don't suppose you've ever played an old video game called "Lemmings"?
>Some just are unlucky.
Lol.
>The whole ecosystem is broken, the expectations of trust are not compatible with the current amount of attacks.
that's kind of my point, except it doesn't mitigate responsibility for participating in that ecosystem.
> Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there.
Doesn't really matter, if the agent is unlocked they can be accessed.
This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user. That said, it requires careful design on the part of the application (and its install process) to not allow a non-elevated application to overwrite some part of the trusted application and get the keys anyway. macOS has the best system here in principle with its bundle signing, but most developer tools are not in bundles so its of limited utility in this circumstance.
> This is not strictly true - most OS keychain stores have methods of authenticating the requesting application before remitting keys (signatures, non-user-writable paths, etc.), even if its running as the correct user.
Isn't that a smartphone-and-app-store-only thing?
As I understand it, no mainstream desktop OS provides the capabilities to, for example, protect a user's browser cookies from a malicious tool launched by that user.
That's why e.g. PC games ship with anti-cheat mechanisms - because PCs don't have a comprehensive attested-signed-code-only mechanism to prevent nefarious modifications by the device owner.
> As I understand it, no mainstream desktop OS provides the capabilities to, for example, protect a user's browser cookies from a malicious tool launched by that user.
macOS sandboxing has been used for this kind of thing for years. Open a terminal window on a new Mac and trying to open the user’s photo library, Desktop, iCloud documents, etc. will trigger a permissions prompt.
Interesting, it's a few years since I've used a Mac.
Descriptions of this stuff online are pretty confusing. Apparently there's an "App Sandbox" and also "Transparency Consent and Control" - I assume from your mention of the photo library describing the latter?
How does this protection interact with IDEs? For some operations conducted in an IDE, like checking out code and collecting dependencies the user grants the software access to SSH keys, artifact repo credentials and suchlike. But unsigned code can also be run as a child process of the IDE - such as when the user compiles and runs their code.
How does the sandboxing protection interact with the IDE and its subprocesses, to ensure only the right subprocesses can access credentials?
They added sandboxing in the 2000s, which does mandatory access control (e.g. you can write a rule that Firefox.app can’t access ~/Library/Keychains) and expanded it with containers (not OCI) which standardize the layout starting with the App Store so they all follow common restrictions for what they can access and where they store different classes of data. Those policies are inherited by child processes (e.g. your Terminal.app permissions apply to CLI tools you run in its windows but not something you start by logging in via SSH) so much of the effort has been standardizing the UX – don’t access photos directly, use the system picker which allows the user to select subsets, etc.
https://developer.apple.com/documentation/security/app-sandb...
So the answer to that question depends on what permissions the IDE has asked for and been granted. It’s likely that the first time you opened a shell inside the IDE you’d get promoted for permission to access protected locations the first time you ran a command which did something protected, but they could ask for something like full disk access at install time to avoid many prompts.
macOS and Windows’s native keychains both support this - they encrypt the secrets with a key that is not accessible to apps that run with user permissions without sudo (macOS) or elevation (Windows). The actual user can still access them, but a normal app (other than the one that stored the secret in the keychain originally) running as that user cannot do so directly.
> Also the OP seemingly implies credentials are stored on-filesystem in plaintext but I might be extrapolating too much there.
To be fair, some tools only support a netrc file for http(s) based auth. Regardless, if you want to use git via http this vector exists almost always.
Serious question: what tools only support netrc for authentication? I'm aware of lots of tools that (unfortunately IMO) support netrc as a source of credentials, but I can't think of a single one that requires it.
Afaik, nix for https-based git(hub/lab/...) repositories and http-auth protected resources (via fetchurl and friends).
It wasn't in their product. It was just on a devs machine
I think the OP is aware of that and I agree with them that it’s bad practice despite how common it is.
For example with AWS, you can use the AWS CLI to sign you in and that goes through the HTTPS auth flow to provide you with temporary access keys. Which means:
1. You don’t have any access keys in plain text
2. Even if your env vars are also stolen, those AWS keys expire within a few hours anyway.
If the cloud service you’re using doesn’t support OIDC or any other ephemeral access keys, then you should store them encrypted. There’s numerous ways you can do this, from password managers to just using PGP/GPG directly. Just make sure you aren’t pasting them into your shell otherwise you’ll then have those keys in plain text in your .history file.
I will agree that It does take effort to get your cloud credentials set up in a convenient way (easy to access, but without those access keys in plain text). But if you’re doing cloud stuff professionally, like the devs in the article, then you really should learn how to use these tools.
> If the cloud service you’re using doesn’t support OIDC or any other ephemeral access keys, then you should store them encrypted. There’s numerous ways you can do this, from password managers to just using PGP/GPG directly. Just make sure you aren’t pasting them into your shell otherwise you’ll then have those keys in plain text in your .history file.
This doesn't really help though, for a supply chain attack, because you're still going to need to decrypt those keys for your code to read at some point, and the attacker has visibility on that, right?
Like the shell isn't the only thing the attacker has access to, they also have access to variables set in your code.
I agree it doesn’t keep you completely safe. However scanning the file system for plain text secrets is significantly easier than the alternatives.
For example, for vars to be read, you’d need the compromised code to be part of your the same project. But if you scan the file system, you can pick up secrets for any project written in any language, even those which differ from the code base that pulled the compromised module.
This example applies directly to the article; it wasn’t their core code base that ran the compromised code but instead an experimental repository.
Furthermore, we can see from these supply chain attacks that they do scan the file system. So we do know that encrypting secrets adds a layer of protection against the attacks happening in the wild.
In an ideal world, we’d use OIDC everywhere and not need hardcoded access keys. But in instances where we can’t, encrypting them is better than not.
It's certainly a smaller surface that could help. For instance, a compromised dev dependency that isn't used in the production build would not be able to get to secrets for prod environments at that point. If your local tooling for interacting with prod stuff (for debugging, etc) is set up in a more secure way that doesn't mean long-lived high-value secrets staying on the filesystem, then other compromised things have less access to them. Add good, phishing-resistant 2FA on top, and even with a keylogger to grab your web login creds for that AWS browser-based auth flow, an attacker couldn't re-use it remotely.
(And that sort of ephemeral-login-for-aws-tooling-from-local-env is a standard part of compliance processes that I've gone through.)
> 1. You don’t have any access keys in plain text
That's not correct. The (ephemeral) keys are still available. Just do `aws configure export-credentials --profile <YOUR_OIDC_PROFILE>`
Sure, they'll likely expire in 1-24 hours, but that can be more than enough for the attacker.
You also can try to limit the impact of the credentials by adding IP restrictions to the assumed role, but then the attacker can just proxy their requests through your machine.
> That's not correct. The (ephemeral) keys are still available. Just do `aws configure export-credentials --profile <YOUR_OIDC_PROFILE>`
That’s not on the file system though. Which is the point I’m directly addressing.
I did also say there are other ways to pull those keys and how this isn’t completely solution. But it’s still vastly better than having those keys in clear text on the file system.
Arguing that there are other ways to circumvent security policies is a lousy excuse to remove security policies that directly protect you against known attacks seen in the wild.
> Sure, they'll likely expire in 1-24 hours, but that can be more than enough for the attacker.
It depends on the attacker, but yes, in some situations that might be more than long enough. Which is while I would strongly recommend people don’t set their OIDC creds to 24 hours. 8 hours is usually long enough, shorter should be required if you’re working on sensitive/high profile systems. And in the case of this specific attack, 8 hours would have been sufficient given the attacker probed AWS while the German team were asleep.
But again, i do agree it’s not a complete solution. However it’s still better than hardcoded access keys in plain text saved in the file system.
> You also can try to limit the impact of the credentials by adding IP restrictions to the assumed role, but then the attacker can just proxy their requests through your machine.
In practice this never happens (attacks proxying) in the wild. But you’re right that might be another countermeasure they employ one day.
Security is definitely a game of ”cat and mouse”. But I wouldn’t suggest people use hardcoded access keys just because there are counter attacks to the OIDC approach. That would be like “throwing the baby out with the bath water.”
> That’s not on the file system though.
They are. In `~/.aws/cli/cache` and `~/.aws/sso/cache`. AWS doesn't do anything particularly secure with its keys. And none of the AWS client libraries are designed for the separation of the key material and the application code.
I also don't think it's even possible to use the commonly available TPMs or Apple's Secure Enclave for hardware-assisted signatures.
> 8 hours is usually long enough. And in the case of this specific attack, 8 hours would have been sufficient given the attacker probed AWS while the German team were asleep.
They could have just waited a bit. 8 hours does not materially change anything, the credential is still long-lived enough.
I love SSO and OIDC but the AWS tooling for them is... not great. In particular, they have poor support for observability. A user can legitimately have multiple parallel sessions, and it's more difficult to parse the CloudTrail. And revocation is done by essentially pushing the policy to prohibit all the keys that are older than some timestamp. Static credentials are easier to manage.
> In practice this never happens (attacks proxying) in the wild. But you’re right that might be another countermeasure they employ one day.
If I remember correctly, LastPass (or was it Okta?) was hacked by an attacker spying on the RAM of the process that had credentials.
And if you look at the timeline, the attack took only minutes to do. It clearly was automated.
I tried to wargame some scenarios for hardware-based security, but I don't think it's feasible at all. If you (as a developer) have access to some AWS system, then the attacker running code on your behalf can also trivially get it.
You can use keyring/keychain with credential_process although it's only a minor shift in security from "being able to read from the fs" to "being able to execute a binary"
> They are. In `~/.aws/cli/cache` and `~/.aws/sso/cache`. AWS doesn't do anything particularly secure with its keys.
Thanks for the correction. That’s disappointing to read. I’d have hoped they’d have done something more secure than that.
> And none of the AWS client libraries are designed for the separation of the key material and the application code.
The client libraries can read from env vars too. Which isn’t perfect either, but on some OSs, can be more secure than reading from the FS.
> If I remember correctly, LastPass (or was it Okta?) was hacked by an attacker spying on the RAM of the process that had credentials.
That was a targeted attack.
But again, I’m not suggesting OIDC solves everything. But it’s still more secure than not using it.
> And if you look at the timeline, the attack took only minutes to do. It clearly was automated.
Automated doesn’t mean it happens the moment the host is compromised. If you look at the timeline, you see that the attack happened over night; hours after the system was compromised.
> They could have just waited a bit. 8 hours does not materially change anything, the credential is still long-lived enough.
Except when you look at the timeline of those specific attack, they probed AWS more than 8 hours after the start of the working day.
A shorter TTL reduces the window of attack. That is a material change for the better. Yes I agree on its own it’s not a complete solution. But saying “it has no material benefit so why bother” is clearly ridiculous. By the same logic, you could argue “why bother rotating keys at all, we might as well keep the same credentials for years”….
Security isn’t a Boolean state. It’s incremental improvements that leave the system, as a whole, more of a challenge.
Yes there will always be ways to circumvent security policies. But the harder you make it, the more you reduce your risk. And having ephemeral access tokens reduces your risk because an attacker then has a shorter window for attack.
> I tried to wargame some scenarios for hardware-based security, but I don't think it's feasible at all. If you (as a developer) have access to some AWS system, then the attacker running code on your behalf can also trivially get it.
The “trivial” part depends entirely on how you access AWS and what security policies are in place.
It can range anywhere from “forced to proxy from the hosts machine from inside their code base while they are actively working” to “has indefinite access from any location at any time of day”.
A sufficiently advanced attack can gain access but that doesn’t mean we shouldn’t be hardening against less sophisticated attacks.
To use an analogy, a burglar can break a window to gain access to your house, but that doesn’t mean there isn’t any benefit in locking your windows and doors.
Agreed.
> A sufficiently advanced attack can gain access but that doesn’t mean we shouldn’t be hardening against less sophisticated attacks.
I'm a bit worried that with the advent of AI, there won't be any real difference between these two. And AI can do recon, choose the tools, and perform the attack all within a couple of minutes. It doesn't have to be perfect, after all.
I've been thinking about it, and I'm just going to give up on trying to secure the dev environments. I think it's a done deal that developers' machines are going to be compromised at some point.
For production access, I'm going to gate it behind hardware-backed 2FA with a separate git repository and build infrastructure for deployments. Read-write access will be available only via RDP/VNC through a cloud host with mandatory 2FA.
And this still won't protect against more sophisticated attackers that can just insert a sneaky code snippet that introduces a deliberate vulnerability.
They are on the filesystem though.
Login then check your .aws/login/cache folder.
Oh that’s disappointing. Thanks for the correction.
> """ I'm strongly in favor of blocking post-install scripts by default. :+1: This is a change that will have a painful adjustment period for our users, but I believe in ~1 year everyone will look back and be thankful we made it. It's nuts that a [pnpm|yarn|npm] install can run arbitrary code in the first place. """
- a pnpm maintainer 1 year ago
https://github.com/pnpm/pnpm/pull/8897
And yet here we are…
Convenience trumps security every time. With people who allegedly know better.
Well pnpm does it by default for quite some time. It’s annoying, yes, but I take a little annoyance if it means I’m more secure.
> stored in our database which was not compromised
Personally I don't really agree with "was not compromised"
You say yourself that the guy had access to your secrets and AWS, I'd definitely consider that compromised even if the guy (to your knowledge) didn't read anything from the database. Assume breach if access was possible.
There are logs for accessing aws resources and if you don't see the access before you revoke it then the data is safe
Unless the attacker used any one of hundreds of other avenues to access the AWS resource.
Are you sure they didn’t get a service account token from some other service then use that to access customer data?
I’ve never seen anyone claim in writing all permutations are exhaustively checked in the audit logs.
It depends on what kind of access we're talking about. If we're talking about AWS resource mutations, one can trust CloudTrail to accurately log those actions. CloudTrail can also log data plane events, though you have to turn it on, and it costs extra. Similarly, RDS access logging is pretty trustworthy, though functionality varies by engine.
What do you mean by “trust cloud trail”
So cloud trail shows the compromised account logging into an EC2 instance every day like normal.
Then service account credentials are used to access user data in S3.
How does cloud trail indicate the compromised credentials were used to access the customer data in S3?
If you have data events enabled for your S3 bucket, CloudTrail will log every access to that bucket along with the identity of the principal used to access it. https://docs.aws.amazon.com/awscloudtrail/latest/userguide/l...
Right and in my example it would be the principal of the service account, not the compromised AWS account.
If you ran a cloud trail query that's essentially "Did Alice access user data in S3 ever?" the answer would be "No"
So that brings us back to the question, what is meant by "trust CloudTrail"
Most non-trivial security investigations involve building chains of events. If SSM Session Manager was used to access the EC2 instance (as is best practice) using stolen credentials, then the investigation would connect access to the instance to the use of instance credentials to access the S3 bucket, as both events would be recorded by CloudTrail.
CloudTrail has what it has. It's not going to record accesses to EC2 instances via SSH because AWS service APIs aren't used. (That's one of the reasons why using Session Manager is recommended over SSH.) But that doesn't mean CloudTrail isn't trustworthy; it just means it's not omniscient.
Ideally you should have a clear audit log of all developer actions that access production resources, and clear records of custody over any shared production credentials (e.g. you should be able to show the database password used by service A is not available outside of it, and that no malicious code was deployed to service A). A lot of places don't do this, of course, but often you can come up with a pretty good circumstantial case that it was unlikely that exfiltration occurred over the time range in question.
Because an attacker would never cover their tracks...
Indeed, being able to trust your audit logs is imperative.
Really appreciate the transparency here. Post-mortems like this are vital for the industry.
I'm curious was the exfiltration traffic distinguishable from normal developer traffic?
We've been looking into stricter egress filtering for our dev environments, but it's always a battle between security and breaking npm install
Wouldn’t the IP allowlist feature on the GitHub organisation work wonders for this kind of attack?
That definitely helps, but I don't think it solves the compromised machine scenario.
If the attacker has shell access to the dev's laptop, they are likely just running commands directly from that machine (or proxying through it). So to GitHub, the traffic still looks like it's coming from the allowed IP.
Allowlists are mostly for stopping usage of a token that got stolen and taken off-device.
> This is one of the frustrating realities of these attacks: once the malware runs, identifying the source becomes extremely difficult. The package doesn't announce itself. The pnpm install completes successfully. Everything looks normal.
Sounds like there’s no EDR running on the dev machines? You should have more to investigate if Sentinel One/CrowdStrike/etc were running.
Yep. I think EDR would have detected, alerted if not completely killed a noisy Trufflehog attack chain
Very offtopic but this caught my eye:
> Total repos cloned: 669
How big is this company? All the numbers I can find online suggest well below 100 people, and yet they have over 600 repos? Is that normal?
We have a ratio of roughly 7:1 (repos to engineers). It was probably closer to 12:1 at some point.
* Spikes/Demo project
* Smaller projects that might have gone live, but have since been migrated elsewhere
* Core services
* Forks of certain supply chain dependencies that we've made improvements to.
My org is currently at 7 people and we have 365 repositories associated with our github org. We've been around for a number of years and I'd guess that impacts the number of repos more than the number of team members.
If they have an architect that loves microservices and thinks every microservice needs its own repo that's what happens (insanity).
Completely normal yes. Repos are cattle not pets.
> Repos are cattle not pets.
What do you mean by this?
A core SRE principle is that "machines/servers are cattle, not pets". They shouldn't be special or bespoke in a way that makes replacement painful or difficult.
I've heard the term used for servers before but not version control repositories. I just don't understand what it would mean for a git repo to be a cattle vs a pet. Like what is an example of a cattle repo vs a pet repo. The metaphore just sounds like gibberish to me idk.
Unless all it means is that that you can have more than a few like the other commenter said but I didn't think that was what the metaphore meant with respect to servers so again I have no idea lol
To me it would mean that a git repo should not have scripts, runners, etc. configured that we don't have the means to easily and readily replace. It should all be documented and understood well enough that we could kill the repo and init another at will.
You can have more than a few
That’s weird, pnpm no longer automatically runs lifecycle scripts like preinstall [1], so unless they were running a very old version of pnpm, shouldn’t they have been protected from Shai-Hulud?
1: https://github.com/pnpm/pnpm/pull/8897
At the end of the article, they talk about how they've since updated to the latest major version of pnpm, which is the one with that change
Let me understand it fully. That means they updated dependencies using old, out of date package manager. If pnpm was up to date, this would no have happened? Sounds totally like their fault then
Yeah, I thought that was the main reason to use pnpm. Very confused.
Maybe the project itself had a postinstall script? It doesn't run lifecycle scripts of dependencies, but it still runs project-level ones.
I have been thinking about this. How do I make my git setup on my laptop secure? Currently, I have my ssh key on the laptop, so if I want to push, I just use git push. And I have admin credentials for the org. How do I make it more secure?
1) Get 1Password, 2) use 1Password to hold all your SSH keys and authorize SSH access [1], 3) use 1Password to sign your Git commits and set up your remote VCS to validate them [2], 4) use GitHub OAuth [3] or the GitHub CLI's Login with HTTPS [4] to do repository push/pull. If you don't like 1Password, use BitWarden.
With this setup there are two different SSH keys, one for access to GitHub, one is a commit signing key, but you don't use either to push/pull to GitHub, you use OAuth (over HTTPS). This combination provides the most security (without hardware tokens) and 1Password and the OAuth apps make it seamless.
Do not use a user with admin credentials for day to day tasks, make that a separate user in 1Password. This way if your regular account gets compromised the attacker will not have admin credentials.
[1] https://developer.1password.com/docs/ssh/agent/ [2] https://developer.1password.com/docs/ssh/git-commit-signing/ [3] https://github.com/hickford/git-credential-oauth [4] https://cli.github.com/manual/gh_auth_login
Okay great advice, thanks. I'm already using Bitwarden and found out they have an SSH Agent feature too [1]. I've tried lastpass, Bitwarden, 1password and I prefer Bitwarden (good UX, very affordable)
[1] https://bitwarden.com/help/ssh-agent/
Bitwarden verbiage deserves to be higher than 1Password, here.
Make sure the gh cli isn’t storing oauth credentials in plaintext as it can silently do.
I already use 1password and have it already installed. Will try this out. Thanks!
One approach I started using a could of years ago was storing SSH private keys in the TPM, and using it via PKCS11 in SSH agent.
One benefit of Microsoft requiring them for Windows 11 support is that nearly every recent computer has a TPM, either hardware or emulated by the CPU firmware.
It guarantees that the private key can never be exfiltrated or copied. But it doesn't stop malicious software on your machine from doing bad things from your machine.
So I'm not certain how much protection it really offers on this scenario.
Linux example: https://wiki.gentoo.org/wiki/Trusted_Platform_Module/SSH
macOS example (I haven't tested personally): https://gist.github.com/arianvp/5f59f1783e3eaf1a2d4cd8e952bb...
Or use a FIDO token to protect your SSH key, which becomes useless without the hardware token.
https://wiki.archlinux.org/title/SSH_keys#FIDO/U2F
That's what I do. For those of us too lazy to read the article, tl;dr:
or, if your FIDO token doesn't support edwards curves: tap the token when ssh asks for it, done.Use the ssh key as usual. OpenSSH will ask you to tap the token every time you use it: silent git pushes without you confirming it by tapping the token become impossible. Extracting the key from your machine does nothing — it's useless without the hardware token.
Except that an attacker can modify the ssh config to enable session multiplexing with a long timeout and then piggy-back off that connection, right?
Looks like on the server side this can be mitigated somewhat by the MaxStartups¹ setting for OpenSSH or equivalent behavior for other services that support SSH auth (e.g., Git forges like GitHub):
So it looks like it's possible to support ControlMaster while still somewhat hampering mass-cloning thousands of repos via SSH key without reauthenticating.Admittedly I'd put this more in the category of making endpoint compromise easier to detect than that of actually preventing any particular theft of data or manipulation of systems. But it might still be worth doing! If it means only a few dozen or only a hundred repos get compromised before detection instead of a few thousand, that's a good thing.
Besides all that (or MaxSessions, as another user mentions), if an attacker compromises a developer laptop and can only open those connections as long as the developer is online, that's one thing. But a plaintext key that they can grab and reuse from their own box is obviously an even sweeter prize!
"The SSH key on my YubiKey is useless to attackers" is obviously the wrong way to think about this, but using a smartcard for SSH keys is still a way to avoid storing plaintext secrets. It's good hygiene.
--
https://www.man7.org/linux/man-pages/man5/sshd_config.5.html
Are you saying that because there are still more complicated potential attacks hardware tokens offer zero benefits?
Anyway, what about the sshd server having this config line:
could that help?There is no defense against a compromised laptop. You should prevent this at all cost.
You can make it a bit more challenging for the attacker by using secure enclaves (like TPM or Yubikey), enforce signed commits, etc. but if someone compromised your machine, they can do whatever you can.
Enforcing signing off on commits by multiple people is probably your only bet. But if you have admin creds, an attacker can turn that off, too. So depending on your paranoia level and risk appetite, you need a dedicated machine for admin actions.
It's more nuanced than that. Modern OSes and applications can, and often do, require re-authentication before proceeding with sensitive actions. I can't just run `sudo` without re-authenticating myself; and my ssh agent will reauthenticate me as well. See, e.g., https://developer.1password.com/docs/ssh/agent/security
The malware can wait until you authenticate and perform its actions then in the context of your user session. The malware can also hijack your PATH variable and replace sudo with a wrapper that includes malicious commands.
It can also just get lucky and perform a 'git push' while your SSH agent happens to be unlocked. We don't want to rely on luck here.
Really, it's pointless. Unless you are signing specific actions from an independent piece of hardware [1], the malware can do what you can do. We can talk about the details all day long, and you can make it a bit harder for autonomously acting malware, but at the end of the day it's just a finger exercise to do what they want to do after they compromised your machine.
[1] https://www.reiner-sct.com/en/tan-generators/tan-generator-f... (Note that a display is required so you can see what specific action you are actually signing, in this case it shows amount and recipient bank account number.)
Do you have evidence or a reproducible test case of a successful malware hijack of an ssh session using a Mac and the 1Password agent, or the sudo replacement you suggested? I assume you fully read the link I sent?
I don't think you're necessarily wrong in theory -- but on the other hand you seem to discount taking reasonable (if imperfect) precautionary and defensive measures in favor of an "impossible, therefore don't bother" attitude. Taken to its logical extreme, people with such attitudes would never take risks like driving, or let their children out of the house.
I can type up a test case on my phone:
The malware puts this in your bashrc or equivalent:
In /tmp/malware/bin/sudo: You get the idea. It can do something similar to the git binary and hijack "git commit" such that it will amend whatever it wants and you will happily sign it and push it using your hardened SSH agent.You say it's unlikely, fine, so your risk appetite is sufficiently high. I just want to highlight the risk.
If your machine is compromised, it's game over.
Typical defense against this is to mount all user-writable filesystems as `noexec` but unfortunately most OSes don't do that out of the box.
It could have created a bash alias then. And I don't think a dev wants to be restricted in creating executables. Again, if a dev can do it, so can the malware.
I remember you could trivially circumvent that with „/lib/ld-linux.so <executable>“. Does that no longer work?
noexec now prevents mmaping files on that filesystem as executable.
Kinda hard to work as a software developer then.
This is absolutely not true.
A compromised laptop should always be treated as a fully compromised. However, you can take steps that drastically reduce the likelihood of bad things happening before you can react (e.g. disable accounts/rotate keys).
Further, you can take actions that inherently limit the ability for a compromise to actually cause impact. Not needing to actually store certain things on the machine is a great start.
You can add a gpg key and subkeys to a yubikey and use gpg-agent instead of ssh-agent for ssh auth. When you commit or push, it asks you for a pin for the yubikey to unlock it.
1 store my ssh key in 1Password and use the 1Password ssh agent. This agents asks for access to the key(s) with Touch ID. Either for each access or for each session etc. one can also whitelist programs but I think this all reduces the security.
There is the FIDO feature which means you don’t need to hackle with gpg at all. You can even use an ssh key as signing key to add another layer of security on the GitHub side by only allowing signed commits.
You can put the ssh privkey on the yubikey itself and protect it with a pin.
You can also just generate new ssh keys and protect them with a pin.
You can set up your repo to disable pushing directly to branches like main and require MFA to use the org admin account, so something malicious would need to push to a benign branch and separately be merged into one that deploys come from.
Pushing directly to main seems crazy - for anything that is remotely important I would use a pull request/merge request pattern
There's nothing wrong with pushing to main, as long as you don't blindly treat the head of the main branch as production-ready. It's a branch like any other; Git doesn't care what its name is.
Yea ofc I was implying that main is the branch that is pushed to production.
Depends on the use case of the repo.
But the attacker could just create a branch, merge request and then merge that?
They can't with git by itself, but if you're also signed in to GitHub or BitBucket's CLI with an account able to approve merges they could use those tools.
We require review on PRs before they can be merged.
I’ve started to get more and more paranoid about this. It’s tough when you’re running untrusted code, but I think I’ve improved this by:
not storing SSH keys on the filesystem, and instead using an agent (like 1Password) to mediate access
Stop storing dev secrets/credentials on the filesystem, injecting them into processes with env vars or other mechanisms. Your password manager could have a way to do this.
Develop in a VM separate from your regular computer usage. On windows this is essential anyway through using WSL, but similar things exist for other OSs
Add a password or hardware 2-factor to your ssh key. And get a password manager with the same for those admin credentials.
Your SSH private key must be encrypted using a passphrase. Never store your private key in the clear!
And what do you do with the passphrase, store it encrypted with a passphrase?
This is what agents are for. You load your private key into an agent so you don't have to enter your passphrase every time you use it. Agents are supposed to be hardened so that your private key can't be easily exfiltrated from them. You can then configure `ssh` to pass requests through the agent.
There are lots of agents out there, from the basic `ssh-agent`, to `ssh-agent` integrated with the MacOS keychain (which automatically unlocks when you log in), to 1Password (which is quite nice!).
This is a good defense for malware that only has read access to the filesystem or a stolen hard drive scenario without disk encryption, but does nothing against the compromised dev machine scenario.
This seems to be the standard thing people miss. All the things that make security more convenient also make it weaker. They boast about how "doing thing X" makes them super secure, pat on the back and done. Completely ignoring other avenues they left open.
A case like this brings this out a lot. Compromised dev machine means that anything that doesn't require a separate piece of hardware that asks for your interaction is not going to help. And the more interactions you require for tightening security again the more tedious it becomes and you're likely going to just instinctively press the fob whenever it asks.
Sure, it raises the bar a bit because malware has to take it into account and if there are enough softer targets they may not have bothered. This time.
Classic: you only have to outrun the other guy. Not the lion.
See my comment above; not every SSH agent is alike.
Which one?
Like, I see the comment about the Keychain integration and all that. But in the end I fail to see (without further explanation but I'm eager to learn if there's something I am unaware of) where this isn't different from what I am saying.
Like yes, my ssh key has a passphrase of course. Which is different from my system one actually. As soon as I log into the system I add the key, which means entering the passphrase once, so I don't have to enter it all the time. That would get old real fast. But now ssh can just use my key to do stuff and the agent doesn't know if it's me or I got compromised by npm installing something. And if you add a hardware token you "just have to tap" each time that's a step back into more security but does add tedium. Depending on how often my workflow uses ssh (or something that uses the key) in the background this will become something most people just blindly "tap" on. And then we are back towards less security but with more setup steps, complications and tedium.
I saw the "or allow for a session", which is a step towards security again, because I may be able to allow a script that does several things with ssh with a single tap, which is great of course. Hopefully that cuts the taps down so much that I don't just blindly tap on every request for it. Like the 1password thing you mentioned. If I do lots of things that make it "ask again" often enough I get pushed into "yeah yeah, I know the drill, just tap" security hole.
Keep in mind that not every agent is so naive as to allow a local client to connect to it without reauthenticating somehow.
1Password, for example, will, for each new application, pop up a fingerprint request on my Mac before handling the connection request and allow additional requests for a configurable period of time -- and, by default, it will lock the agent when you lock your machine. It will also request authentication before allowing any new process to make the first connection. See e.g. https://developer.1password.com/docs/ssh/agent/security
You memorize it, or keep it in 1Password. 1Password can manage your SSH keys, and 1Password can/does require a password, so it's still protected with something you know + something you have.
One option is to remember it.
I don’t think that’s considered secure enough, see the other answers and the push for passkeys.
I mean, if passphrases were good for anything you’d directly use them for the ssh connection? :)
Passphrases, when strong enough, are fine when they are not traversing a medium that can be observed by a third party. They're not recommended for authenticating a secure connection over a network, but they’re fine for unlocking a much longer secret that cannot be cracked via guessing, rainbow tables, or other well known means. Hell, most people unlock their phones with a 4 digit passcode, and their computers with a passphrase.
> when they are not traversing a medium that can be observed by a third party
Isn't that why all those security experts are pushing for SSL everywhere and 30 second certificate expiration? To make the medium unobservable by a third party?
If you believe them, passphrases should be okay over fiber you don't control too.
One thing I forgot to mention is what the trust relationship looks like. Passphrases used for authentication are known by both parties and could be leaked by the other side or stolen from them, while private keys remain only available to you. With public key authentication, the other party only has your public key, which is freely shareable.
And yes, we all know that 2FA, passkeys, etc. are all better than passphrases, and that layer 3 wire encryption is important.
I’m merely responding to your blanket assertion that passphrases aren’t “secure enough,” but sometimes they are.
It's secure enough.
> Currently, I have my ssh key on the laptop ...
My SSH keys aren't on my computer: they're safely hidden on a hardware token, behind a secure element, like a Yubikey.
Devices like the Yubikey do precisely exist because computers aren't things to be trusted. So their reason for being is to offer a minimal attack surface.
When I git fetch/pull/push I just do it. But it requires me to physically use my Yubikey. It's not 100% foolproof but it's way better than having SSH keys only protected by a password.
So Git over SSH, on a Git/SSH server that supports Yubikeys.
Not a perfect defense, but sufficient to make your key much harder to exploit: Use a Yubikey (or similar) resident SSH key, with the Yubikey configured to require a touch for each authentication request.
If github, take a look at gh cli or git credential manager:
https://docs.github.com/en/get-started/git-basics/caching-yo...
I wouldn't say that's better. Now your .config directory contains a github token that can do more than just repo pull/push, and it is trivially exfiltrated. Though similar thing could be said for browser cookies.
password-protect your key (preferably with a good password that is not the same password you use to log in to your account). If you use a password it's encrypted; otherwise its stored on plaintext and anybody who manages to get a hold of your laptop can steal the private key.
You can use split-ssh on Qubes OS, such that your development environment won't have the access to your private keys: https://forum.qubes-os.org/t/split-ssh/19060
I am loving the ancient Lovecraftian horror vibe of these exploit names. Good for raising awareness, I guess!
AFAIK Shai-Hulud is the sandworm in Frank Herbert's Dune (but also an American metalcore band)
Shai Hulud is the god that lives inside the sandworms in Dune.
Noted!
The Torvalds commits were a common post infection signature, common in the random repos that published secrets (Microsoft documented https://www.microsoft.com/en-us/security/blog/2025/12/09/sha...)
It was a really noisy worm though, and it looked like a few actors also jumped on the exposed credentials making private repos public and modifying readmes promoting a startup/discord.
NPM post-install scripts considered harmful.
There has to be a tool that allows you (or an AI) to easily review post-install scripts before you install the package.
Could an attacker confuse a reviewing AI by adding a comment to their install script?
As mentioned in the article, good NPM package managers just do this now.
pnpm does it by default, yarn can be configured. Not sure about npm itself.
Got any pointers on how to configure this for yarn? I'm not turning anything up in the yarn documentation or in my random google searches.
npm still seems to be debating whether they even want to do it. One of many reasons I ditched npm for yarn years ago (though the initial impetus was npm's confused and constantly changing behaviors around peer dependencies)
Yarn is unfortunately a dead-end security-wise under current maintainership.
If you are still on yarn v1 I suggest being consistent with '--ignore-scripts --frozen-lockfile' and run any necessary lifecycle scripts for dependencies yourself. There is @lavamoat/allow-scripts to manage this if your project warrants it.
If you are on newer yarn versions I strongly encourage to migrate off to either pnpm or npm.
newer yarn versions are _less_ secure than the ancient/abandoned yarn 1? :(
Any links for further reading on security problems "under current maintainership"?
enableScripts: false in .yarnrc.yml https://yarnpkg.com/configuration/yarnrc#enableScripts
And then opt certain packages back in with dependenciesMeta in package.json https://yarnpkg.com/configuration/manifest#dependenciesMeta....
Obviously blocking install scripts is a good thing, but this is just a false sense of security. If you install a package you will likely execute some code from it too, so the malware can just run then. And that is what the next attack will do as everyone starts using pnpm (or if npm blocks it too).
It's not a false sense of security imo. Code often runs in its own environment, for example a container. We're "used to" sandboxing/ isolating runtime code. It's the package installation process that gets less attention.
The approach the attacker took makes little sense to me, perhaps someone else has an explanation for it? At first they monitored what's going on and then silently exfiltrated credentials and private repos. Makes sense so far. But then why make so much noise with trying to force push repositories? It's Git, surely there's a clone of nearly everything on most dev machines etc.
It's most likely two or more separate attackers operating. The first malware, Shai Hulud 2, exfiltrates credentials from the infected dev machine to new public GitHub repositories. As the repositories are public and searchable via GitHub's interfaces, any malicious attacker aware of the attack can easily grab the credentials and launch any attack, whether it's a noisy destructive script or some sophisticated ransomware.
Malware sometimes suffers from feature creep too.
Given that all the stolen credentials were made public, I was hoping that someone would build a haveibeenpwned style site. We know we were compromised on at least a few tokens, but it would be nice to be able to search using a compromised token to find out what else leaked. We’ve rotated everything we could think of but not knowing if we’ve missed something sucks.
Reach out if you'd like me to check - I did the same for the trigger.dev team in fact[1].
(personal site linked in bio, who links you onward to my linkedin)
[1] https://x.com/ramimacisabird/status/1994598075520749640?s=20
Doesn't it publish the repos to your Github account? Just clone and look at what was stolen.
On the follow up Wiz blog they suggested that the exfiltration was cross-victim https://www.wiz.io/blog/shai-hulud-2-0-aftermath-ongoing-sup...
As the sibling comment said, the worm used stolen GitHub credentials from other victims, and randomly distributed the uploads between victims.
Also everything was double base64 encoded which makes it impossible to use GitHub search.
We don't have a clear explanation of the destructive behavior, right? It looks like it had no real purpose, and there were much more effective ways of destroying their repos. Very script kiddie-like, which does not really fit the main complexity of the virus. Very surprising.
It hides the malware's trail, and disguises which keys were leaked, making rotation harder
The socket.dev deconstruction of the worm (https://socket.dev/blog/shai-hulud-strikes-again-v2) suggests that the destructive actions on GitHub were not part of the malware itself.
Points for an excellent post-mortem.
Would they detect this if the attackers just silently keep leaking the information, as opposed to go destructive about it?
"The simultaneous activity from US and India confirmed we were dealing with a single attacker using multiple VPNs or servers, not separate actors."
Did it really? It's not clear to me why the possibility that the exfiltrated credentials were shared with other actors, each acting independently, is ruled out.
Yeah, that claim makes no sense by itself, and I found it contradictory as one actor accessing repos over two vpns at the same time seems unlikely.
I think if you look at all the actions, and see that they don't overlap, then it might make sense.
If access was: Location: Repo USA: 1,2,3,4 India: 5,6,7,8
The it is reasonable to assume that the access was from that same actor, as it is coordinated in some way.
> This incident involved one of our engineers installing a compromised package on their development machine, which led to credential theft and unauthorized access to our GitHub organization.
The org only has 4-5 engineers. So you can imagine the impact a large org will have.
I'm wondering why storing creds in env variables as plain text is acceptable - e.g. they better be dynamically fetched from a secret manager with 2FA in the way
It’s almost like Microsoft sells security products and runs the most insecure JavaScript package manager to build those security products and couldn’t switch off of it even if the engineers in the org recommended a more secure JavaScript execution context— and that’s realistically why anthropic bought an engine.
> Running npm install is not negligence.
I beg to differ and look forward to running my own fiefdom where interpreter/JIT languages are banned in all forms.
It has nothing to do with interpreters or JIT, it has nothing to do with npm at all. All package managers have the insane security model of "arbitrary code execution with no constraints".
It just so happens that all of those languages share the worst design points, such as the need for a package manager at all and the classic "eval and equivalents run arbitrary code".
>All package managers have the insane security model of "arbitrary code execution with no constraints".
Not all of them, just the most popular ones for these highly sophisticated, well thought-out bunch of absolute languages.
What language does not have a popular package manager that provides code execution?
I tend to agree but think npms post install hook is a degree worse. Triggering during install, silently because npm didn't like someone using the feature to ask for donations, is worse than requiring you to load and run the package code.
Which package managers don't contain an equivalent feature for running code as part of the install process?
Do you really mean this literally? Even the Linux kernel contains tens of thousands of lines of Python, and more lines of shell. Is that undesirable?
Surprised that people allow force-push on git. If it needs to be done, it should only be done after consultation and disabled after.
It was on development branches. The threat actor was trying to delete development work.
Their main branch was already protected. I don't think it makes sense to protect every single branch in a repo? Since not all devs will have the ability to turn this off
[dead]