> On Mac, some command-line tools included with macOS or the Xcode Command Line Tools (including curl, git, python3, ruby, perl, and openssl) check certificates against /etc/ssl/cert.pem. For certificates that chain to a root CA certificate in this file, these tools don't check Apple's block or whether the certificate has been revoked. This file will be updated in a future macOS update. If you use these tools and don't want to wait for the update, you can remove root CA certificates from the file.
This isn't entirely accurate. Plenty of apps run Terminal commands in the background (for some reason). For example Rewind.ai (before they got acquired and installed a kill switch, Adobe Flash-style) runs `top` every two seconds, presumably to figure out its own CPU usage. What the fucking shit? But I think it's not out of the question that other apps run other crap too.
Also, if they truly didn't care about Terminal users, they would not have even published this article!!
eMudhra generated multiple weak private keys and included their public counterpart in multiple intermediate certificates. You need to check that all the intermediates are rejected.
DV/OV/EV are "Domain Validated", "Organization Validated" and "Extended Validation" which refer to different levels of checking that the CA should done before issuing a certificate with a particular name in it.
> This isn't entirely accurate. Plenty of apps run Terminal commands in the background (for some reason).
A lot do it because sometimes there is no convenient library wrapping what they are trying to check or execute that they are aware of, and running a cli command is generally cheap unless you are doing it in a tight loop. A lot of the time this is “it'll do until I find the right library or bindings” code that should have been fixed before release, but it isn't all as bad as running top as you describe.
CLI tools are not to be feared or considered somehow hacky. However much big tech wants people to think that computing is all about some magic apps, Mac OS is still unix, and composing specialized programs together is very much in the spirit of the unix approach to computing.
> CLI tools are not to be feared or considered somehow hacky.
I agree, you're on a *nix OS, calling a CLI tool should be a perfectly valid way to do something.
IMO the real issue is that Apple's built-in CLI tools aren't properly integrated with the rest of the system. Apple's curl used to use the same certificate store as the rest of the OS, then Apple changed it.
>used to use the same certificate store as the rest of the OS
It was technically curl that dropped SecureTransport but I guess because no one at Apple was willing to maintain it and it apparently is a frozen API that doesn't support newer than TLS 1.2 (!). For a while their inbuilt curl could use both but preferred LibreSSL anyway.
You're right, that was just the example I had on hand. Maybe a better example could be something like a video downloader that bundles yt-dlp, that could end up using this certificate store rather than the Apple APIs.
> Plenty of apps run Terminal commands in the background (for some reason).
Huh, I tried to do this a long time ago and permissions bit me and I quit. I was making a nice UI that would run ffmpeg to stitch together some video files, so it would use native UI elements to pick the input files and output location, then generate the ffmpeg command and run it. It worked fine on my machine but it ran into permissions on my friend's machine who needed it and I gave up because they found something better.
They published the article for the people they care above, to make it clear to them they should not worry about all these reports about Apple being vulnerable to this issue.
God I'm so sick of this "everything-is-a-conspiracy" mindset. Equally likely that they're incompetent, that they simply forgot, that it's difficult because of legacy code reasons.
Going against the grain here but I think this is decent security engineering from Apple. They patched the highest exploitable things immediately (Safari, Mail, etc). The residual risk is using these command line tools while someone is actively MITMing your connection.
How often are people “git clone”ing a repo without browsing to the GitHub site first? Safari would throw up certificate errors. And also, TLS on Git doesn’t even authenticate the code you’re downloading is legitimate at all. It could be modified by the Git host or a supply chain attack, etc.
Yea, it’s not great. Ideally the risk is reduced to zero from patching. But a lot of the times we accept some level of cybersecurity risk out of practicality. This is why we have responsible disclosure windows lasting months. It takes a while to do proper release engineering and testing for something like a whole OS update.
One of the certificates literally has a private key of 4; it's as compromised as it could quite possibly be. This is not a "one of the intermediates have a weak and possibly crackable key" or "a bunch of incorrect certs were issued but the issuance was patched".
To a MITM attacker, git clones can be easily distinguished from visiting in Safari. You could choose to only poison git clones; without poisoning anything else.
Heck, you can probably build an exploit kit in a hour; camp out around a Starbucks or McDonalds with open wifi, and start compromising git clones and deliver malicious payloads.
An attacker absolutely cannot “easily distinguish” between a git clone and visiting the website and swap the certificate.
To know whether a request is a git client or web browser, an attacker needs to be able to inspect the contents of the web request (to check the user agent or whatever). If the site is using a legitimate certificate, the attacker has no way to inspect the contents because it is encrypted with a key they do not control.
If an attacker used their own, eMudhra-signed certificate, it would throw the certificate error in Safari.
So let’s take a step back and look at this threat model.
1) An attacker is MITMing your connection. This is already not easily exploitable by most threat actors short of being on a public Wi-Fi network, otherwise you are likely at the nation-state level attacker at this point.
2) An attacker knows exactly which Git repo you’re about to clone and has prepared a malicious version (noting they can’t snoop on your encrypted web traffic).
3) An attacker has developed a way to differentiate between Git over HTTPS and Safari, by using a side channel “fingerprinting” method that works with your exact versions of git and macOS.
4) You don’t pin the git clone to a particular hash or signature.
This is already a very low level of exploitability. All of these things need to be satisfied for the attack you’re describing to work.
Now let’s look at a more practical attack that exists regardless of if you have the patch from Apple.
1) A maintainer of an open source project you use has their GitHub account compromised due to credential stuffing and commits a malicious version.
2) You don’t pin the git clone to a particular hash or signature.
This attack is significantly more exploitable and has happened over and over again in supply chain attacks.
But look at the commonality: in both, no certificate pinning or signature checking was done. If you follow best practices, this stops both attacks dead cold.
This is the security engineering. It’s not just checking boxes. It’s evaluating risks.
> This is the security engineering. It’s not just checking boxes. It’s evaluating risks.
This is absolutely not security engineering. This is nit picking a specific attack factor and trying to argue that it’s impractical. I suspect you are wrong, but it doesn’t matter, because you are ignoring the broader class of attacks and focusing on one specific example. The class issue is that on a Mac, you cannot trust any terminal-based application to honor the CRL. This means that any site accessed from a terminal based application cannot be trusted. There are a billion ways to exploit that.
This is not good security engineering from Apple. There is no way to justify this. Humans make mistakes, and that’s just life, but we shouldn’t pretend that this is an acceptable state or an acceptable response.
So I realize this comment likely won’t be seen but I’m responding anyway because I work on safety critical systems at $DAYJOB. A large part of that is safety and security engineering and risk management and it’s a topic I find interesting.
People have a tendency to misjudge risk. This is a well-established psychological phenomenon, with two of the most common examples being plane crashes and shark attacks. These things happen statistically very rarely, but we often perceive these risks significantly higher than what they actually are. Similar biases lead to situations where someone is deathly afraid of flying, but will happily drive on the expressway while scrolling Instagram on their phone.
Going back to the situation with Apple. Leaving the command line tools trusting the broken certificates feels really bad to me. I don’t like it, probably because it shatters a fundamental security feature that I’m very familiar with and interact with many times a day.
However in systems engineering, it’s hazardous to let our perception of risk dictate how we do things. It leads to blind spots, poor decision making and costly or deadly outcomes. It’s like deciding to drive somewhere instead of flying because it feels safer. This is the same thing as spending a bunch of time to mitigate an actual low-risk cyber threat that feels bad vs an actual high risk threat that feels unlikely.
To work around this, we’ve developed mental models designed to combat our psychological biases. Risk management frameworks, threat modeling, and failure modes & effects analysis are all examples of this.
So I look at the threat modeling I did for this issue. This is the exact same mental framework practically all cybersecurity professionals accept as the proper way to do security engineering, precisely because it allows us to put aside our biases.
Are there “a billion” ways to exploit this issue? I agree with you, yes, there are a ton! But take a step back and look at the commonality between all of them:
1. You are using vulnerable CLI tools on macOS,
2. You are under active MITM attack, and
3. The attacker has developed some sort of payload that exploits your use of the CLI tool.
All of these must be true simultaneously for any attack to succeed.
Let’s start at 1. This by itself filters out probably 90-99% of Mac users. So the exploitability is now reduced to at least a tenth. Now 2. MITM feels very bad and perhaps common occurrence. Maybe because we learn about it in school, or that we have all these security measures to stop it. However, statistically it is very rare. Maybe your corporate network or ISP does, but they’re doing it to serve you ads, or to stop you from leaking trade secrets. They are not launching a highly targeted MITM attack for your specific CLI tools on a MacBook.
Could an attacker set up a rogue Wi-Fi AP in a McDonald’s and get you that way? Yes, absolutely they can. However I contend the actual probability of this happening is very, very low. Maybe if you find yourself updating pip packages frequently at unsecured public WiFi networks you should consider applying Apple’s workaround.
Then 3. Even after all of that, they still need to develop a weaponized payload for your exact usage of the CLI. The icing on the cake is that they have to do all 3 of these things before Apple patches this for good in the next week or so.
Overall, I agree it feels a little shitty of Apple to not have a better fix for this. However, do you really think Apple, one of the largest companies on Earth, skipped threat modeling and decided to just not patch this high profile vulnerability for no reason? No, absolutely not.
What likely happened here was the security engineers at Apple did roughly the same threat modeling I did, came to the conclusion that the exploitability of this is very low, and combined it with their compensating controls (revocation on Safari and other desktop applications) to assess it as an overall low risk. They likely weighed the low risk against an even greater risk of bricking thousands of laptops in a rushed OS update and decided this was the best option.
1) Why wouldn't you have a SSOT for certs and instead rely on patching different apps individually?
2) Doesn't this violate defense in depth? Exploits for a widely disclosed issue now have turnaround activation within hours, not weeks. Not patching all surface areas and actively disclosing as such seems very much like drawing up a map for a jailbroken LLM to gin up a rudimentary attack.
3) Also doesn't this violate the principle of secure by default especially for power users who are more likely to be the ones with higher privileges across distributed systems?
Historically, stuff like openssl and cli command tools used flat files like /etc/ssl/certs.pem.
Browsers came along and they wanted to maintain their own cert store because operating system flat files were often out of date.
Operating systems grew their own stores because they wanted some kind of registry that was more integrated with the rest of their ecosystem.
Nobody coordinated. CLI tools and unix libraries don't care about browsers or O/S registries, operating systems have requirements so that they can't use the flat files, everyone screams about stuff like systemd turning unix into windows.
And it will continue to proliferate. System management tools often ship their own certs since the O/S certs can be out of date and untrustworthy and may not be able to talk to AWS before you bootstrap the management tool and update the system certs because people still run ancient shit like RHEL5. This is more or less the same reason browsers ship their own as well. I've made the decision to ship an embedded cert store in a product before, and nothing has fundamentally changed that would cause me to rethink that decision (even though for people who have an up-to-date O/S it is strictly additional cost that they now have to track a cert store in another product--but you have to build for the lowest common denominator).
1) I would argue that different trust stores are a carryover macOS’s Unix heritage. Having multiple roots of trust is actually very common. For instance Firefox maintains their own root of trust from the system, it’s called NSS. Different programming languages like Java and Rust also often use different roots. It’s not amazing, and has holes, but a lot of it comes down to historical reasons and a certain level of risk acceptance.
2) Yes, but actually, this is exactly why we have defense in depth. This is a temporary measure while a permanent patch is coming. Defense in depth means a single hole does not result in compromise. I would argue that this situation with Apple demonstrates defense in depth actually working as intended. If this was the permanent solution, and no further patch was coming from Apple, then yes, it would violate defense in depth.
It's wild to see people supporting this bizarre anti-security response. Not only are they dragging their feet in fixing the security issue, they're actively reverting any manual fixes silently with each new update. That's legitimately worse than doing nothing at all! It's also disconcerting to see folks advocate for the "let's wait to fix this critical security issue because we want to make it slightly easier in the future" approach, as if Apple is incapable of deleting ~12 lines from a file as part of the normal patch release? If they can revert your manual fixes, they obviously ship that file and so could have shipped a fix with that file. Genuinely disturbing to support any of this.
Yes, a huge part of why I bought Macs for my company is for some of this to be automatically handled. Only now are we getting any communication and still no date on when this will stop being rolled back on updates.
Please, Apple, break the process on getting a fix out for this. I want to stop doing a morning check in with my team about certs.
Which is kinda wild considering there's also a lot of openness zealots, too. People advocating for open source hardware, software, right to repair, etc.
I'd really like to know why it's taking Apple so long to ship a fix. Especially if they can explain how to update an existing computer, what's the general holdup? Is it their release process? Is there some underlying complexity that's poorly understood?
No, this is probably just the result of Apple getting hammered over and over on every release for bugs slipping into the releases and updates.
People were just losing their minds on here over bugs in the 27.2 beta release that hasn't even shipped yet, with wild conspiracy theories about what Apple was doing.
So you wind up with a typical megacorp release process where nothing can happen if it isn't signed off in triplicate and two operators crack open the books with the missile codes and turn the keys at the same time to launch the update.
The result is that shipping a single file for a critical security update takes way too long.
Wonderful AWK command there. For those with hobbies other than AWK, here's what it does:
- set variable s to that string: -v s='=== /C=IN/OU=emSign PKI/O=eMudhra Technologies Limited/CN=emSign ECC Root CA - G3'
- on lines matching the regex between //, set skip to false: /^(===|###) /{skip=0}
- on lines matching s, set skip to true: $0==s{skip=1}
- if skip is false, print the line: !skip
Note the last part where it just says !skip without a curly braces. AWK defaults to print in that case.
AWK is a great little language. But you know how some people get hung up on the useless use of cat, I get triggered whenever I see a grep piped into an awk.
And on the subject of the useless use of cat, I like the form because it makes input a normal command(replaceable by any other command), rather than a shell special form. I sort of wish that "tee" would not default to emitting on stdout to better act as cat's compliment.
I'm rally disappointed in the Apple response here. I can't understand why they wouldn't consider `/etc/ssl/cert.pem` in scope for a security update. This arbitrary boundary has no justification at all, and makes people less secure.
It is in-scope for a security update, and they say one will be coming.
They just couldn’t modify cert.pem without a security update like they did for their own SSL APIs. Hopefully a security update will be coming soon.
There is justification for controlling the scope of security updates. Apple practices incredible defense in depth. This is one of the reasons they can claim such privacy guarantees compared to other vendors. But, I agree that they should have been able to update this file automatically, and hope they will add that capability for the next time something like this happens.
This is some weak sauce. I've just ditched iPhone for GrapheneOS and Apple Watch for Pebble. I'm getting more convinced something like a Framework with Linux is the answer.
Good on Apple for providing such a detailed article. I disagree that they're doing badly by not updating this file remotely on current software versions in the wild -- most likely their Rapid Security Responses / Background Security Improvements framework does not already support this file, which is why they state a full operating system update will be required. Most likely that will be fixed for the future, but in the meantime, being transparent and open about what administrators can do about this now is the RIGHT APPROACH in this situation, and no other vendor has been nearly as thorough about this.
Google, Microsoft, Mozilla, several TLS libraries and language runtimes, and even many Linux distros run by volunteers had this issue comprehensively fixed over a week ago: https://emudhra.we-have-a-problem.com/#developers
Meanwhile, it's unsafe to run `git clone` on macOS unless you manually edit some files, and then edit them again after a macOS update puts the dangerous root certificates back.
Apple has been the most thorough in documenting this transparently and honestly. They didn't have the process for updating that file in the wild, and most likely that will be fixed.
Apple security updates, especially automatic ones, likely don't have the capability to simply write to arbitrary files on the system or run arbitrary commands at install time. Of all vendors, certainly from Apple I'd expect this.
Most likely, other vendors either already used a similar store as the root of trust system-wide (didn't have their own separate security stacks), or their security updates do have the ability to write anywhere or do anything.
And of course, Linux distros and other platforms where the component can be updated separately from the operating system have an even easier time.
This is not Apple intentionally leaving terminal users high and dry. The article literally is the proof otherwise while they no-doubt work on a better solution.
The article states a software update will be required to fix the store:
> can still accept these certificates until a future software update
This suggests they don't have the capability to patch it with Rapid Security Responses / Background Security Improvements, given that they haven't.
Sure, I speculate that they likely would have already if they could have. Sure, I speculate that if they couldn't have, they might add the capability for the future. Sure, I speculate about why they wouldn't have already had a process for this. Whatever whatever. Disregard all that if you want, but the fact remains in the article.
All of that is Apple's fault - they designed and created MacOS. It's not like they're publishing to a third party platform.
Why would I judge Apple only on the quality of this docs page (which is good) but not on their poor work that resulted in this situation in the first place?
This is a legacy BSD store that macOS inherited. Apple has their own security stack that already responded to the incident within 2 hours in most cases, which means all first-party and most third-party apps were already covered. They seemingly just didn't have a mechanism to update the legacy BSD store out-of-band, which is more of an oversight than "poor work".
Apple created and designed the update system for their operating system from scratch. The fact it doesn't include the ability to update a security-critical file quickly is entirely their fault.
Most (if not all) of the popular BSDs managed to ship the update ages ago. I would expect a trillion dollar company like Apple to have better and more thoughtful engineering than random volunteer projects.
> I'm sorry, but how is revoking a bad cert not supported in an easy way? Along with adding a cert and using it to establish a secure connection, that seems to be the core features of a cert store?
> Yet somehow, you manage to come to the conclusion that the vendor requiring manual work is the most thorough? I mean, in documenting their failure, maybe.
> With all their talk of caring about user privacy, this seems incompetent at best.
To be fair other vendors almost universally don't document their failures at all or provide any remediation steps to users whatsoever. I always enjoy the exceptions to this rule, but it's not the norm whatsoever.
You seem to be ignoring that Apple's security framework responded to the incident within about 2 hours in most cases. Safari and other macOS apps were protected right off the bat.
It's only this legacy BSD store that needs an OS update, almost certainly because they simply didn't already have a process in place to update it out-of-band. Because they're vertically integrated, they probably don't have an arbitrary do-anything feature that can paper over the lack of foresight here, like other vendors might.
I'd honestly consider that a feature, but sure, the benefits would outweigh the risks in this one particular case.
They're doing badly because they have a market cap at 5 trillion USD ($5,070,000,000,000 to $5,110,000,000,000) and chose to ditch the first principals of a CA store. Doing impressive backflips to compensate for lack of understanding first principals is exactly that.
Can you explain what first principles you mean? /etc/ssl/cert.pem has been a fairly standard convention on BSDs, and Linux distros usually have an equivalent somewhere too, like for example /etc/ssl/certs/ca-certificates.crt.
I don't blame Apple for overlooking it when they have their own security stack that all their own apps use, which already responded to the incident within two hours of the revocation. As another commenter pointed out, Terminal commands probably just weren't a priority of their threat model. I do hope they fix that, but mentioning their market cap is not gonna help here.
It's on the front page right now. Their dev tools are a complete mess when it comes to actually using the trust store as designed. Why would you dismiss this outright?
I'm not saying it doesn't matter that dev tools are affected. Those are perhaps some of the most important tools and riskiest to leave vulnerable, because that could result in malware shipped to users from a trustworthy developer, or malware executed on the developer's machine, rather than simply phishing. I just don't think it's fair to assume they did it intentionally. It's a mistake, and one that should be fixed, don't get me wrong. But there's a reason it happened and that seems fair enough to me.
In the same degree as involuntary manslaughter is not intentional murder. Sure they made a mistake. Fairness has limits when you're doing it wrong and know better.
The article warned of this already (re-applying the fix after updates). That update was likely already in the pipeline before this debacle. No one can say for sure though.
"AppleCar sometimes explodes on bumpy roads unless you manually disarm the bomb in the trunk before driving. If you do not drive on bumpy roads, you don't need to do anything. If you have disarmed the bomb, Apple Updates will automatically re-arm the bomb after every update. Please remember to manually disarm the bomb after each update."
It's right there in the article, look how proactive they're being!
LOL'd, you're literally correct though. I could unironically see Apple doing that given just a few things. And I'd prefer it to a vendor who stays silent about it until they have a full update out, and then just calls the deaths unfortunate or something. Or like, a vendor who already has the update ready, but withholds it while people continue to blow up because an embargo is more important for some reason.
> most likely their Rapid Security Responses / Background Security Improvements framework does not already support this file
Why would it support this file? Are CA trust stores really that important? /s
> and no other vendor has been nearly as thorough about this
1. That's totally not true. Apple has messed up something that is trivial to fix.
2. The EU's CRA is going to crush them if they don't get their shit together.
They didn't foresee this, and that's on them, sure. But their response to this, given the situation, is just about the best I could expect from them given the circumstances.
Would it have been better if the next operating system update already shipped a fixed store rather than another broken one? Sure. Would it have been even better if they had the ability to remotely fix the store on the many millions of already online Apple devices? Absolutely. But if they couldn't do either of those things, which seems likely at this point, this is the next best thing while they get the rest of their shit in order.
I don't think it's particularly constructive to point out how everybody else has already been able to patch this in all places. Apple's clearly in a spot right now. Blame them for that, but don't blame them for picking the best option at their disposal right now.
> On Mac, some command-line tools included with macOS or the Xcode Command Line Tools (including curl, git, python3, ruby, perl, and openssl) check certificates against /etc/ssl/cert.pem. For certificates that chain to a root CA certificate in this file, these tools don't check Apple's block or whether the certificate has been revoked. This file will be updated in a future macOS update. If you use these tools and don't want to wait for the update, you can remove root CA certificates from the file.
What a clusterfuck
> If you don't use Terminal as part of your workflow, you don't need to do anything.
they solved the issue for the people they care about
This isn't entirely accurate. Plenty of apps run Terminal commands in the background (for some reason). For example Rewind.ai (before they got acquired and installed a kill switch, Adobe Flash-style) runs `top` every two seconds, presumably to figure out its own CPU usage. What the fucking shit? But I think it's not out of the question that other apps run other crap too.
Also, if they truly didn't care about Terminal users, they would not have even published this article!!
> Plenty of apps run Terminal commands in the background
Most notably agents like Claude Code and Codex? Easy to check if you ask it to run:
What is the dv-g3b- part supposed to mean? I.e. why was this specific name chosen for the subdomain?
eMudhra generated multiple weak private keys and included their public counterpart in multiple intermediate certificates. You need to check that all the intermediates are rejected.
See the lists in https://bugzilla.mozilla.org/show_bug.cgi?id=2083449; I believe that "dv-g3b-" corresponds to the intermediate CA certificate listed as "EM DV TLS CA - G3B"
DV/OV/EV are "Domain Validated", "Organization Validated" and "Extended Validation" which refer to different levels of checking that the CA should done before issuing a certificate with a particular name in it.
DV is probably "Domain validated" describing the type of cert and g3b is the specific root cert the chain starts from.
> This isn't entirely accurate. Plenty of apps run Terminal commands in the background (for some reason).
A lot do it because sometimes there is no convenient library wrapping what they are trying to check or execute that they are aware of, and running a cli command is generally cheap unless you are doing it in a tight loop. A lot of the time this is “it'll do until I find the right library or bindings” code that should have been fixed before release, but it isn't all as bad as running top as you describe.
CLI tools are not to be feared or considered somehow hacky. However much big tech wants people to think that computing is all about some magic apps, Mac OS is still unix, and composing specialized programs together is very much in the spirit of the unix approach to computing.
> CLI tools are not to be feared or considered somehow hacky.
I agree, you're on a *nix OS, calling a CLI tool should be a perfectly valid way to do something.
IMO the real issue is that Apple's built-in CLI tools aren't properly integrated with the rest of the system. Apple's curl used to use the same certificate store as the rest of the OS, then Apple changed it.
>used to use the same certificate store as the rest of the OS
It was technically curl that dropped SecureTransport but I guess because no one at Apple was willing to maintain it and it apparently is a frozen API that doesn't support newer than TLS 1.2 (!). For a while their inbuilt curl could use both but preferred LibreSSL anyway.
https://daniel.haxx.se/blog/2025/01/14/secure-transport-supp...
You're right, that was just the example I had on hand. Maybe a better example could be something like a video downloader that bundles yt-dlp, that could end up using this certificate store rather than the Apple APIs.
> Plenty of apps run Terminal commands in the background (for some reason).
Huh, I tried to do this a long time ago and permissions bit me and I quit. I was making a nice UI that would run ffmpeg to stitch together some video files, so it would use native UI elements to pick the input files and output location, then generate the ffmpeg command and run it. It worked fine on my machine but it ran into permissions on my friend's machine who needed it and I gave up because they found something better.
They published the article for the people they care above, to make it clear to them they should not worry about all these reports about Apple being vulnerable to this issue.
God I'm so sick of this "everything-is-a-conspiracy" mindset. Equally likely that they're incompetent, that they simply forgot, that it's difficult because of legacy code reasons.
Is this conspiracy you're talking about in the room with us now?
Going against the grain here but I think this is decent security engineering from Apple. They patched the highest exploitable things immediately (Safari, Mail, etc). The residual risk is using these command line tools while someone is actively MITMing your connection.
How often are people “git clone”ing a repo without browsing to the GitHub site first? Safari would throw up certificate errors. And also, TLS on Git doesn’t even authenticate the code you’re downloading is legitimate at all. It could be modified by the Git host or a supply chain attack, etc.
Yea, it’s not great. Ideally the risk is reduced to zero from patching. But a lot of the times we accept some level of cybersecurity risk out of practicality. This is why we have responsible disclosure windows lasting months. It takes a while to do proper release engineering and testing for something like a whole OS update.
I really don't understand your justification.
One of the certificates literally has a private key of 4; it's as compromised as it could quite possibly be. This is not a "one of the intermediates have a weak and possibly crackable key" or "a bunch of incorrect certs were issued but the issuance was patched".
To a MITM attacker, git clones can be easily distinguished from visiting in Safari. You could choose to only poison git clones; without poisoning anything else.
Heck, you can probably build an exploit kit in a hour; camp out around a Starbucks or McDonalds with open wifi, and start compromising git clones and deliver malicious payloads.
An attacker absolutely cannot “easily distinguish” between a git clone and visiting the website and swap the certificate.
To know whether a request is a git client or web browser, an attacker needs to be able to inspect the contents of the web request (to check the user agent or whatever). If the site is using a legitimate certificate, the attacker has no way to inspect the contents because it is encrypted with a key they do not control.
If an attacker used their own, eMudhra-signed certificate, it would throw the certificate error in Safari.
Yes, I'm assuming network MITM and DPI.
The whole point of TLS and certificates is to counter network MITM attacks.
Browsers and clients emit a lot of fingerprints, you can probably figure it out loosely from the TLS initialisation.
So let’s take a step back and look at this threat model.
1) An attacker is MITMing your connection. This is already not easily exploitable by most threat actors short of being on a public Wi-Fi network, otherwise you are likely at the nation-state level attacker at this point.
2) An attacker knows exactly which Git repo you’re about to clone and has prepared a malicious version (noting they can’t snoop on your encrypted web traffic).
3) An attacker has developed a way to differentiate between Git over HTTPS and Safari, by using a side channel “fingerprinting” method that works with your exact versions of git and macOS.
4) You don’t pin the git clone to a particular hash or signature.
This is already a very low level of exploitability. All of these things need to be satisfied for the attack you’re describing to work.
Now let’s look at a more practical attack that exists regardless of if you have the patch from Apple.
1) A maintainer of an open source project you use has their GitHub account compromised due to credential stuffing and commits a malicious version.
2) You don’t pin the git clone to a particular hash or signature.
This attack is significantly more exploitable and has happened over and over again in supply chain attacks.
But look at the commonality: in both, no certificate pinning or signature checking was done. If you follow best practices, this stops both attacks dead cold.
This is the security engineering. It’s not just checking boxes. It’s evaluating risks.
> This is the security engineering. It’s not just checking boxes. It’s evaluating risks.
This is absolutely not security engineering. This is nit picking a specific attack factor and trying to argue that it’s impractical. I suspect you are wrong, but it doesn’t matter, because you are ignoring the broader class of attacks and focusing on one specific example. The class issue is that on a Mac, you cannot trust any terminal-based application to honor the CRL. This means that any site accessed from a terminal based application cannot be trusted. There are a billion ways to exploit that.
This is not good security engineering from Apple. There is no way to justify this. Humans make mistakes, and that’s just life, but we shouldn’t pretend that this is an acceptable state or an acceptable response.
So I realize this comment likely won’t be seen but I’m responding anyway because I work on safety critical systems at $DAYJOB. A large part of that is safety and security engineering and risk management and it’s a topic I find interesting.
People have a tendency to misjudge risk. This is a well-established psychological phenomenon, with two of the most common examples being plane crashes and shark attacks. These things happen statistically very rarely, but we often perceive these risks significantly higher than what they actually are. Similar biases lead to situations where someone is deathly afraid of flying, but will happily drive on the expressway while scrolling Instagram on their phone.
Going back to the situation with Apple. Leaving the command line tools trusting the broken certificates feels really bad to me. I don’t like it, probably because it shatters a fundamental security feature that I’m very familiar with and interact with many times a day.
However in systems engineering, it’s hazardous to let our perception of risk dictate how we do things. It leads to blind spots, poor decision making and costly or deadly outcomes. It’s like deciding to drive somewhere instead of flying because it feels safer. This is the same thing as spending a bunch of time to mitigate an actual low-risk cyber threat that feels bad vs an actual high risk threat that feels unlikely.
To work around this, we’ve developed mental models designed to combat our psychological biases. Risk management frameworks, threat modeling, and failure modes & effects analysis are all examples of this.
So I look at the threat modeling I did for this issue. This is the exact same mental framework practically all cybersecurity professionals accept as the proper way to do security engineering, precisely because it allows us to put aside our biases.
Are there “a billion” ways to exploit this issue? I agree with you, yes, there are a ton! But take a step back and look at the commonality between all of them:
1. You are using vulnerable CLI tools on macOS,
2. You are under active MITM attack, and
3. The attacker has developed some sort of payload that exploits your use of the CLI tool.
All of these must be true simultaneously for any attack to succeed.
Let’s start at 1. This by itself filters out probably 90-99% of Mac users. So the exploitability is now reduced to at least a tenth. Now 2. MITM feels very bad and perhaps common occurrence. Maybe because we learn about it in school, or that we have all these security measures to stop it. However, statistically it is very rare. Maybe your corporate network or ISP does, but they’re doing it to serve you ads, or to stop you from leaking trade secrets. They are not launching a highly targeted MITM attack for your specific CLI tools on a MacBook.
Could an attacker set up a rogue Wi-Fi AP in a McDonald’s and get you that way? Yes, absolutely they can. However I contend the actual probability of this happening is very, very low. Maybe if you find yourself updating pip packages frequently at unsecured public WiFi networks you should consider applying Apple’s workaround.
Then 3. Even after all of that, they still need to develop a weaponized payload for your exact usage of the CLI. The icing on the cake is that they have to do all 3 of these things before Apple patches this for good in the next week or so.
Overall, I agree it feels a little shitty of Apple to not have a better fix for this. However, do you really think Apple, one of the largest companies on Earth, skipped threat modeling and decided to just not patch this high profile vulnerability for no reason? No, absolutely not.
What likely happened here was the security engineers at Apple did roughly the same threat modeling I did, came to the conclusion that the exploitability of this is very low, and combined it with their compensating controls (revocation on Safari and other desktop applications) to assess it as an overall low risk. They likely weighed the low risk against an even greater risk of bricking thousands of laptops in a rushed OS update and decided this was the best option.
That’s what security engineering is.
Some questions from a netsec novice:
1) Why wouldn't you have a SSOT for certs and instead rely on patching different apps individually?
2) Doesn't this violate defense in depth? Exploits for a widely disclosed issue now have turnaround activation within hours, not weeks. Not patching all surface areas and actively disclosing as such seems very much like drawing up a map for a jailbroken LLM to gin up a rudimentary attack.
3) Also doesn't this violate the principle of secure by default especially for power users who are more likely to be the ones with higher privileges across distributed systems?
Historically, stuff like openssl and cli command tools used flat files like /etc/ssl/certs.pem.
Browsers came along and they wanted to maintain their own cert store because operating system flat files were often out of date.
Operating systems grew their own stores because they wanted some kind of registry that was more integrated with the rest of their ecosystem.
Nobody coordinated. CLI tools and unix libraries don't care about browsers or O/S registries, operating systems have requirements so that they can't use the flat files, everyone screams about stuff like systemd turning unix into windows.
And it will continue to proliferate. System management tools often ship their own certs since the O/S certs can be out of date and untrustworthy and may not be able to talk to AWS before you bootstrap the management tool and update the system certs because people still run ancient shit like RHEL5. This is more or less the same reason browsers ship their own as well. I've made the decision to ship an embedded cert store in a product before, and nothing has fundamentally changed that would cause me to rethink that decision (even though for people who have an up-to-date O/S it is strictly additional cost that they now have to track a cert store in another product--but you have to build for the lowest common denominator).
1) I would argue that different trust stores are a carryover macOS’s Unix heritage. Having multiple roots of trust is actually very common. For instance Firefox maintains their own root of trust from the system, it’s called NSS. Different programming languages like Java and Rust also often use different roots. It’s not amazing, and has holes, but a lot of it comes down to historical reasons and a certain level of risk acceptance.
2) Yes, but actually, this is exactly why we have defense in depth. This is a temporary measure while a permanent patch is coming. Defense in depth means a single hole does not result in compromise. I would argue that this situation with Apple demonstrates defense in depth actually working as intended. If this was the permanent solution, and no further patch was coming from Apple, then yes, it would violate defense in depth.
3) Related to above, yes. But it’s not permanent.
> How often are people “git clone”ing a repo without browsing to the GitHub site first?
People are going to be running pull or push much more than clone, and will often be running those without opening the origin on Safari.
> How often are people “git clone”ing a repo without browsing to the GitHub site first?
What’s the point of this question? Would going to GitHub.com in my browser somehow protect me from a MITM attack here?
It's wild to see people supporting this bizarre anti-security response. Not only are they dragging their feet in fixing the security issue, they're actively reverting any manual fixes silently with each new update. That's legitimately worse than doing nothing at all! It's also disconcerting to see folks advocate for the "let's wait to fix this critical security issue because we want to make it slightly easier in the future" approach, as if Apple is incapable of deleting ~12 lines from a file as part of the normal patch release? If they can revert your manual fixes, they obviously ship that file and so could have shipped a fix with that file. Genuinely disturbing to support any of this.
Yes, a huge part of why I bought Macs for my company is for some of this to be automatically handled. Only now are we getting any communication and still no date on when this will stop being rolled back on updates.
Please, Apple, break the process on getting a fix out for this. I want to stop doing a morning check in with my team about certs.
I don't think it's controversial to say that there are a lot of Apple zealots here.
Which is kinda wild considering there's also a lot of openness zealots, too. People advocating for open source hardware, software, right to repair, etc.
I'd really like to know why it's taking Apple so long to ship a fix. Especially if they can explain how to update an existing computer, what's the general holdup? Is it their release process? Is there some underlying complexity that's poorly understood?
Probably they didn't want to hold up the 27.2 release by throwing this in??
Did they forget about their “Background Security Improvements” feature??
https://support.apple.com/en-us/102657
Only CLI users are affected.
They can follow the 4 steps (3 was not optional for me) to remediate.
Many users have no idea they are cli users, hence they would never attempt to remediate.
No, this is just laziness or stupidity on apples part.
No, this is probably just the result of Apple getting hammered over and over on every release for bugs slipping into the releases and updates.
People were just losing their minds on here over bugs in the 27.2 beta release that hasn't even shipped yet, with wild conspiracy theories about what Apple was doing.
So you wind up with a typical megacorp release process where nothing can happen if it isn't signed off in triplicate and two operators crack open the books with the missile codes and turn the keys at the same time to launch the update.
The result is that shipping a single file for a critical security update takes way too long.
Related: Ten days later, Git on macOS still trusts the compromised eMudhra certificates https://news.ycombinator.com/item?id=50271640
Wonderful AWK command there. For those with hobbies other than AWK, here's what it does:
Note the last part where it just says !skip without a curly braces. AWK defaults to print in that case.AWK is a great hobby, if anyone's looking to get into a new one. If sed is map and grep is filter, then awk is reduce.
AWK is a great little language. But you know how some people get hung up on the useless use of cat, I get triggered whenever I see a grep piped into an awk.
And on the subject of the useless use of cat, I like the form because it makes input a normal command(replaceable by any other command), rather than a shell special form. I sort of wish that "tee" would not default to emitting on stdout to better act as cat's compliment.
I'm rally disappointed in the Apple response here. I can't understand why they wouldn't consider `/etc/ssl/cert.pem` in scope for a security update. This arbitrary boundary has no justification at all, and makes people less secure.
Come on Apple, do better.
It is in-scope for a security update, and they say one will be coming. They just couldn’t modify cert.pem without a security update like they did for their own SSL APIs. Hopefully a security update will be coming soon.
Ah, taking weeks for updates in a security environment where minutes matter!
There is justification for controlling the scope of security updates. Apple practices incredible defense in depth. This is one of the reasons they can claim such privacy guarantees compared to other vendors. But, I agree that they should have been able to update this file automatically, and hope they will add that capability for the next time something like this happens.
This is some weak sauce. I've just ditched iPhone for GrapheneOS and Apple Watch for Pebble. I'm getting more convinced something like a Framework with Linux is the answer.
that should really be a security update
It is, isn't it?
Apple states in the article that a full operating system update will be required to fix the file.
A full operating system update, which includes auto-enabling a bunch of unwanted AI features and 50-GB of local models. Ugh.
Good on Apple for providing such a detailed article. I disagree that they're doing badly by not updating this file remotely on current software versions in the wild -- most likely their Rapid Security Responses / Background Security Improvements framework does not already support this file, which is why they state a full operating system update will be required. Most likely that will be fixed for the future, but in the meantime, being transparent and open about what administrators can do about this now is the RIGHT APPROACH in this situation, and no other vendor has been nearly as thorough about this.
Google, Microsoft, Mozilla, several TLS libraries and language runtimes, and even many Linux distros run by volunteers had this issue comprehensively fixed over a week ago: https://emudhra.we-have-a-problem.com/#developers
Meanwhile, it's unsafe to run `git clone` on macOS unless you manually edit some files, and then edit them again after a macOS update puts the dangerous root certificates back.
In what world has Apple been the most "thorough"?
Apple has been the most thorough in documenting this transparently and honestly. They didn't have the process for updating that file in the wild, and most likely that will be fixed.
Apple security updates, especially automatic ones, likely don't have the capability to simply write to arbitrary files on the system or run arbitrary commands at install time. Of all vendors, certainly from Apple I'd expect this.
Most likely, other vendors either already used a similar store as the root of trust system-wide (didn't have their own separate security stacks), or their security updates do have the ability to write anywhere or do anything.
And of course, Linux distros and other platforms where the component can be updated separately from the operating system have an even easier time.
This is not Apple intentionally leaving terminal users high and dry. The article literally is the proof otherwise while they no-doubt work on a better solution.
There are so many "likely" in this comment that I don't know if I should care about a single word at all. Looks pure speculation if not fiction to me.
Looking forward to a comment that is much more grounded than substantial than this.
The article states a software update will be required to fix the store:
> can still accept these certificates until a future software update
This suggests they don't have the capability to patch it with Rapid Security Responses / Background Security Improvements, given that they haven't.
Sure, I speculate that they likely would have already if they could have. Sure, I speculate that if they couldn't have, they might add the capability for the future. Sure, I speculate about why they wouldn't have already had a process for this. Whatever whatever. Disregard all that if you want, but the fact remains in the article.
All of that is Apple's fault - they designed and created MacOS. It's not like they're publishing to a third party platform.
Why would I judge Apple only on the quality of this docs page (which is good) but not on their poor work that resulted in this situation in the first place?
This is a legacy BSD store that macOS inherited. Apple has their own security stack that already responded to the incident within 2 hours in most cases, which means all first-party and most third-party apps were already covered. They seemingly just didn't have a mechanism to update the legacy BSD store out-of-band, which is more of an oversight than "poor work".
Apple created and designed the update system for their operating system from scratch. The fact it doesn't include the ability to update a security-critical file quickly is entirely their fault.
Most (if not all) of the popular BSDs managed to ship the update ages ago. I would expect a trillion dollar company like Apple to have better and more thoughtful engineering than random volunteer projects.
They didn't inherit it. They chose it. If you choose to build on something, you should support it, and support it well.
"They're not fixing the security issue, but at least they're documenting it" is a weird argument to make.
My reply to a deleted reply:
> I'm sorry, but how is revoking a bad cert not supported in an easy way? Along with adding a cert and using it to establish a secure connection, that seems to be the core features of a cert store?
> Yet somehow, you manage to come to the conclusion that the vendor requiring manual work is the most thorough? I mean, in documenting their failure, maybe.
> With all their talk of caring about user privacy, this seems incompetent at best.
To be fair other vendors almost universally don't document their failures at all or provide any remediation steps to users whatsoever. I always enjoy the exceptions to this rule, but it's not the norm whatsoever.
You seem to be ignoring that Apple's security framework responded to the incident within about 2 hours in most cases. Safari and other macOS apps were protected right off the bat.
It's only this legacy BSD store that needs an OS update, almost certainly because they simply didn't already have a process in place to update it out-of-band. Because they're vertically integrated, they probably don't have an arbitrary do-anything feature that can paper over the lack of foresight here, like other vendors might.
I'd honestly consider that a feature, but sure, the benefits would outweigh the risks in this one particular case.
They're doing badly because they have a market cap at 5 trillion USD ($5,070,000,000,000 to $5,110,000,000,000) and chose to ditch the first principals of a CA store. Doing impressive backflips to compensate for lack of understanding first principals is exactly that.
Can you explain what first principles you mean? /etc/ssl/cert.pem has been a fairly standard convention on BSDs, and Linux distros usually have an equivalent somewhere too, like for example /etc/ssl/certs/ca-certificates.crt.
I don't blame Apple for overlooking it when they have their own security stack that all their own apps use, which already responded to the incident within two hours of the revocation. As another commenter pointed out, Terminal commands probably just weren't a priority of their threat model. I do hope they fix that, but mentioning their market cap is not gonna help here.
It's on the front page right now. Their dev tools are a complete mess when it comes to actually using the trust store as designed. Why would you dismiss this outright?
https://emudhra.we-have-a-problem.com/apple.html
I'm not saying it doesn't matter that dev tools are affected. Those are perhaps some of the most important tools and riskiest to leave vulnerable, because that could result in malware shipped to users from a trustworthy developer, or malware executed on the developer's machine, rather than simply phishing. I just don't think it's fair to assume they did it intentionally. It's a mistake, and one that should be fixed, don't get me wrong. But there's a reason it happened and that seems fair enough to me.
In the same degree as involuntary manslaughter is not intentional murder. Sure they made a mistake. Fairness has limits when you're doing it wrong and know better.
They may not have known better back then, even if by now it seems obvious.
Where exactly is it good, that:
* apple told admins how to fix the vulnerability * apple shipped update 27.2, with no fix (today) * the update reverted the admin fix quietly
The article warned of this already (re-applying the fix after updates). That update was likely already in the pipeline before this debacle. No one can say for sure though.
"AppleCar sometimes explodes on bumpy roads unless you manually disarm the bomb in the trunk before driving. If you do not drive on bumpy roads, you don't need to do anything. If you have disarmed the bomb, Apple Updates will automatically re-arm the bomb after every update. Please remember to manually disarm the bomb after each update."
It's right there in the article, look how proactive they're being!
LOL'd, you're literally correct though. I could unironically see Apple doing that given just a few things. And I'd prefer it to a vendor who stays silent about it until they have a full update out, and then just calls the deaths unfortunate or something. Or like, a vendor who already has the update ready, but withholds it while people continue to blow up because an embargo is more important for some reason.
> most likely their Rapid Security Responses / Background Security Improvements framework does not already support this file
Why would it support this file? Are CA trust stores really that important? /s
> and no other vendor has been nearly as thorough about this
1. That's totally not true. Apple has messed up something that is trivial to fix. 2. The EU's CRA is going to crush them if they don't get their shit together.
They didn't foresee this, and that's on them, sure. But their response to this, given the situation, is just about the best I could expect from them given the circumstances.
Would it have been better if the next operating system update already shipped a fixed store rather than another broken one? Sure. Would it have been even better if they had the ability to remotely fix the store on the many millions of already online Apple devices? Absolutely. But if they couldn't do either of those things, which seems likely at this point, this is the next best thing while they get the rest of their shit in order.
I don't think it's particularly constructive to point out how everybody else has already been able to patch this in all places. Apple's clearly in a spot right now. Blame them for that, but don't blame them for picking the best option at their disposal right now.
Is there a one-liner fix for this?
4 lines, at the bottom of the article
Don't have a stranger here summarize/concatenate them for you
Good point. I am reaching vulnerability fatigue in these past months tbh.