Funny to see this after I spent the morning troubleshooting and fixing my crowdsec install on my debian 13 vps. Apparently they stopped supplying a community blocklist to my machine because I'm running the old debian packaged version instead of directly from them (http 500). I had a LLM build a blocklist from publicly available sources rather than tie myself more tightly to their SaaS platform.
I feel like a massive blocklist is simultaneously exactly the kind of thing an LLM is likely to hallucinate and exactly the kind of thing you don't want to rely on an LLM for.
If they had it build it from public sources then I assume it was combined via tool usage rather than direct generation. Beyond the hallucination risk, the latter would just be a waste of tokens & time.
> the Tanstack compromise is very likely to have been the leak vector
....appears to have been backdoored to extract an API key with authorization to read the private codebase.
...
> immediately rotated all required tokens & credentials to prevent further incidents.
Rotating the API key doesn't quite put them in a position to "prevent further incidents" does it? The next PyPI/npm supply chain issue will just get the new key?
I suppose whatever they use that key for should be reviewed and re-scoped if possible?
Does github let you restrict where you can originate requests using a given API key? or are we just not there yet?
On the funny side, reading the website tagline, apparently they claim to know who is attacking you, they just happen to miss out on who attacked them.
Turns out they are not really a security company, just an aggregator of bad IPs. Ideally this kind of aggregator problem is best suited for a trusted not-for-profit company where providing the data needs some level of credibility and querying the data costs you nominal fee to keep the setup floating.
It wasn't the Github that was compromised, it was the access to their private repository that was compromised so somewhere down the line the security best practices are in question for sure. Self hosted repos available on public internet would have met the same fate, may be worse, given github does provides some level of security.
Even regarding the blast radius, I do not really believe any company is honest about it. They do not have tools to verify it, if the user information was accessed with leaked token or real token. The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility. Any platform engineer knows that your CICD system has the keys to the kingdom.
The article you're commenting on says how they were compromised.
> The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility
That's not how this works, at all. You need to have enough evidence that you can confidently demonstrate that there is no sign of a broader breach. If you get sued and can't do that, you're in trouble, because being unable to do that shows that you were acting negligently.
What, exactly, is the definition of a "security company" in your mind? Threat Intel companies definitely fall under that normally, so I'm curious what you think it is.
Also, the idea that this type of thing could just be stood up as a "not-for-profit" company and ran for peanuts is kind of silly. How would the nominal fee pay for the engineers and infrastructure? Or would this just be a volunteer effort and you'd like people to do this for free for you?
Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
From what I have seen a large chunk of internet exists and stands on the shoulder of folks who did the volunteer work cause they were passionate about it and enjoyed that part. Once built, the nominal fee for API to check IP address should cover the costs way easily for the servers.
Letsencrypt is a great example, it did took away the big money from all these commercial CA's, who used to issue blue, green and what not kind of checkmarks. Thats one big reason reason why the migration to HTTPS happened faster.
The basic software is open source, and the list is free if you're running the tool and contributing detections back. They do have some curated lists that you have to pay for.
It's quite a bit less expensive than most other commercial products of this kind that I've looked at.
So I guess your answer is yes, you would like someone to do this work for you for free, because others have done other work for free. And then once the free work is done, you will happily pay a nominal fee to gain the benefit of all the free work?
> Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
This could, quite literally, be any company on earth then? But not CrowdSec, because they had a single security issue? Every company on earth has had those, including every security company.
There is discussion and then there is argument. Discussion is more about sharing and learning where as argument becomes all about proving a point, specifically individuals targeted point of view.
My observation is generic and more about state of things rather than focusing on one entity.
Also lets not belittle the hard work by just labelling it out as free. If only money had been motivation for everyone then the world would have been a different place. And like a lot of people I have done my share of passionate work that provided satisfaction to me and money for others, thats way tangential though
> My observation is generic and more about state of things
You are the one who stated a specific company wasn't a security company. When asked to define what one was, your definition included any company on the planet.
> Also lets not belittle the hard work by just labelling it out as free
Who is belittling it? My point is that it's hard work and should be rewarded, not expected by someone to be done for them.
fallacy and apologist point of view here. Shall we dig in?
fallacy is "Oh I guess you mean" .. insert large unsolvable and probably unpopular derailment. Bonus points for aggressive labeling of the opponent being opposed to money.
apologist - "Any company on earth" .. We all stand together, Every Company On Earth .. does this warrent serious replies?
> aggressive labeling of the opponent being opposed to money
What? Where, exactly, did I do that? I stated they were opposed to _spending money_ on a service that they would find useful.
> Any company on earth
Their definition of a "security company" would include almost any type of company on earth, as long as they took security seriously. I asked them for a definition of a security company, since they claimed that CrowdSec wasn't one, and their definition had no boundaries.
Improve your reading comprehension and maybe study some actual logic, not whatever reddit nonsense you've got going here. But other than that, A+ work dude. I'm sure you were really impressed with what you wrote. "Shall we dig in?" ... ffs.
We implemented CrowdSec for bot/scraping mitigation. The architecture is sound, but it ended up having an unacceptable false positive rate for us. This may be an issue with any kind of IP reputation approach. After a couple of months of work getting it ready to go I had to turn it off after a couple of days.
We had the main community blocklist and several of their pricey paid blocklists enabled in a PoC capacity. We had a lot of legitimate users end up blocked. In some cases these may have been VPN exit nodes, or users on CG-NAT, or devices on a shared network with some other compromised / bot device. I didn't get 100% of the details, just that we were inundated with support requests from real users that ended up blocked.
Given the number of residential proxies I see scraping the couple of sites I have responsibility for I don't find IP address-based blocking useful anymore. That ship has sailed.
Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).
I run an attractive site to AI scrapers and yes the IP ship has indeed sailed in favor of residential proxies. They are non-stop and could have ingested all of our content many times over every day. JA4 is useless too because they have the ability to spoof this or use a very common fingerprint like Chrome that you cannot block without blowback. These bots will only use a given IP a couple of dozen times and then use another node. They use 10s of thousands of IPs. You can tell because they will come in waves from a residential or mobile network
What kind of fingerprinting are you thinking of? JA4? I haven't found a way to do that inexpensively at our scale, but we may have to go that route - looking at CloudFront bot mitigation.
For behavioral, we have Anubis honeypot functionality turned on, but it doesn't seem to be effective for 99% of scrapers. Anubis is also running behind TLS termination, so I don't think it can do full JA4. It does have the less robust JA4H apparently, but I'm not sure how effective that will be.
Edit: Oh yeah, forgot to mention - it's almost 100% residential proxies. Primarily China Telecom and China Unicom. Unfortunately those providers are HUGE and also host a ton of legitimate users all over Asia.
> What kind of fingerprinting are you thinking of?
I'm minding a proprietary application where this is being discussed. The vendor is talking about running Javascript in the client to detect headless browsers, bots, etc. I'm guessing they're going to license something off-the-shelf. The technical specifics aren't being discussed. They're at the "wow the stakeholders" phase.
I'd like to do something ourselves but, at the size of the org I'm talking about (with virtually no in-house dev resources), COTS rules the day.
Interesting, did you implement only IP reputation (via blocklist) or did you deploy the WAF as well? Regarding bot scrapping, you would probably want to try the new bot detection feature recently released
I have written my own honeypots to reduce the false positive rate. I simply have things like a VM with RDP and SSH open to the internet and any IP that tries to login gets banned at the firewall for x days. It works really well.
Sounds like an exploit took the credentials needed to extract the code, makes me wonder if a Ubikey + SSL cert for git access would have prevented the entire leak.
> CrowdSec source code consists of two parts: a private one and another that hosts our Free Open Source Software (i.e., the Security Engine), which is public by design and therefore out of scope. The private part, though, contains the source code for our SaaS console, some AWS Cloud routines, some connectors, and automations.
This leak seems incredibly mundane. customers would have the security engine installed, but not the console.
> CrowdSec source code consists of two parts: a private one and another that hosts our Free Open Source Software (i.e., the Security Engine), which is public by design and therefore out of scope.
yet another security oriented company that doesn’t practice what they preach.
On the flip side, there was allegedly no PII leaked. But this event is still a red flag as it means their internal ops are absolutely shit. So it’s another vendor receiving a PNG flag.
Funny to see this after I spent the morning troubleshooting and fixing my crowdsec install on my debian 13 vps. Apparently they stopped supplying a community blocklist to my machine because I'm running the old debian packaged version instead of directly from them (http 500). I had a LLM build a blocklist from publicly available sources rather than tie myself more tightly to their SaaS platform.
Wait … that’s lame. If it’s in Debian repos I expect it to work with full functionality (up to the version number)
I can understand the myriad reasons for why a company would be inconvenienced by that.
But on the other hand, it seems like a reason for me to never use the vendor
I feel like a massive blocklist is simultaneously exactly the kind of thing an LLM is likely to hallucinate and exactly the kind of thing you don't want to rely on an LLM for.
If they had it build it from public sources then I assume it was combined via tool usage rather than direct generation. Beyond the hallucination risk, the latter would just be a waste of tokens & time.
Modern agent harnesses rely heavily on tools, as well as writing code, to avoid hallucinations affecting deterministic values.
Which means it'd be a lot cheaper, to use tools, and public sources, and just automate it the traditional way...
Ah, I have to check that, thanks!
Ah, I just learned this, thanks for the heads up.
I suppose whatever they use that key for should be reviewed and re-scoped if possible?
Does github let you restrict where you can originate requests using a given API key? or are we just not there yet?
On the funny side, reading the website tagline, apparently they claim to know who is attacking you, they just happen to miss out on who attacked them.
Turns out they are not really a security company, just an aggregator of bad IPs. Ideally this kind of aggregator problem is best suited for a trusted not-for-profit company where providing the data needs some level of credibility and querying the data costs you nominal fee to keep the setup floating.
I suppose CrowdSec isn't used to analyze GitHub's traffic, so they wouldn't have much info to go on.
If they had self-hosted their own repos, they might have had more luck.
It wasn't the Github that was compromised, it was the access to their private repository that was compromised so somewhere down the line the security best practices are in question for sure. Self hosted repos available on public internet would have met the same fate, may be worse, given github does provides some level of security.
Even regarding the blast radius, I do not really believe any company is honest about it. They do not have tools to verify it, if the user information was accessed with leaked token or real token. The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility. Any platform engineer knows that your CICD system has the keys to the kingdom.
The article you're commenting on says how they were compromised.
> The thing that works in their favor is that no one else can verify it either which absolves them from any responsibility
That's not how this works, at all. You need to have enough evidence that you can confidently demonstrate that there is no sign of a broader breach. If you get sued and can't do that, you're in trouble, because being unable to do that shows that you were acting negligently.
GitHub Enterprise has at least some level audit log
What, exactly, is the definition of a "security company" in your mind? Threat Intel companies definitely fall under that normally, so I'm curious what you think it is.
Also, the idea that this type of thing could just be stood up as a "not-for-profit" company and ran for peanuts is kind of silly. How would the nominal fee pay for the engineers and infrastructure? Or would this just be a volunteer effort and you'd like people to do this for free for you?
Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
From what I have seen a large chunk of internet exists and stands on the shoulder of folks who did the volunteer work cause they were passionate about it and enjoyed that part. Once built, the nominal fee for API to check IP address should cover the costs way easily for the servers.
Letsencrypt is a great example, it did took away the big money from all these commercial CA's, who used to issue blue, green and what not kind of checkmarks. Thats one big reason reason why the migration to HTTPS happened faster.
The basic software is open source, and the list is free if you're running the tool and contributing detections back. They do have some curated lists that you have to pay for.
It's quite a bit less expensive than most other commercial products of this kind that I've looked at.
So I guess your answer is yes, you would like someone to do this work for you for free, because others have done other work for free. And then once the free work is done, you will happily pay a nominal fee to gain the benefit of all the free work?
> Any company where understanding of security practices has a direct impact on its revenue from early phases can be considered as a security company in my view.
This could, quite literally, be any company on earth then? But not CrowdSec, because they had a single security issue? Every company on earth has had those, including every security company.
There is discussion and then there is argument. Discussion is more about sharing and learning where as argument becomes all about proving a point, specifically individuals targeted point of view.
My observation is generic and more about state of things rather than focusing on one entity.
Also lets not belittle the hard work by just labelling it out as free. If only money had been motivation for everyone then the world would have been a different place. And like a lot of people I have done my share of passionate work that provided satisfaction to me and money for others, thats way tangential though
> My observation is generic and more about state of things
You are the one who stated a specific company wasn't a security company. When asked to define what one was, your definition included any company on the planet.
> Also lets not belittle the hard work by just labelling it out as free
Who is belittling it? My point is that it's hard work and should be rewarded, not expected by someone to be done for them.
fallacy and apologist point of view here. Shall we dig in?
fallacy is "Oh I guess you mean" .. insert large unsolvable and probably unpopular derailment. Bonus points for aggressive labeling of the opponent being opposed to money.
apologist - "Any company on earth" .. We all stand together, Every Company On Earth .. does this warrent serious replies?
What in the world are you going on about?
> aggressive labeling of the opponent being opposed to money
What? Where, exactly, did I do that? I stated they were opposed to _spending money_ on a service that they would find useful.
> Any company on earth
Their definition of a "security company" would include almost any type of company on earth, as long as they took security seriously. I asked them for a definition of a security company, since they claimed that CrowdSec wasn't one, and their definition had no boundaries.
Improve your reading comprehension and maybe study some actual logic, not whatever reddit nonsense you've got going here. But other than that, A+ work dude. I'm sure you were really impressed with what you wrote. "Shall we dig in?" ... ffs.
We implemented CrowdSec for bot/scraping mitigation. The architecture is sound, but it ended up having an unacceptable false positive rate for us. This may be an issue with any kind of IP reputation approach. After a couple of months of work getting it ready to go I had to turn it off after a couple of days.
Not sure what you implemented.
They provide several IP blacklists. None of those seem to be false positives. You can also add custom 3rd party blocklists.
They also provide several different rulesets. It is up to you to choose which ones to use and fine tune. LLMs can be very helpful with that.
And there are 3rd party dashboards and tools that help you manage it more easily.
I use the free version as a simple WAF on multiple servers and it blocks a lot of bots. It did require some initial finetuning though.
Are there any better open source solutions?
We had the main community blocklist and several of their pricey paid blocklists enabled in a PoC capacity. We had a lot of legitimate users end up blocked. In some cases these may have been VPN exit nodes, or users on CG-NAT, or devices on a shared network with some other compromised / bot device. I didn't get 100% of the details, just that we were inundated with support requests from real users that ended up blocked.
Given the number of residential proxies I see scraping the couple of sites I have responsibility for I don't find IP address-based blocking useful anymore. That ship has sailed.
Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).
I run an attractive site to AI scrapers and yes the IP ship has indeed sailed in favor of residential proxies. They are non-stop and could have ingested all of our content many times over every day. JA4 is useless too because they have the ability to spoof this or use a very common fingerprint like Chrome that you cannot block without blowback. These bots will only use a given IP a couple of dozen times and then use another node. They use 10s of thousands of IPs. You can tell because they will come in waves from a residential or mobile network
What kind of fingerprinting are you thinking of? JA4? I haven't found a way to do that inexpensively at our scale, but we may have to go that route - looking at CloudFront bot mitigation.
For behavioral, we have Anubis honeypot functionality turned on, but it doesn't seem to be effective for 99% of scrapers. Anubis is also running behind TLS termination, so I don't think it can do full JA4. It does have the less robust JA4H apparently, but I'm not sure how effective that will be.
Edit: Oh yeah, forgot to mention - it's almost 100% residential proxies. Primarily China Telecom and China Unicom. Unfortunately those providers are HUGE and also host a ton of legitimate users all over Asia.
> What kind of fingerprinting are you thinking of?
I'm minding a proprietary application where this is being discussed. The vendor is talking about running Javascript in the client to detect headless browsers, bots, etc. I'm guessing they're going to license something off-the-shelf. The technical specifics aren't being discussed. They're at the "wow the stakeholders" phase.
I'd like to do something ourselves but, at the size of the org I'm talking about (with virtually no in-house dev resources), COTS rules the day.
Interesting, did you implement only IP reputation (via blocklist) or did you deploy the WAF as well? Regarding bot scrapping, you would probably want to try the new bot detection feature recently released
This was just blocklist based. We had the main community list and a handful of the curated paid lists enabled.
wrt bot detection - this sounds very much like Anubis which we're also using with some success.
I have written my own honeypots to reduce the false positive rate. I simply have things like a VM with RDP and SSH open to the internet and any IP that tries to login gets banned at the firewall for x days. It works really well.
Sounds like an exploit took the credentials needed to extract the code, makes me wonder if a Ubikey + SSL cert for git access would have prevented the entire leak.
Time to scrutinize those CrowdSec deployments even closer. Hopefully, it's nothing critical that gets exploited.
> CrowdSec source code consists of two parts: a private one and another that hosts our Free Open Source Software (i.e., the Security Engine), which is public by design and therefore out of scope. The private part, though, contains the source code for our SaaS console, some AWS Cloud routines, some connectors, and automations.
This leak seems incredibly mundane. customers would have the security engine installed, but not the console.
Oof, a security company leaking its own source code is quite the look. Hope it's not their latest version of the engine.
From TFA:
Security firm hacked.
Security firm putting their source code on Github
A tale as old as time.
Not to be confused with CrowdStrike, a serious [0] security company
[0] https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_ou...
of course... Ask Fable if the private code is valuable or not...
yet another security oriented company that doesn’t practice what they preach.
On the flip side, there was allegedly no PII leaked. But this event is still a red flag as it means their internal ops are absolutely shit. So it’s another vendor receiving a PNG flag.
CrowdSec. CrowdStrike.
I was going to register http://CrowdShite.com/ but somebody beat me to it.