First, find out who's on the other end of a few hundred IP addresses. Start with ones in the US. Sue for damages. Use discovery to find out what's on the other end. Sue the maker of that device. If it turns out to be an appliance or smart TV, it may be possible to consolidate cases into one case against the manufacturer. Criminal negligence, tort interference with contract, harassment, Computer Fraud and Abuse act violation...
Maybe a restraining order prohibiting the sale of "smart TV" known to be able to host attacks. Have imports seized by Customs and Border Protection. That would get a manufacturer's attention.
The manufacturer's EULA will not help the manufacturer, because the plaintiff, the party being attacked, is not a party to the EULA at all.
I agree they they should, but that would be hard before, now in the IoT-hell where even your lightbulbs and internet-facing and capable of being proxies seems like a herculean effort.
Pretty sure I saw an article on HN a few days ago about, in part, how a bunch on seemingly innocuous apps for smart tvs, stuff like screen savers and the like, all ran proxy servers (in the users residential address) under the hood. I think it was in the GamerNexus investigation on the whole LG Tv spying on people IIRC.
A random Internet-connected smart plug made by KOCKJAKD and sold on Aliexpress? Almost impossible to catch.
A major brand like LG, with US presence and sold in brick-and-mortar stores? Much more possible. It also makes a much juicier target for lawyers, thus making it much more likely to get sued. And once there is a precedent, other manufactures would be in danger too. Hopefully after losing a whole bunch of money, the manufacturers will start cracking down on those residential proxies.
(Except that discovery is going to be painful. "Use discovery to find out what's on the other end" is easy to write, but in practice it means regular people dealing with bailiffs just because they happen to buy wrong brand of smart TV)
> I agree they they should, but that would be hard before, ... seems like a herculean effort.
| We choose to go to the Moon in this decade and do the other things, not because they are easy, but because they are hard; because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one we intend to win, and the others, too.
So what, they are hard. So are so many of humanity's greatest achievements.
Honestly, these companies win when we buy into the belief that these things are too hard. They're lazy and are just like any of us that make excuses to not do chores or other things that we should. But the reality is that for pursing our civilization forward we should pursue the toughest problems. It is just about will. It'll be tiring. Some will kick and scream, throwing tantrums. But we aren't just any animal, we're humans. We can do literally anything if we decide it's worthwhile.
Importantly, we shouldn't just jump onto the next hype train, we should be passionate, nuanced, and pursue for the sake of pursuit. We don't have to put all our eggs in one basket. We shouldn't. We have enough time, resources, and energy to pursue so much. But as soon as at pretend we live in a finite world we tend to self destruct and fight over constraints we've made up. The universe is limitless, as are our minds.
Fortunately you only need to find a one of the many possible devices to sue. Just winning a couple of these will send a message. The goal isn't the $$$ directly (the lawyers are the only ones who win) it is to send a message that you are forever responsible for your vulnerable devices and so you better make them secure.
The hard part is if any of the devices are from someplace that doesn't have a US (or whatever country they are in) presence. That country my ignore the lawsuit.
It's much less hard than you think, we just think it's hard because we are engineers not lawyers.
Proxies have limited and expensive bandwidth and couldn't launch a DDoS anyway because incoming traffic equals outgoing traffic. They are innocent here. You need a DDoS software module to make a DDoS.
Exactly, the harmed party is not subject to binding arbitration. The harmed party isn't going to bring someone to court for their bad devices - nobody has enough assets to be worth suing (other than perhaps to compel them cooperate with the discovery process) . However the company that made those devices is worth suing.
Alibaba is an interesting question though - if the device is from China (or some such) it isn't clear what the courts can do...
The problem is most of the botnet zombies are in places with very little rule of law like eastern europe/russia/south america/china. It is an exercise in futility and the richest customers just opt to just spend money on more protection than shutting down the zombies.
If the zombie is in the US, hosts like Google or Amazon will take 1+ month to respond.
Author here. This attack was extremely broad. I saw parts of this attack come from my own home ISP's ASN, though not my IP thankfully. If we just "blocked those places" there would be a lot of collateral damage. As it stood, we did temporarily bump up rate limiting for the biggest attack ASNs and we absolutely heard from real, regular users about it (Sorry to those affected).
Either it's coming from certain places and you can block those places, or it's coming from everywhere including some that are within your legal jurisdiction.
Sorry for the stress that the team had mitigating this attack. Yet I'm always excited to see signs of competence on the attacking side: a targeted application-specific and adaptive attack at L7? Wow!
Here's another guess: a DDoS to steal your attention and mask other intrusion attempts.
> Defenses need to have broader rate limits across more than just IPs (ASNs, hostnames, etc.).
The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.
> Attackers actively search for non-cacheable paths (e.g. dynamic redirects, search endpoints, and 404s).
To continue with my previous point. As long as individual server's resources permit (memory, socket limit) stall requests before processing them. Low reputation IPs get stalled for longer and the requests that exceed the queue get dropped. The idea is graceful degradation:
A good rep IP will not be stalled by sleep(). A poor rep IP will be stalled, but eventually receive its answer instead of some 429/403 (i.e. a user who opened many tabs at once). A bad rep IP will be slowed down by the wait times + rate-limits (queue exceeded) before getting completely banned for good. The point is to have more granularity before throwing errors at random users at the server-level.
> The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.
The penalty box strategy I described in the post is along these lines. It penalizes excessive expensive requests directly. Specifically, it does add those to a score and will rate limit more broadly as necessary.
A more interesting question is, what exactly do the attackers gain from hitting read the docs? Most of their docs hosting is static/easily CDN cached. Unlike database bound sites, you would need a lot more traffic to overload pure/mostly static hosting. Maybe it's a malicious AI lab looking to deny their competitors training data? As far as infosec profiling goes, this is probably the oddest case I have heard of.
I am thinking it's probably an AI lab that misconfigured their data scraper (made it too agentic) and it ended up looking like a DDoS.
The new generation of scrapers are all agentic and self healing. (As an example see YC's https://parse.bot)
Author here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscaling. However, as others have suggested, we may have been a test run for a real target.
Do you have a list of the addresses that were hammering your site? Have you tried any of the techniques I list here? [1] Do sets of the IP's show up in here [2]? Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers? Have you tried blackhole routing any of them from an out of band management console?
# only useful if not behind a CDN
for Ip in $(cat /dev/shm/list-of-attackers.ipset);do ip route add blackhole "${Ip}" 2>/dev/null;done
[Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.
> Do you have a list of the addresses that were hammering your site?
I could get a list of IPs, but it wouldn't be particularly helpful. It was hundreds of millions of unique IPs. Most IPs made fewer than 100 requests and then disappeared.
> Have you tried any of the techniques I list here?
We were already doing essentially all of the techniques there and they were not helpful in this attack. This wasn't some spider run amok. This was a dedicated attack with intelligence behind it.
> Do sets of the IP's show up in here [2]?
If we're looking at individual IPs, we've already lost.
> Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers?
The article outlines most of this. It was a mix of residential and major/minor hosting providers and some corporates. If I were to say which was the largest, it was minor hosting providers. However, the attack pivoted between coming from different sources. At first it was coming mostly from minor hosting providers/VPS. By the end, it was entirely residential and corporate IPs. Tor was not involved at all (RTD supports accessing through Tor, but you are more likely to get a challenge). All the expected headers were there and UAs cycled between a very large set of standard UAs for normal browsers/devices. The attack was 95%+ HTTP2/HTTP3.
> [Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.
It's trivial to setup an IP list[1] in CF and then you can apply all sorts of rules (lower rate limits, outright challenges, etc.) to it. It can be managed through Terraform as well. You can also use CIDR notation. This along with more classifications to specific ASNs are something we are looking at.
Inside this .bz2 [1] there is a directory called bh_routes/bh_vps_120/ and in that there are many AS files named for many of the VPS/Server providers. If you site does not need inbound connections from such places that is a starting point to reduce the traffic a bit. Each file has a comment with the AS#/whois name if you want to double check who/what you may be blocking. Not perfect but it may make a dent in the traffic. Or ask Cloudflare if they have an option to block anything that is not residential or LTE wireless. Even if you want to allow bots, maybe have a "shields up" mode where bots are sent away until the attack passes.
Another potential option would be to configure nginx IP limits using the Cloudflare header that represents the IP, set the shared memory size rather high and return 525 to the IP's exceeding a limit to avoid them hitting your redirect rules though it feels like CF should be able to create something custom for this assuming its a paid account.
Any IP that is not VPS/server should be archived for the feds as they are working on shutting down residential proxy providers including apps that are turning peoples cell phones into proxies.
> Most of their docs hosting is static/easily CDN cached
The article says
> and it purposefully attacked areas that bypassed caching
So that doesn't work. Also, it seems that they were trying to cause financial harm, not to take down the infrastructure but to make it costly for the org itself. That's smart.
Nothing to do with AI, but just the general storm of trying to force the Internet into a proprietary, tightly-controlled walled garden with strong identity verification.
> they were overwhelming a hardcoded Nginx redirect (a simple rewrite regex directive)
1. I wonder how much optimization ngx_http_rewrite_module has? Does it precompile the patterns? LLM said yes, this SO answer [1] says that a JIT config option must be on. I consider "Just in Time" to be a half measure when the config itself is static.
2. From looking at NGINX docs, it looks to me there are some pitfalls to writing these rules. Like you must manually make sure to short-circuit the rewrites to exit early?
3. The caveat of regex is that catastrophically backtracking regexes do look simple. I don't see this issue being talked about enough. See links, if you, the reader, haven't heard of it yet.
this might be a AI driven attack and readthedocs was just a test target.
What surprised me is how easy it was to evade the cloudflare defenses. I know it was easy to evade CF, but I would expected CF to do a better job at blocking L7 DDOS.
CF is really good in defending against the L4 DDOS, but not L7.
this means that cloudflare is really not useful much in the era of Agentic DDOS driven by thousands agents across the globe
> One decision we made is to always give real users an escape hatch. Read the Docs very rarely issues outright blocks or bans to specific IPs or user agents. Instead, our "worst" is a JavaScript challenge, and if a user solves a challenge, they are very unlikely to get challenged again for the next day or so.
Finally, a competent response that doesn't leave the users hang out to dry.
I'm so tired of seeing incompetents with measures like "blackhole 2 continents" deployed even outside active attacks.
My assumption would be that it would drastically reduce the attack to be borderline irrelevant. I've never turned on Under Attack so somebody else may have more insight and the docs[1] don't describe precisely what happens besides a JS interstitial.
I know that JS challenges, both interactive and non-interactive, can be solved by bots. I've seen it. However, I suspect that the challenges just get harder and harder until the attack levels drop.
It changes the economics of the attack because it requires the attacker to do compute before they can make requests. Depending on how good the fingerprinting is, it can also get very expensive (eg. requiring you to run a full browser, rather than merely computing a few sha256 hashes)
That has only partially mitigated much smaller attacks (residential proxy scraping etc) on my employer's site.
We're currently on the "Business" plan, but I'm coming to the conclusion that we need to upgrade to the "Enterprise Advantage" plan for the JA3/4 fingerprinting and detection ID features.
* JA3s are mostly useless. JA4s supersede them entirely.
* Using JA4s in rate limits is pretty useful and helps a lot against proxy scraping. It was not very helpful in this attack.
* Bot detections are somewhat helpful but they don't solve scrapers/attacks by themselves. They're useful as a 2nd/3rd data point (eg. low bot score + bot detection + something else)
My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?
I don't quite understand your post, but is your question why don't the ISPs of the sources of the abusive traffic sort it out?
The distributed nature of DDoS means each participating host isn't sending that much traffic, and there are often tens or hundreds of thousands of participating hosts. An ISP should verify claims of abuse before cutting off customers, and since most of the customers are presumably unaware of what their systems are doing, there will be a lot of unhappy customers and then you've got to spend a lot of customer support time on helping them clean up their systems so they can get back online.
I spent a fair amount of time sending out abuse reports for phishing / malware senders about a decade ago, and most abuse reporting addresses are a black hole. Even if you do get to someone who will do something about abuse, they won't do it quickly.
There's be a few high profile longer term DDoS attacks lately, but when I was running infra that got a lot of stuff, it was mostly people kicking the tires on DDoS as a service offerings and most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Starting from there, this DDoS sounds like layer 7 DDoS which is easy to track to the immediate senders, but a ton of DDoS is volumetric stuff, often volumetric reflection attacks where the senders spoof your address. If you're getting that, best you can do is get the reflectors kicked off (or cleaned up) ... tracing back to the sending hosts means getting a reflector (and their ISPs) engaged to do a lot of labor intensive work.
All of that investigation stuff takes qualified people lots of time, that's your money reason it doesn't happen.
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument?
As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such a reoccurring source of abuse needs to either clean up or find themselves a different ISP to spread harm onto the net
I get what you're saying about that this won't solve an ongoing attack right this minute, or even by next week. But if we just let it all happen then the solution is going to be either (1) we all buy equipment that can handle something like a terabit per second and arm our infrastructure to the teeth or (2) centralize all traffic through a vetting entity who decides which client gets to visit the internet today. So far we're headed towards the latter and nobody really wants that. Abuse messages will have to slowly trickle down from victims to originating ISPs to users, and if users didn't willingly sign up, then to wherever users are getting this malware (Google's app store will be a big component). Stopping this at the source seems to me a much more desirable long-term solution
> As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious.
It sure is --- but an ISP would want to observe the traffic themselves, and if it's a 90 second attack every so often, chances are they won't see it when they look. When it's volumetric reflection, you can probably tell them how to send a request and see the response, and maybe they'll contact the customer, but maybe they'll just sit on it. As a victim, the ROI for reporting just wasn't there.
I wasn't getting huge traffic flows, and I was mostly getting attacks against www, which wasn't my actual service, so making sure volumetric attacks below my interface rate were shrugged off and taking simple actions like dropping requests from http clients with user-agent Wordpress were good enough. If the volumetric attacks were much over 10G, my host would have null routed my servers, which is annoying but highly scalable --- many ISPs support a BGP blackhole community, so my host can add my attacked IP to that and their upstreams will drop inbound packets when they enter the ISPs network.
I can't find a reference now, but I've seen things that allowed for more specific blackholing, such as by source or destination port number or by protocol. If my host's ISPs are dropping all UDP and IP fragments to my IP under attack, I could keep serving my TCP traffic and ignore a huge DDoS. I wouldn't even be able to measure the size of the DDoS.
> an ISP would want to observe the traffic themselves
My ISP didn't, when they got access logs from a service I attacked as a teenager and asked me to explain that to get the connection unblocked
Idk, at the moment we're simply not even trying to set a standard. Maybe it would work reasonably well when the ISP needs to observe the traffic and, after a few days of the initial report, they observe a netflow that matches a new abuse report. Even if we set low standards, currently, too few people are sending abuse notifications instead of just sticking it behind the great internet vetting service and calling it good
> maybe they'll contact the customer, but maybe they'll just sit on it.
That's the core point no? If they don't care about their abusive traffic, nullroute their ranges. If admins consistently do that, the abuse has to stop or the ISP goes out of business
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs.
It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it".
What do you expect the ISPs to do in this story? We are talking about TSL connections. Block port 80 and 443 and expect the costumers to use your HTTPS-Proxy. Than they could inspect and block individual actions.
> If your hosting provider is on the list you are collateral damage.
Yes, and this sucks. I moved ISPs because the original one had burned IP addresses that you can't send email from. Every ISP that gets the ranges burned like that will eventually either goes out of business or gets their act together. Not by tomorrow, but eventually
Like, the only other alternative outcome I see is that everyone has to pass through a central surveillance point that decides who's benign and who's naughty, and since nobody wants that... what else are we to do but report abuse?
> What do you expect the ISPs to do in this story? We are talking about TLS connections.
What they've done to me when I abused a service as a teenager, the ISP got an abuse notification: cut off the connection, ask the subscriber wtf this traffic is and how they're going to make sure it doesn't happen again (at which point my dad, the subscriber, came to me and asked if I knew something about this.... yeah ^^')
TLS doesn't matter because the ones receiving the abusive traffic can say what it was. Their access logs will contain the decrypted information and that should be put in an abuse report. If the customer denies everything, turn on netflow logging for a month and see if a future abuse report comes in that can be correlated against the netflow logs. Or have netflow logs stored for 1h by default and retain the entries for which an abuse notification came in (grep for IP addresses in incoming notifications). Lots of reasonable options there; this isn't the difficulty. It's getting people to send that abuse notification so the ISP can identify the problematic subscribers
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did.
The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.
I definitely fall into that, on linux it's just a lot of extensions, scripts etc. Thing is, it's also brittle and not really useful for anybody to talk about it? Not sure I get what you mean with the third sentence.
Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?
For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0].
Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are.
It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate.
It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo.
I run a similar service, and we get almost daily attacks like this. Sometimes it's a specific high-profile customer, other times it's broader.
I can't speak for RTD, but I think it's less "documentation site" and more just that we sit on the domains of high-profile products and the tools are just looking for any hole they can find?
Often it's even the company themselves, for whatever reason (security research, etc).
Interesting that the Under Attack Mode wasn’t used at all here. I understand not wanting to break APIs but I feel temporarily challenging non-API usage could have at least helped without impacting users too much?
I talked about that directly in the post. We didn't want to just challenge everyone. We use JS challenges but we try to use them sparingly. The rest of the ops team and I were fighting to stay up but it never got so bad that it was a choice between complete outage and using the Under Attack mode.
> Attackers found and deliberately targeted URLs that resulted in cache misses, such as non-existent pages with unique paths (404s) as well as temporary redirects (302s)
I'd like to see more of a legal response.
First, find out who's on the other end of a few hundred IP addresses. Start with ones in the US. Sue for damages. Use discovery to find out what's on the other end. Sue the maker of that device. If it turns out to be an appliance or smart TV, it may be possible to consolidate cases into one case against the manufacturer. Criminal negligence, tort interference with contract, harassment, Computer Fraud and Abuse act violation... Maybe a restraining order prohibiting the sale of "smart TV" known to be able to host attacks. Have imports seized by Customs and Border Protection. That would get a manufacturer's attention.
The manufacturer's EULA will not help the manufacturer, because the plaintiff, the party being attacked, is not a party to the EULA at all.
I agree they they should, but that would be hard before, now in the IoT-hell where even your lightbulbs and internet-facing and capable of being proxies seems like a herculean effort.
Pretty sure I saw an article on HN a few days ago about, in part, how a bunch on seemingly innocuous apps for smart tvs, stuff like screen savers and the like, all ran proxy servers (in the users residential address) under the hood. I think it was in the GamerNexus investigation on the whole LG Tv spying on people IIRC.
That's why GP mentioned "Smart TV".
A random Internet-connected smart plug made by KOCKJAKD and sold on Aliexpress? Almost impossible to catch.
A major brand like LG, with US presence and sold in brick-and-mortar stores? Much more possible. It also makes a much juicier target for lawyers, thus making it much more likely to get sued. And once there is a precedent, other manufactures would be in danger too. Hopefully after losing a whole bunch of money, the manufacturers will start cracking down on those residential proxies.
(Except that discovery is going to be painful. "Use discovery to find out what's on the other end" is easy to write, but in practice it means regular people dealing with bailiffs just because they happen to buy wrong brand of smart TV)
Honestly, these companies win when we buy into the belief that these things are too hard. They're lazy and are just like any of us that make excuses to not do chores or other things that we should. But the reality is that for pursing our civilization forward we should pursue the toughest problems. It is just about will. It'll be tiring. Some will kick and scream, throwing tantrums. But we aren't just any animal, we're humans. We can do literally anything if we decide it's worthwhile.
Importantly, we shouldn't just jump onto the next hype train, we should be passionate, nuanced, and pursue for the sake of pursuit. We don't have to put all our eggs in one basket. We shouldn't. We have enough time, resources, and energy to pursue so much. But as soon as at pretend we live in a finite world we tend to self destruct and fight over constraints we've made up. The universe is limitless, as are our minds.
Fortunately you only need to find a one of the many possible devices to sue. Just winning a couple of these will send a message. The goal isn't the $$$ directly (the lawyers are the only ones who win) it is to send a message that you are forever responsible for your vulnerable devices and so you better make them secure.
The hard part is if any of the devices are from someplace that doesn't have a US (or whatever country they are in) presence. That country my ignore the lawsuit.
If you found, say, four LG TVs running as a DDOS node, you'd have a pretty good case.
It's much less hard than you think, we just think it's hard because we are engineers not lawyers.
Proxies have limited and expensive bandwidth and couldn't launch a DDoS anyway because incoming traffic equals outgoing traffic. They are innocent here. You need a DDoS software module to make a DDoS.
AFAIK most people's contract with their ISP includes fine print that forbids a lot of the nasty things these IoT and "smart" devices do.
Doesn't matter from the point of view of a lawsuit from an outside attacked party.
IANAL, but a harmed party outside of the 'binding arbitration' nonsense might be a way work around arbitration and drag the company into a court.
Exactly, the harmed party is not subject to binding arbitration. The harmed party isn't going to bring someone to court for their bad devices - nobody has enough assets to be worth suing (other than perhaps to compel them cooperate with the discovery process) . However the company that made those devices is worth suing.
Alibaba is an interesting question though - if the device is from China (or some such) it isn't clear what the courts can do...
What do they do that you think is prohibited by a residential ISP contract?
Running a proxy server and running a botnet, at least.
The problem is most of the botnet zombies are in places with very little rule of law like eastern europe/russia/south america/china. It is an exercise in futility and the richest customers just opt to just spend money on more protection than shutting down the zombies.
If the zombie is in the US, hosts like Google or Amazon will take 1+ month to respond.
Then IP-range block those places until they get their affairs in order.
Author here. This attack was extremely broad. I saw parts of this attack come from my own home ISP's ASN, though not my IP thankfully. If we just "blocked those places" there would be a lot of collateral damage. As it stood, we did temporarily bump up rate limiting for the biggest attack ASNs and we absolutely heard from real, regular users about it (Sorry to those affected).
Either it's coming from certain places and you can block those places, or it's coming from everywhere including some that are within your legal jurisdiction.
I'd love to see China and Russia sued for hammering my personal websites for the past 20 years... will it ever happen? Hell no, LOL.
Man, it wasn't until reading this that I realised I could get sexually aroused by the mere prospect of legal / governmental competence.
Sorry for the stress that the team had mitigating this attack. Yet I'm always excited to see signs of competence on the attacking side: a targeted application-specific and adaptive attack at L7? Wow!
Here's another guess: a DDoS to steal your attention and mask other intrusion attempts.
> Defenses need to have broader rate limits across more than just IPs (ASNs, hostnames, etc.).
The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.
> Attackers actively search for non-cacheable paths (e.g. dynamic redirects, search endpoints, and 404s).
To continue with my previous point. As long as individual server's resources permit (memory, socket limit) stall requests before processing them. Low reputation IPs get stalled for longer and the requests that exceed the queue get dropped. The idea is graceful degradation:
A good rep IP will not be stalled by sleep(). A poor rep IP will be stalled, but eventually receive its answer instead of some 429/403 (i.e. a user who opened many tabs at once). A bad rep IP will be slowed down by the wait times + rate-limits (queue exceeded) before getting completely banned for good. The point is to have more granularity before throwing errors at random users at the server-level.
> The description/approach seems static and limited? Why not maintain a leaky bucket that counts each request (tickets/points) with higher cost for expensive requests (404, redirects). As the IP's reputation deteriorates (IPv4/32), it begins to spill over to a broader subnet like IPv4/31 then /30 and so on. Fight adaptivity with adaptivity. // maybe I describe something totally obvious, I'm not involved in the web ddos protection side of things.
The penalty box strategy I described in the post is along these lines. It penalizes excessive expensive requests directly. Specifically, it does add those to a score and will rate limit more broadly as necessary.
A more interesting question is, what exactly do the attackers gain from hitting read the docs? Most of their docs hosting is static/easily CDN cached. Unlike database bound sites, you would need a lot more traffic to overload pure/mostly static hosting. Maybe it's a malicious AI lab looking to deny their competitors training data? As far as infosec profiling goes, this is probably the oddest case I have heard of.
I am thinking it's probably an AI lab that misconfigured their data scraper (made it too agentic) and it ended up looking like a DDoS.
The new generation of scrapers are all agentic and self healing. (As an example see YC's https://parse.bot)
Author here. This was not a misconfigured data scraper. We see those every week[1]. This attack wasn't scraping useful content. It was almost entirely 404s and 302s and pulled virtually zero real docs. It specifically looked for URLs not served by the CDN and when it found a pattern, did millions of variations of it. Whether built by an AI or not, it was designed to cause outages and financial damage from autoscaling. However, as others have suggested, we may have been a test run for a real target.
[1] https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse...
Do you have a list of the addresses that were hammering your site? Have you tried any of the techniques I list here? [1] Do sets of the IP's show up in here [2]? Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers? Have you tried blackhole routing any of them from an out of band management console?
[Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
[2] - https://github.com/firehol/blocklist-ipsets/
> Do you have a list of the addresses that were hammering your site?
I could get a list of IPs, but it wouldn't be particularly helpful. It was hundreds of millions of unique IPs. Most IPs made fewer than 100 requests and then disappeared.
> Have you tried any of the techniques I list here?
We were already doing essentially all of the techniques there and they were not helpful in this attack. This wasn't some spider run amok. This was a dedicated attack with intelligence behind it.
> Do sets of the IP's show up in here [2]?
If we're looking at individual IPs, we've already lost.
> Are the bots mostly residential, VPS, Tor? What is the HTTP protocol breakdown? HTTP/1.1, 2.0, 3.0? Are they missing any expected client headers?
The article outlines most of this. It was a mix of residential and major/minor hosting providers and some corporates. If I were to say which was the largest, it was minor hosting providers. However, the attack pivoted between coming from different sources. At first it was coming mostly from minor hosting providers/VPS. By the end, it was entirely residential and corporate IPs. Tor was not involved at all (RTD supports accessing through Tor, but you are more likely to get a challenge). All the expected headers were there and UAs cycled between a very large set of standard UAs for normal browsers/devices. The attack was 95%+ HTTP2/HTTP3.
> [Edit] appears you are behind Cloudflare so the blackhole would be up to them. One could still return a 429 or 525 to the attackers.
It's trivial to setup an IP list[1] in CF and then you can apply all sorts of rules (lower rate limits, outright challenges, etc.) to it. It can be managed through Terraform as well. You can also use CIDR notation. This along with more classifications to specific ASNs are something we are looking at.
[1] https://developers.cloudflare.com/waf/tools/lists/custom-lis...
Inside this .bz2 [1] there is a directory called bh_routes/bh_vps_120/ and in that there are many AS files named for many of the VPS/Server providers. If you site does not need inbound connections from such places that is a starting point to reduce the traffic a bit. Each file has a comment with the AS#/whois name if you want to double check who/what you may be blocking. Not perfect but it may make a dent in the traffic. Or ask Cloudflare if they have an option to block anything that is not residential or LTE wireless. Even if you want to allow bots, maybe have a "shields up" mode where bots are sent away until the attack passes.
Another potential option would be to configure nginx IP limits using the Cloudflare header that represents the IP, set the shared memory size rather high and return 525 to the IP's exceeding a limit to avoid them hitting your redirect rules though it feels like CF should be able to create something custom for this assuming its a paid account.
Any IP that is not VPS/server should be archived for the feds as they are working on shutting down residential proxy providers including apps that are turning peoples cell phones into proxies.
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
[flagged]
> Most of their docs hosting is static/easily CDN cached
The article says
> and it purposefully attacked areas that bypassed caching
So that doesn't work. Also, it seems that they were trying to cause financial harm, not to take down the infrastructure but to make it costly for the org itself. That's smart.
Nothing to do with AI, but just the general storm of trying to force the Internet into a proprietary, tightly-controlled walled garden with strong identity verification.
Yeah I’d be fascinated to hear the attacker’s perspective. Why go through so much work just to slightly inconvenience a random website?
There’s no financial motive and they’re not a political target. Why bother?
> they were overwhelming a hardcoded Nginx redirect (a simple rewrite regex directive)
1. I wonder how much optimization ngx_http_rewrite_module has? Does it precompile the patterns? LLM said yes, this SO answer [1] says that a JIT config option must be on. I consider "Just in Time" to be a half measure when the config itself is static.
2. From looking at NGINX docs, it looks to me there are some pitfalls to writing these rules. Like you must manually make sure to short-circuit the rewrites to exit early?
3. The caveat of regex is that catastrophically backtracking regexes do look simple. I don't see this issue being talked about enough. See links, if you, the reader, haven't heard of it yet.
[1] https://stackoverflow.com/questions/59284921/how-much-impact...
[3.1] https://joshua.hu/nginx-directives-regex-redos-denial-of-ser...
[3.2] https://en.wikipedia.org/wiki/ReDoS
[3.3] https://infrafolks.com/blog/regex-backtracking-devops/
[3.4] https://www.regular-expressions.info/catastrophic.html
[3.5] https://gixy.io/plugins/regex_redos/
this might be a AI driven attack and readthedocs was just a test target.
What surprised me is how easy it was to evade the cloudflare defenses. I know it was easy to evade CF, but I would expected CF to do a better job at blocking L7 DDOS.
CF is really good in defending against the L4 DDOS, but not L7.
this means that cloudflare is really not useful much in the era of Agentic DDOS driven by thousands agents across the globe
Cloudflare is totally worthless actually. It is trivial to evade any part of it.
> One decision we made is to always give real users an escape hatch. Read the Docs very rarely issues outright blocks or bans to specific IPs or user agents. Instead, our "worst" is a JavaScript challenge, and if a user solves a challenge, they are very unlikely to get challenged again for the next day or so.
Finally, a competent response that doesn't leave the users hang out to dry.
I'm so tired of seeing incompetents with measures like "blackhole 2 continents" deployed even outside active attacks.
There’s an assumption that turning on Cloudflare’s “under attack” mode would mitigate the attack.
Given how adaptive the rest of the attack was, I would be very curious to find out how it would approach that obstacle.
My assumption would be that it would drastically reduce the attack to be borderline irrelevant. I've never turned on Under Attack so somebody else may have more insight and the docs[1] don't describe precisely what happens besides a JS interstitial.
I know that JS challenges, both interactive and non-interactive, can be solved by bots. I've seen it. However, I suspect that the challenges just get harder and harder until the attack levels drop.
[1] https://developers.cloudflare.com/fundamentals/reference/und...
It changes the economics of the attack because it requires the attacker to do compute before they can make requests. Depending on how good the fingerprinting is, it can also get very expensive (eg. requiring you to run a full browser, rather than merely computing a few sha256 hashes)
That has only partially mitigated much smaller attacks (residential proxy scraping etc) on my employer's site.
We're currently on the "Business" plan, but I'm coming to the conclusion that we need to upgrade to the "Enterprise Advantage" plan for the JA3/4 fingerprinting and detection ID features.
I get put off by "Contact Sales" pricing.
Here's my take:
* JA3s are mostly useless. JA4s supersede them entirely.
* Using JA4s in rate limits is pretty useful and helps a lot against proxy scraping. It was not very helpful in this attack.
* Bot detections are somewhat helpful but they don't solve scrapers/attacks by themselves. They're useful as a 2nd/3rd data point (eg. low bot score + bot detection + something else)
isn't JA4 also useless because it is so easy to spoof tls. For eg. cycletls for nodejs etc..
It will probably be useless one day. In practice, it is still useful today though not for this attack.
There's also the JA4+ suite (https://github.com/FoxIO-LLC/ja4), in addition to standard JA4.
I see Cloudflare's in the business of selling "protection"
Cloudflare protects many ddos sites from each other's attacks
My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?
I don't quite understand your post, but is your question why don't the ISPs of the sources of the abusive traffic sort it out?
The distributed nature of DDoS means each participating host isn't sending that much traffic, and there are often tens or hundreds of thousands of participating hosts. An ISP should verify claims of abuse before cutting off customers, and since most of the customers are presumably unaware of what their systems are doing, there will be a lot of unhappy customers and then you've got to spend a lot of customer support time on helping them clean up their systems so they can get back online.
I spent a fair amount of time sending out abuse reports for phishing / malware senders about a decade ago, and most abuse reporting addresses are a black hole. Even if you do get to someone who will do something about abuse, they won't do it quickly.
There's be a few high profile longer term DDoS attacks lately, but when I was running infra that got a lot of stuff, it was mostly people kicking the tires on DDoS as a service offerings and most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Starting from there, this DDoS sounds like layer 7 DDoS which is easy to track to the immediate senders, but a ton of DDoS is volumetric stuff, often volumetric reflection attacks where the senders spoof your address. If you're getting that, best you can do is get the reflectors kicked off (or cleaned up) ... tracing back to the sending hosts means getting a reflector (and their ISPs) engaged to do a lot of labor intensive work.
All of that investigation stuff takes qualified people lots of time, that's your money reason it doesn't happen.
> most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument?
As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such a reoccurring source of abuse needs to either clean up or find themselves a different ISP to spread harm onto the net
I get what you're saying about that this won't solve an ongoing attack right this minute, or even by next week. But if we just let it all happen then the solution is going to be either (1) we all buy equipment that can handle something like a terabit per second and arm our infrastructure to the teeth or (2) centralize all traffic through a vetting entity who decides which client gets to visit the internet today. So far we're headed towards the latter and nobody really wants that. Abuse messages will have to slowly trickle down from victims to originating ISPs to users, and if users didn't willingly sign up, then to wherever users are getting this malware (Google's app store will be a big component). Stopping this at the source seems to me a much more desirable long-term solution
> As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious.
It sure is --- but an ISP would want to observe the traffic themselves, and if it's a 90 second attack every so often, chances are they won't see it when they look. When it's volumetric reflection, you can probably tell them how to send a request and see the response, and maybe they'll contact the customer, but maybe they'll just sit on it. As a victim, the ROI for reporting just wasn't there.
I wasn't getting huge traffic flows, and I was mostly getting attacks against www, which wasn't my actual service, so making sure volumetric attacks below my interface rate were shrugged off and taking simple actions like dropping requests from http clients with user-agent Wordpress were good enough. If the volumetric attacks were much over 10G, my host would have null routed my servers, which is annoying but highly scalable --- many ISPs support a BGP blackhole community, so my host can add my attacked IP to that and their upstreams will drop inbound packets when they enter the ISPs network.
I can't find a reference now, but I've seen things that allowed for more specific blackholing, such as by source or destination port number or by protocol. If my host's ISPs are dropping all UDP and IP fragments to my IP under attack, I could keep serving my TCP traffic and ignore a huge DDoS. I wouldn't even be able to measure the size of the DDoS.
> an ISP would want to observe the traffic themselves
My ISP didn't, when they got access logs from a service I attacked as a teenager and asked me to explain that to get the connection unblocked
Idk, at the moment we're simply not even trying to set a standard. Maybe it would work reasonably well when the ISP needs to observe the traffic and, after a few days of the initial report, they observe a netflow that matches a new abuse report. Even if we set low standards, currently, too few people are sending abuse notifications instead of just sticking it behind the great internet vetting service and calling it good
> maybe they'll contact the customer, but maybe they'll just sit on it.
That's the core point no? If they don't care about their abusive traffic, nullroute their ranges. If admins consistently do that, the abuse has to stop or the ISP goes out of business
www.uceprotect.net does that for email. It's a DNS blacklist that puts hole networks on it, even if "only" a individual hosts SPAMs.
It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it".
What do you expect the ISPs to do in this story? We are talking about TSL connections. Block port 80 and 443 and expect the costumers to use your HTTPS-Proxy. Than they could inspect and block individual actions.
> If your hosting provider is on the list you are collateral damage.
Yes, and this sucks. I moved ISPs because the original one had burned IP addresses that you can't send email from. Every ISP that gets the ranges burned like that will eventually either goes out of business or gets their act together. Not by tomorrow, but eventually
Like, the only other alternative outcome I see is that everyone has to pass through a central surveillance point that decides who's benign and who's naughty, and since nobody wants that... what else are we to do but report abuse?
> What do you expect the ISPs to do in this story? We are talking about TLS connections.
What they've done to me when I abused a service as a teenager, the ISP got an abuse notification: cut off the connection, ask the subscriber wtf this traffic is and how they're going to make sure it doesn't happen again (at which point my dad, the subscriber, came to me and asked if I knew something about this.... yeah ^^')
TLS doesn't matter because the ones receiving the abusive traffic can say what it was. Their access logs will contain the decrypted information and that should be put in an abuse report. If the customer denies everything, turn on netflow logging for a month and see if a future abuse report comes in that can be correlated against the netflow logs. Or have netflow logs stored for 1h by default and retain the entries for which an abuse notification came in (grep for IP addresses in incoming notifications). Lots of reasonable options there; this isn't the difficulty. It's getting people to send that abuse notification so the ISP can identify the problematic subscribers
[flagged]
> Every victim (such as readthedocs), or even people sharing blocklists
The report makes it pretty clear that the attack was distributed enough that profiling for blocklists was ineffective.
Whose IPs were they using then? All clean IP addresses from ISPs who are responsive to abuse reports? The post says no such thing
https://www.scispace.com/pdf/drawbridge-software-defined-ddo...
I saw this quite a few years ago. It is possibly relevant. I don't know what, if anything, came of it since then.
I'm curious if anybody could speculate who would be attacking a documentation silo, and to what end?
I'm the author of the blog. I don't know. Internally, we were half joking that we were going to get ransom notice, but we never did.
The only thing that sort of correlates with this attack is that before it started, we began rolling out some slightly more aggressive rate limits one by one. This was mostly because anytime any new "company" thinks they're going to catchup with Claude/OpenAI, they scrape us very aggressively (and they're not respectful about it). My guess is that the attackers behind this attack were already probing us (they were) and they thought the window of opportunity might be closing.
Good to know. I use your site (with a manual transmission user-agent) often, and it's fantastic. Thanks for your work and the writeup!
I've never seen the phrase "manual transmission user-agent". Using your own browser yourself is the new stick shift. Love it.
It feels like everyone's rebuilding their own desktop experience. Kind of Minecraft with folders and text files.
The really interesting part of this is how little people talk about what they're doing, and it doesn't feel secretive in any way.
I definitely fall into that, on linux it's just a lot of extensions, scripts etc. Thing is, it's also brittle and not really useful for anybody to talk about it? Not sure I get what you mean with the third sentence.
Just like you say. It's not a secret business plan, it just doesn't really feel useful to talk about.
What do you mean exactly?
Just curious, if you're tolerant of scraping, do you make an archive of all your content available so that scraping is unnecessary, and if so do the scrapers prefer that?
Current evidence is that scrapers mostly aren't nearly considerate or sophisticated enough to take an "archive of all content" option if one exists.
See https://people.kernel.org/monsieuricon/creepy-crawlies which describes how the https://git.kernel.org gets hammered by crawlers all the time even though you could run a single `git clone` and get the data that way instead.
This is exactly the problem, unfortunately.
For somebody who knows a bit how things are set up, or is willing to spend 10 minutes researching, it's a no-brainer that you can just "git clone" entire linux kernel development history, or download entire wikipedia [0].
Alas, large number of scrapers are not willing to spend those 10 minutes, it would appear. So, here we are.
[0] https://dumps.wikimedia.org/
You can tell Claude to clone from github for Linux stuff all you want... it's still going to try web, and fail, before doing what you asked it to do.
Is there a standard for exposing such sitedata dumps? If not, it's not really surprising that they don't.
It's terabytes of content and other than we're the host not really related to each other. However, for most projects, it's possible to download a zip file of all the HTML docs for that project. We have a lower rate limit to pull these, but a scraper can pull thousands of docs at once. We only host a few hundred thousand projects so pulling a zip of the latest docs for all of them could be done in a day or two at a very reasonable rate.
It's also possible to request the docs already processed into markdown[1]. Lastly, basically all of the docs come from Git. A smart scraper could just clone a project's repo.
[1] https://docs.readthedocs.com/platform/stable/reference/markd...
Could be testing in preparation for attacking something more critical?
Maybe just for the pleasure of doing it, too.
I run a similar service, and we get almost daily attacks like this. Sometimes it's a specific high-profile customer, other times it's broader.
I can't speak for RTD, but I think it's less "documentation site" and more just that we sit on the domains of high-profile products and the tools are just looking for any hole they can find?
Often it's even the company themselves, for whatever reason (security research, etc).
Either testing for something bigger OR demonstrating their power to a 3rd party with minimal real disruption
Could have been a live-fire exercise by a nation state.
Edit: why the down vote? That is literally in the realm of possibility!
Due to mandatory scheduler maintenance, this test has been replaced with a live-fire CORS designed for military Androids.
If you are an iPhone user, please proceed quickly to the chamber lock.
Interesting that the Under Attack Mode wasn’t used at all here. I understand not wanting to break APIs but I feel temporarily challenging non-API usage could have at least helped without impacting users too much?
I talked about that directly in the post. We didn't want to just challenge everyone. We use JS challenges but we try to use them sparingly. The rest of the ops team and I were fighting to stay up but it never got so bad that it was a choice between complete outage and using the Under Attack mode.
> without impacting users too much?
You mean essentially locking out users with non-default Chrome configuration.
Yippee, free load test!
If automatic scaling were free, this would be true...
[flagged]
[flagged]
[dead]
Sure it wasn't a AI crawler?
Why would an AI crawler do this?
> Attackers found and deliberately targeted URLs that resulted in cache misses, such as non-existent pages with unique paths (404s) as well as temporary redirects (302s)
There's no evidence the crawlers are AI-related.