One of the best (and most humorous) posts I've read on the internals of Linux servers + networking.
If you enjoyed this post, you might also like the "Lord of the io_uring" series which walks through how epoll/kqueue work and then onto async services with io_uring:
Great article from the technical aspect, but I also enjoyed the humour a lot. I chuckled with the GB joke, and including the Wolfram screenshot reminded me of the blogs from the ~2005 era, that allowed themselves to include this type of humor.
It was like in the 2000s, blogs wanted to share informatin but also be funny, something similar to a talk in a event? Later they only focused on SEO, so they optimized every word and paragraph, removing jokes, etc. I remember noticing this with the Digital Ocean docs, that ranked 1 for every tutorial in how to install stuff on servers.
Whilst enjoyable reading style I feel like it’s a bit shallow in the concepts. Also the statement: “hft firms saturate 400 gbps and that is x ns” is way to simplified. It really depends on the PPS, the bigger your packets are the lower pps. Also there is rss load balancing to multiple cores and obviously a form of buffering.
> The goal is to run 15 million web servers on one VM with 10GB of RAM and 1 CPU core.
The title is underselling the story. I thought the webserver would've been some extremely beefy server, but no. It's 10GB of RAM. That's a basic M4 mac mini territory (and even that one is much more beefier than OP's server)
Cutting through abstraction and getting to the heart of what you actually want the computer to do is evidently the key to high performance programming.
You can go much lower. The goal was set at "15 million web servers" which is dependent on the definition of "a web server" to include some implementation details. From the client perspective it actually means 15 million IP/port combinations that respond to HTTP - which you can do all in one go with an appropriate wildcard selector (such as with iptables transparent proxy)
An excellent resource optimization article, i.e., given a limited or deeply constrained amount of X, can I get a specific amount of Y, given that the traditional relationship between X and Y is constrained by some set of criteria previously thought to be fixed in nature -- well, let's see if we can re-evaluate those criteria and somehow unconstrain it...
In this case, the constraint is "one VM with 10GB of RAM and 1 CPU core", which would definitely be able to run a single web server fairly well, and probably could run tens of them fairly well, as well.
But, when we get up to hundreds of them simultaneously, thousands of them, tens of thousands of them, well, the associated resource constraint problems just seem to compound at every level...
But let's not let that stop us, let's "go for broke" -- and attempt 15 Million!
It's an excellent thought and engineering exercise...
Anyway, the article is a great read. Worth reading by anyone into optimization of computing resources, and definitely worth re-reading in the future.
micro kernel: a bit like microservices; eth process talks to ip process, ip process talks to tcp process, tcp process talks to web server; multiple IPCs per packet/PDU, rather slow, single crashing process can still take out everything.
exo kernel: web server asks kernel to get packets matching some filter, kernel passes packets matching the filter to web server, web server uses a eth/ip/tcp library to process the packets;
My favourite one-sentence explanation of an exokernel was from the explanation of a web server called Cheetah running on a kernel called XOK: to save time, the web server stores preformatted TCP packets on disk.
This one sentence is such a profound layering violation that it changes your perspective on the possible ways to write programs.
> But hey, embarrassment is a cost of entry - let’s dive into networking in the kernel.
Putting uncomfortable and sad motivational speak into funny speak! You are not logically supposed to feel good when you fail, but your brain's language parts make you feel good chemically! Great idea!
This smells of someone with superficial knowledge vibe-coding. Smoltcp has such low concurrent connection limits by default there's no way it would be used in any meaningful way to serve web content on the internet. Fine for its intended target, embedded devices, but as a general purpose web server - no chance.
One of the best (and most humorous) posts I've read on the internals of Linux servers + networking.
If you enjoyed this post, you might also like the "Lord of the io_uring" series which walks through how epoll/kqueue work and then onto async services with io_uring:
https://unixism.net/loti/async_intro.html#async-intro
Great article from the technical aspect, but I also enjoyed the humour a lot. I chuckled with the GB joke, and including the Wolfram screenshot reminded me of the blogs from the ~2005 era, that allowed themselves to include this type of humor.
It was like in the 2000s, blogs wanted to share informatin but also be funny, something similar to a talk in a event? Later they only focused on SEO, so they optimized every word and paragraph, removing jokes, etc. I remember noticing this with the Digital Ocean docs, that ranked 1 for every tutorial in how to install stuff on servers.
Anyway, good times! I miss them.
Agreed - great article, awesome humour - what a fun read!
Loved his “About” page too:
> I am Artem, living in sunny Finland, writing code at Rivet and running ultramarathons to save money on gas.
That was really nice read, finally an article that is easy to read, simple but super technical, and most importantly not written by AI.
electricity and shit got me! And seeing the emojis in L1.
Fun to see LLM respond with "yes! literally electricity and shit"
Life is much better using AI as a verb, not a noun.
Really interesting and love the humor. Made me chuckle over morning coffee multiple times.
Finding something today that makes you go “i want to subscribe to more of this”, is rare.
Yeah, liked the About Me page, too.
Came back to HN simply to say, what a fantastic article! Interesting, entertaining, it even had me laughing out loud at points. Great job!
It's been months since an article was both interesting and well-written enough to hook me all the way through. What a breath of fresh air!
Whilst enjoyable reading style I feel like it’s a bit shallow in the concepts. Also the statement: “hft firms saturate 400 gbps and that is x ns” is way to simplified. It really depends on the PPS, the bigger your packets are the lower pps. Also there is rss load balancing to multiple cores and obviously a form of buffering.
It’s so easy to forget how much power we have in our systems and how it’s possible to strip abstractions away to get at that power.
Love the humour and simplicity!
> The goal is to run 15 million web servers on one VM with 10GB of RAM and 1 CPU core.
The title is underselling the story. I thought the webserver would've been some extremely beefy server, but no. It's 10GB of RAM. That's a basic M4 mac mini territory (and even that one is much more beefier than OP's server)
Cutting through abstraction and getting to the heart of what you actually want the computer to do is evidently the key to high performance programming.
You can go much lower. The goal was set at "15 million web servers" which is dependent on the definition of "a web server" to include some implementation details. From the client perspective it actually means 15 million IP/port combinations that respond to HTTP - which you can do all in one go with an appropriate wildcard selector (such as with iptables transparent proxy)
I suggest removing the 'web' and just title it 15M servers on one VM.
Seeing 'web', together with its first few sections, made me initially think about http parsing/switching and almost made me stop reading.
It's a breath of fresh air to read an article this well written. I learned, I laughed, I even relearned.
I see a very good use of this for my project umrashrf/postbase to serve millions of users with a small home server.
Nice article, I hadn't come across AF_XDP before.
The next step is to support proper encryption (sine in modern days raw HTTP is out of favor).
> Ten Great Britains divided by 15 million gives around half of kilo britain.
I audibly exhaled from my nose, thank you.
> Not much.
Hilarious.
I don't understand the 3 axis graph with just 2 dimensions. Love the humor and technical depth!
One of the vertical axis is charted using a dotted line and other with a continuous line
An excellent resource optimization article, i.e., given a limited or deeply constrained amount of X, can I get a specific amount of Y, given that the traditional relationship between X and Y is constrained by some set of criteria previously thought to be fixed in nature -- well, let's see if we can re-evaluate those criteria and somehow unconstrain it...
In this case, the constraint is "one VM with 10GB of RAM and 1 CPU core", which would definitely be able to run a single web server fairly well, and probably could run tens of them fairly well, as well.
But, when we get up to hundreds of them simultaneously, thousands of them, tens of thousands of them, well, the associated resource constraint problems just seem to compound at every level...
But let's not let that stop us, let's "go for broke" -- and attempt 15 Million!
It's an excellent thought and engineering exercise...
Anyway, the article is a great read. Worth reading by anyone into optimization of computing resources, and definitely worth re-reading in the future.
Related:
The C10K Problem (1999):
https://en.wikipedia.org/wiki/C10k_problem
https://www.kegel.com/c10k.html
Fun article !
Wonderful article.
I was thinking for the last couple of days to have 10M+ app connections on one laptop as an extreme cost optimization excercise. Great timing.
Just think how cheap functions would be if hyperscalers didn't use VMs for them ...
With the trend of moving everything to userland, maybe Linux finally become a microkernel.
with ebpf, I'd argue it's getting closer to an exokernel (https://sites.cs.ucsb.edu/~rich/class/cs270/papers/dawson-ex...)
micro kernel: a bit like microservices; eth process talks to ip process, ip process talks to tcp process, tcp process talks to web server; multiple IPCs per packet/PDU, rather slow, single crashing process can still take out everything.
exo kernel: web server asks kernel to get packets matching some filter, kernel passes packets matching the filter to web server, web server uses a eth/ip/tcp library to process the packets;
My favourite one-sentence explanation of an exokernel was from the explanation of a web server called Cheetah running on a kernel called XOK: to save time, the web server stores preformatted TCP packets on disk.
This one sentence is such a profound layering violation that it changes your perspective on the possible ways to write programs.
“Linux was clearly written at a time when RAM was big and cheap, sigh“ - excellent!
> But hey, embarrassment is a cost of entry - let’s dive into networking in the kernel.
Putting uncomfortable and sad motivational speak into funny speak! You are not logically supposed to feel good when you fail, but your brain's language parts make you feel good chemically! Great idea!
Loved it, technical, funny, easy to read. It has been a long time I have not seen only like this.
One of the best reads I got from this site!
Hook articles like this directly to my veins
This was not what I was expecting - in a good way. Well written, funny, and devoid of AI slop.
Wait till he hears about the HOST header.
A socket address is 6 or 18 bytes. A host header will often be longer.
What's the relevance of that fact?
They mentioned it. Referred to it as “virtual hosting.”
This smells of someone with superficial knowledge vibe-coding. Smoltcp has such low concurrent connection limits by default there's no way it would be used in any meaningful way to serve web content on the internet. Fine for its intended target, embedded devices, but as a general purpose web server - no chance.