As is traditional with most explanations of HTTP caching, it doesn't mention Vary header. Although apparently some CDNs (e.g. Cloudflare) straight up ignore it for some reason [0].
I would say "vary" is the wrong way to solve that problem. The issue is that there can easily be a bunch of stupid inconsequential differences between accept headers, far beyond simply asking for type x versus type y. Slightly different priorities, order, including an extra mime in the list, putting some irrelevant format nobody uses first just in case, etc.
An optimal solution would involve: the response listing which alternate content-types can be returned for that endpoint, the cache considering the accept header, if it sees a type from the alternates list higher in the accept header priority than whatever it has in cache, then it would forward the request to the server. Once it had all the alternatives in cache, it would pass them through according to the accept without hitting the server.
The closest existing header to the above would be the link header, if you give it rel=alternate, and type as the mime type. It's not clear what href you would be, since it usually is to a different document, but we want the same url but a different mime type. So clearly this would be an abuse of the header, but could work.
Sure, except I doubt most people want to uglify all their urls with extensions for occasional alternates. Plus, if the url with the extension gets past around instead of the original (as would inevitably be done) you're back to square one.
I had thought about recommending that people just use an alternate link as intended, to point to an alternate format. I think that would work best using existing web standards as intended, but it has the downside of initially serving the original format regardless of the content type.
Good call! Honestly I just wanted to wrap it up before the holidays, but you’re right that a small section on Vary would have been useful.
Things like non-conforming caching services made me punt actual suggestions to a later article, as I wasn’t sure how my sense of the RFC interacted with the real world. HTTP Caching Tests seems like a great resource for this, but only includes Fastly out of the big providers, and it seems to be doing okay with Vary. https://cache-tests.fyi/
> the cache MUST NOT use that stored response without revalidation unless all the presented request header fields nominated by that Vary field value match those fields in the original request
You’ll find that some have creative readings of MUST NOT.
Wanted to highlight MDN's HTTP caching guide[0] that OP links in the conclusion. It's written at a higher level than the underlying reference material and has been a great resource I've turned to several times in the last few years.
I think setting FileETag None solved it. With that setup, the browser won't use stale JS/CSS/whatever bundles, instead always validating them against the server, but when the browser already has the correct asset downloaded earlier, it will get a 304 and avoid downloading a lot of stuff. Pretty simple and works well for low traffic setups.
It was surprisingly easy to mess up, or having your translation bundles have cached out of date versions in the browser.
(nothing against other web servers, Apache2 was just a good fit for other reasons)
As many have pointed out here, the nature of caching has changed in the current climate of ubiquitous HTTPS, and I want to add a paragraph or two about it. Is there a good summary somewhere that I could reference? What are the the usual, most prevalent uses of HTTP intermediaries involving caches, besides CDNs and origin-controlled caches (eg Varnish)?
HN is full of noobs loudly proclaiming what they don't know is true these days. Ubiquitous HTTPS does not change the nature of private browser caches, and only nullify the proxy related cache headers if the origin encrypts traffic all the way to the client, which is quite rare in real life, unless we are merely talking about a dude serving this blog from his basement computer.
In general, your answer depends on where the TLS cert terminates. In most situation a CDN or a reverse proxy is involved, and the TLC cert you use to encrypt traffic from the origin to the proxy is different from the one the proxy uses to encrypt traffic from it to the browser. Whenever a MITM intermediary is involved, you should read the intermediary's documentation. These usually include Cloudflare, AWS Cloudfront, Akamai etc. With with exceptions, like the Vary header as pointed out elsewhere, these vendors largely follow HTTP caching semantics for proxy caches.
It clearly notes that it's "a refresher", does not claim that it's novel research, and extensively links to the reference documents. It is, essentially, a review article (https://en.wikipedia.org/wiki/Review_article). And there's absolutely nothing wrong with that.
Hell, the author could probably have called it a primer and I think it'd have been fair.
Dunno man, sometimes I write blog posts for my own benefit, to document my knowledge and understanding of something. I could put it in a private note, but I can also put it in my blog and who knows, maybe someone else can benefit from it - even if it’s nothing you couldn’t google research yourself or god forbid, ask an LLM to summarize for you.
No need to be mean and assume the worst possible purpose :)
I’m sorry you didn’t get anything out of it. I wasn’t operating at the edge of caching knowledge, just a person refreshing and clarifying for themselves how caching works. Some things were new to me, and after spending so much time with the RFC, I just thought others may benefit or, more selfishly, would point out errors or ways to make it better.
I mean, do those <meta> tags really suggest someone who’s into SEO? Call me stale but what I really want is validation :-)
It is not uncommon for enterprises to intercept HTTPS for inspection and logging. They may or may not also do caching of responses at the point where HTTPS is intercepted.
I previously experimented a bit with Squid Cache on my home network for web archival purposes, and set it up to intercept HTTPS. I then added the TLS certificate to the trust store on my client, and was able to intercept and cache HTTPS responses.
In the end, Squid Cache was a little bit inflexible in terms of making sure that the browsed data would be stored forever as was my goal.
This Christmas I have been playing with using mitmproxy instead. I previously used mitmproxy for some debugging, and found out now that I might be able to use it for archival by adding a custom extension written in Python.
It’s working well so far. I browse HTTPS pages in Firefox and I persist URLs and timestamps in SQLite and write out request and response headers plus response body to disk.
My main focus at the moment is archiving some video courses that I paid for in the past, so that even the site I bought the courses from ceased operation I will still have those video courses. After I finish archiving the video courses, I will proceed to archiving other digital things I’ve bought like VST plugins, sample packs, 3d assets etc.
And after that I will give another shot at archiving all the random pages on the open web that I’ve bookmarked etc.
For me, archiving things by using an intercepting proxy is the best way. I have various manually organised copies of files from all over the place, both paid stuff and openly accessible things. But having a sort of Internet Archive of my own with all of the associated pages where I bought things and all the JS and CSS and images surrounding things is the dream. And at the moment it seems to be working pretty well with this mitmproxy + custom Python extension setup.
I am also aware of various existing web scrapers and internet archival systems for self hosting and have tried a few of them. But for me the system I am doing is the ideal.
Some of it is different, but the basics are still the same and still relevant. Just today I've been working with some of this.
I took a Django app that's behind an Apache server and added cache-control and vary headers using Django view decorators, and added Header directives to some static files that Apache was serving. This had 2 effects:
* Meant I could add mod_cache to the Apache server and have common pages cached and served directly from Apache instead of going back to Django. Load testing with vegeta ( https://github.com/tsenart/vegeta ) shows the server can now handle multiples more simultaneous traffic than it could before.
* Meant users browsers now cache all the CSS/JS. As users move between HTML pages, there is now often only 1 request the browser makes. Good for snappier page loads with less server load.
But yeah, updating especially the sections on public vs private caches with regards to HTTPS would be good.
CDNs manage user TLS certificates and that is one of the advantages of using them.
A node server could negociate https close to the user, do caching stuff and create an other https connection to your local server (or reuse an existing one).
At one point with http only your isp could do its own cache, large corporate it networks could have a cache, etc. which was very efficient for caching. But horrible for privacy. Now we have CDN edge caching etc but nothing like the multi layer caching that was available with http.
As is traditional with most explanations of HTTP caching, it doesn't mention Vary header. Although apparently some CDNs (e.g. Cloudflare) straight up ignore it for some reason [0].
[0] https://news.ycombinator.com/item?id=38346382
There was a recent discussion on X about this that had a couple of Cloudflare people chip in, including their CTO:
https://xcancel.com/simonw/status/1988984600346128664
The highlight from that thread https://xcancel.com/dok2001/status/1989005141450846470#m
I would say "vary" is the wrong way to solve that problem. The issue is that there can easily be a bunch of stupid inconsequential differences between accept headers, far beyond simply asking for type x versus type y. Slightly different priorities, order, including an extra mime in the list, putting some irrelevant format nobody uses first just in case, etc.
An optimal solution would involve: the response listing which alternate content-types can be returned for that endpoint, the cache considering the accept header, if it sees a type from the alternates list higher in the accept header priority than whatever it has in cache, then it would forward the request to the server. Once it had all the alternatives in cache, it would pass them through according to the accept without hitting the server.
The closest existing header to the above would be the link header, if you give it rel=alternate, and type as the mime type. It's not clear what href you would be, since it usually is to a different document, but we want the same url but a different mime type. So clearly this would be an abuse of the header, but could work.
That's tangentially related to the Vary header. Not only Accept can go into its value, you know.
And an optimal solution IMHO would be for the origin server to simply return 302 to a specific resource, selected upon the value of the Accept header:
Sure, except I doubt most people want to uglify all their urls with extensions for occasional alternates. Plus, if the url with the extension gets past around instead of the original (as would inevitably be done) you're back to square one.
I had thought about recommending that people just use an alternate link as intended, to point to an alternate format. I think that would work best using existing web standards as intended, but it has the downside of initially serving the original format regardless of the content type.
> if the url with the extension gets past around instead of the original (as would inevitably be done) you're back to square one.
Why? It has no "Vary" header, and it's the one that's supposed to get cached anyhow.
If people see it in the url bar and copy paste it from there. In the case of images if they "copy image url".
Good call! Honestly I just wanted to wrap it up before the holidays, but you’re right that a small section on Vary would have been useful.
Things like non-conforming caching services made me punt actual suggestions to a later article, as I wasn’t sure how my sense of the RFC interacted with the real world. HTTP Caching Tests seems like a great resource for this, but only includes Fastly out of the big providers, and it seems to be doing okay with Vary. https://cache-tests.fyi/
Vary is Very important.
> the cache MUST NOT use that stored response without revalidation unless all the presented request header fields nominated by that Vary field value match those fields in the original request
You’ll find that some have creative readings of MUST NOT.
Great write up!
Wanted to highlight MDN's HTTP caching guide[0] that OP links in the conclusion. It's written at a higher level than the underlying reference material and has been a great resource I've turned to several times in the last few years.
[0]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac...
I found that Cache-Control with no-cache worked pretty well EXCEPT Apache2 would fail to return 304 when also compressing some of the resources: https://stackoverflow.com/questions/896974/apache-is-not-sen...
I think setting FileETag None solved it. With that setup, the browser won't use stale JS/CSS/whatever bundles, instead always validating them against the server, but when the browser already has the correct asset downloaded earlier, it will get a 304 and avoid downloading a lot of stuff. Pretty simple and works well for low traffic setups.
It was surprisingly easy to mess up, or having your translation bundles have cached out of date versions in the browser.
(nothing against other web servers, Apache2 was just a good fit for other reasons)
As many have pointed out here, the nature of caching has changed in the current climate of ubiquitous HTTPS, and I want to add a paragraph or two about it. Is there a good summary somewhere that I could reference? What are the the usual, most prevalent uses of HTTP intermediaries involving caches, besides CDNs and origin-controlled caches (eg Varnish)?
HN is full of noobs loudly proclaiming what they don't know is true these days. Ubiquitous HTTPS does not change the nature of private browser caches, and only nullify the proxy related cache headers if the origin encrypts traffic all the way to the client, which is quite rare in real life, unless we are merely talking about a dude serving this blog from his basement computer.
In general, your answer depends on where the TLS cert terminates. In most situation a CDN or a reverse proxy is involved, and the TLC cert you use to encrypt traffic from the origin to the proxy is different from the one the proxy uses to encrypt traffic from it to the browser. Whenever a MITM intermediary is involved, you should read the intermediary's documentation. These usually include Cloudflare, AWS Cloudfront, Akamai etc. With with exceptions, like the Vary header as pointed out elsewhere, these vendors largely follow HTTP caching semantics for proxy caches.
For 10+ years in the industry I can safely say that almost nobody knows or cares about HTTP caching. It’s sad.
This is nothing new and doesn't add anything new to the topic, so am I the only that thinks this is just an attempt at boosting their SEO through HN?
It clearly notes that it's "a refresher", does not claim that it's novel research, and extensively links to the reference documents. It is, essentially, a review article (https://en.wikipedia.org/wiki/Review_article). And there's absolutely nothing wrong with that.
Hell, the author could probably have called it a primer and I think it'd have been fair.
Dunno man, sometimes I write blog posts for my own benefit, to document my knowledge and understanding of something. I could put it in a private note, but I can also put it in my blog and who knows, maybe someone else can benefit from it - even if it’s nothing you couldn’t google research yourself or god forbid, ask an LLM to summarize for you.
No need to be mean and assume the worst possible purpose :)
I’m sorry you didn’t get anything out of it. I wasn’t operating at the edge of caching knowledge, just a person refreshing and clarifying for themselves how caching works. Some things were new to me, and after spending so much time with the RFC, I just thought others may benefit or, more selfishly, would point out errors or ways to make it better.
I mean, do those <meta> tags really suggest someone who’s into SEO? Call me stale but what I really want is validation :-)
A lot of this seems irrelevant these days with https everywhere.
It is not uncommon for enterprises to intercept HTTPS for inspection and logging. They may or may not also do caching of responses at the point where HTTPS is intercepted.
I previously experimented a bit with Squid Cache on my home network for web archival purposes, and set it up to intercept HTTPS. I then added the TLS certificate to the trust store on my client, and was able to intercept and cache HTTPS responses.
In the end, Squid Cache was a little bit inflexible in terms of making sure that the browsed data would be stored forever as was my goal.
This Christmas I have been playing with using mitmproxy instead. I previously used mitmproxy for some debugging, and found out now that I might be able to use it for archival by adding a custom extension written in Python.
It’s working well so far. I browse HTTPS pages in Firefox and I persist URLs and timestamps in SQLite and write out request and response headers plus response body to disk.
My main focus at the moment is archiving some video courses that I paid for in the past, so that even the site I bought the courses from ceased operation I will still have those video courses. After I finish archiving the video courses, I will proceed to archiving other digital things I’ve bought like VST plugins, sample packs, 3d assets etc.
And after that I will give another shot at archiving all the random pages on the open web that I’ve bookmarked etc.
For me, archiving things by using an intercepting proxy is the best way. I have various manually organised copies of files from all over the place, both paid stuff and openly accessible things. But having a sort of Internet Archive of my own with all of the associated pages where I bought things and all the JS and CSS and images surrounding things is the dream. And at the moment it seems to be working pretty well with this mitmproxy + custom Python extension setup.
I am also aware of various existing web scrapers and internet archival systems for self hosting and have tried a few of them. But for me the system I am doing is the ideal.
Some of it is different, but the basics are still the same and still relevant. Just today I've been working with some of this.
I took a Django app that's behind an Apache server and added cache-control and vary headers using Django view decorators, and added Header directives to some static files that Apache was serving. This had 2 effects:
* Meant I could add mod_cache to the Apache server and have common pages cached and served directly from Apache instead of going back to Django. Load testing with vegeta ( https://github.com/tsenart/vegeta ) shows the server can now handle multiples more simultaneous traffic than it could before.
* Meant users browsers now cache all the CSS/JS. As users move between HTML pages, there is now often only 1 request the browser makes. Good for snappier page loads with less server load.
But yeah, updating especially the sections on public vs private caches with regards to HTTPS would be good.
These are still used in CDN and internal browser caching
CDNs manage user TLS certificates and that is one of the advantages of using them.
A node server could negociate https close to the user, do caching stuff and create an other https connection to your local server (or reuse an existing one).
Https everywhere with your CDN in middle.
If you implement any of the ends of a HTTP communication caching is still very important.
This website is chock full of site operators raging mad at web crawlers created by people that didn't bother to implement proper caching mechanisms.
how is https making caching irrelevant?
At one point with http only your isp could do its own cache, large corporate it networks could have a cache, etc. which was very efficient for caching. But horrible for privacy. Now we have CDN edge caching etc but nothing like the multi layer caching that was available with http.
That sounds like it is one expiration bug away from debugging hell
Besides MITM proxies, server-side proxies can also do caching. Thus applications should use the Vary: header.
Just the opposite, caching is everywhere now. How do you think a CDN works?
Can you elaborate on what is the reasoning here?