As with what others had said, this is less of a robots.txt and more of a sitemap.
The issue with this is that website owners don't want to do this. Take Reddit removing the API for example. Everyone just switched to scraping the Reddit website instead for any remaining third party clients.
Yes, APIs were supposed to be a compromise to lower the resources needed on both sides, but Reddit's stock price is linked to the value of "their" data, so...
Alternatively, malicious website owners may make incorrect Aura files to mislead user agents. Then we're back to screen scraping as the ground truth, because behaving like a human is the best way to avoid discrimination.
The Reddit example gets right to the heart of it: when a platform loses control over its data and how it makes money, it just shuts the door on open access.
And that s really what aura is trying to address. If we dont figure something out the web could easily end up just being a handful of big ai sites, and all the smaller, independent sites might just fade away.
The goal with aura is to give control back to the people running the websites. It s not about blocking ai but about giving sites a clear, standard way to say "here s how you can work with me meaningfully...".
This means an agent can do something specific and useful without costly, aimless scraping, and it lets site owners build cool, new features just for ais.
and you re right to worry about malicious manifests but that s a trust problem. A site that lies in its aura file would get a bad reputation fast just like a phishing site does now.
At the end of the day aura is a bet that we can build an open, capability based web where site creators can join in on the AI revolution on their own terms. Time will tell if it's the right technical answer, but it s a conversation i think we absolutely need to have to keep the web diverse and creative.
> While websites serve both human readers and LLMs, the latter benefit from more concise, expert-level information gathered in a single, accessible location.
The main difference is
-llms.txt is for reading content
--while aura is for performing actions
-llms.txt tells an AI here is a clean, simple version of this page for you to read
--Aura tells an AI here is a capability called create_post, it needs a title and content, and you can use it by POSTing to the /api/posts endpoint.
llms.txt is for reading, aura is for doing.
You also asked why you need npm install. You dont!
To add Aura to your own website, you only need to create one static file
- public/.well-known/aura.json. That's it.
It s just as simple as creating a llms.txt file.
The npm install command in my project is for people who want to download and run my full reference implementation (the example server and client).
It is not a requirement for the protocol itself.
And for your last question, Aura is new so no other websites are using it yet.
Honestly I love the llmstxt idea if only because it implies that I can get an unbloated version of the web for my own use. Reader mode gets way easier if a .md is available everywhere.
Which of course means this is never going to fly with any site that needs to show you ads.
That s a reallyy sharp observation and you re right it s the core challenge for the web as we know it. For any site that depends on ads hiding the UI is a complete nonstarter.
The key is that aura isnt really about creating another reader mode. Its goal is to fundamentally separate a website s functions from its visual presentation paving the way for a truly intent based web.
Think of it this way
-instead of an ai trying to find and click the "post comment" button it can just use the "post_comment" capability the site offers through a clean api.
While this seems to sidestep the ad model it actually enables a more direct one. A site could specify that a particular action, say a premium translation feature, requires a small payment or an api key.
It s a way to get paid for the actual value you provide, not just for ad views.
This could even change how search works like a search engine that indexes what sites can do, not just what they say. Your personal ai could then find and execute the "book a flight" capability from multiple airlines to find the best deal for you all without you ever having to load a webpage. It s a different way of thinking about the web s economy moving from attention to action.
I’m not sure this is targeted correctly. As a general rule, new protocols for the web work best when they are associated with individual URLs, not with the somewhat nebulous concept of a site.
Have you considered something like <script type="text/llm"> or Link: <https://api.example.com/llms/foo>; rel="llm:foo", or just normal content negotiation on individual pages?
I chose a single site-wide /.well-known/aura.json manifest primarily for agent efficiency. The goal is for an agent to understand the website s entire capability map with a single, predictable request, rather than having to crawl and parse individual pages to piece that map together. Aura is conceptually similar to robots.txt one file that defines the rules and possibilities for the whole site.
However you are right that context is critical for individual URLs.
I handle that with the dynamic aura-state http header. While the manifest is the static map of everything possible, the aura-state header in each response tells the agent what's available right now for that specific page or state.
(e.g "you are logged in so create_post is now available").
So i get the best of both worlds
efficient site-wide discovery
dynamic
*state-aware execution
You're right...
Aura.json file itself is completely voluntary. A badly behaved agent can just ignore it.
But this is where the model differs from robots.txt. Aura isnt the fence, it s the official map to the gates. The real enforcement happens on the backend at the API endpoint, just like any normal web app.
For example, aura manifest says the create_post capability needs auth. If an agent ignores that and POSTs to /api/posts without a valid cookie, our server's API will reject it with a 401. The manifest doesnt do the blocking, the backend does. It just tells the cooperative agents the rules of the road ahead of time.
So the real incentive for an agent to use Aura isnt about avoiding punishment, it s about the huge upside in efficiency and reliability. Why scrape a page and guess at DOM elements when you can make a single, clean API call that you know will work? It saves the agent developer time, compute resources, and the headache of maintaining brittle scrapers.
So;
robots.txt tells good bots what they shouldn't do.
aura.json tells them what they can do, and gives them the most efficient way to do it, all backed by the server's actual security logic.
The primary purpose of robots.txt isn't to deny access. That's just a sideline. The intended purpose is to do exactly what this aura proposal does: to provide guidance to crawlers as to what parts of the site are valuable to crawl. That's why it's voluntary: it's main reason for existing is to benefit the crawlers in the first place.
In that light, I guess your proposal makes a certain amount of sense. I don't think it addresses what a lot of web sites want, but that's not necessarily a bad thing. Own your niche.
You re right that not every website needs this today. My bet is that this becomes essential for any site that wants to be a verb (a place to do actions), not just a noun (a place to read content), in the emerging agent driven web. Thanks for the thoughtful discussion.
No, sitemap serves a similar but different purpose. The design goal of robots.txt is to guide crawlers to the parts of the site that are worth crawling. That it's used as a very weak method of access control is a hack.
Fair point.. If a website's manifest is wrong any agent trying to use it will just get an error. The agent will learn not to trust that site and the site owner just ends up making their own features unusable. The real incentive is to be useful. Websites that provide an honest manifest are the ones that will actually work with agents.
If someone s goal is simply to break the interaction with agents they can certainly do that by publishing a broken manifest. But that s no different from how the web works today. There will always be a divide between those who build useful services and those who act in bad faith. The solution isnt in the protocol itself, but in the community that uses it. Just as we now have services that warn us about malicious websites a healthy AURA ecosystem would naturally lead to open, community run reputation services. An agent could check a site s aura.json against such a service before trusting it. If a manifest is found to be intentionally misleading its reputation would drop, and agents would collectively learn to ignore it. The system corrects itself through shared information. Trust has to be earned, not just declared in a file.
The comparison to OpenAPI is the main thing to address and you re right to ask why it isn t enough.
OpenAPI is fantastic for describing a static API for a developer to read. But the web is more than that its a dynamic stateful environment built for human interaction. The current trend of forcing AI agents to navigate this human-centric web with screen scraping and DOM manipulation is brittle and I believe, unsustainable. Its like sending a robot into a grocery store to read the label on every single can instead of just asking the manager for the inventory list.
This is where Aura tries to be different in two key ways
Control & Permission:not just Documentation: Aura is designed from the website owner's perspective. It's a way for a site to say "This is my property and here are the explicit rules for how an automated agent can interact with it." The aura.json file is a handshake a declaration of consent. It gives control back to the site owner.
Statefulness(This is the big one): An OpenAPI spec is stateless. It cant tell an agent what it can do right now based on its current context. This is what the AURA-State header solves. So for example before you log in the AURA-State might only show you list_posts and login capabilities. After you successfully call login the very next response from the server includes a new AURA-State header that now unlocks capabilities like create_post and update_profile. The agent discovers its new powers dynamically. This state management is core to the protocol and doesn't really have a parallel in OpenAPI.
You re right to be skeptical and as I said in my post maybe Aura isnt the final answer. But I strongly believe the web needs a native capability-aware layer for the coming wave of AI agents. The current path of brute force interaction feels like it will break the open, human-centric web we ve all built.
The key distinction is that im not trying to reinvent the API format wheel, im building a new steering wheel for a new kind of driver the AI agent. The actual "poorly reinvented wheel" today is screen scraping and brittle DOM manipulation which is how most agents are forced to interact with websites. AURA is the direct answer to that chaos.
A spec like jsonapi is excellent for structuring the payload of a single, known api endpoint. AURA operates at a higher level of abstraction, site-wide capability discovery and state management.
An agent doesnt just need to know how to format a request it first needs to know what is possible to do and whether it can do it right now. This is the problem json:api isnt designed to solve. For example, when an agent first visits a site it fetches the aura.json manifest and sees the login and list_posts capabilities. The create_post capability isnt even visible to it. After the agent successfully uses the login capability, the server's next response includes the dynamic aura-state header, which now advertises that create_post and logout are available. This state-aware dynamic map of available actions which changes with the user s context, is the core of the protocol. It s a formal language for the if-then logic of a user interface, built on established standards like rfc 6570 uri templates so it s a stable foundation. It s not another way to format json, it s a way for a website to declare its interactive grammar to the web.
The current way agents interact with the web is a problem.My view isnt that we should "cater" to them but that we should define the terms of engagement before they define it for us. Right now they re using scraping. That gives site owners zero control. Aura is an attempt to hand control back to the site owner by providing a clear aura.json manifest. It s about consent. As for using AI to build it, I believe you have to deeply understand a technology to help steer it.
Hm, I don't see much reason to use it unless the website in question is paid. Web today is run on ad revenue and not only won't the site extract any revenue from the agent, it even won't look as a person for ad purposes.
The way I see it the choice for websites is quickly becoming not ads vs. no ads but getting scraped for free by ai companies vs. offering a clean managed front door.
Right now, when an ai scrapes a site, it bypasses the entire user experience that site owners have carefully designed to guide visitors and keep them engaged. The ai just extracts the data and leaves completely ignoring the layout, the related content, and any other part of the site s strategy. Aura is a way for a site to manage that interaction, and this opens up a couple of possibilities. For paid services the aura.json manifest can technically define which capabilities require payment. Your ai agent connected to your payment method, could then directly pay for a api call on your behalf to complete a task.
But perhaps more interestingly for adsupported sites aura enables a new kind of advertising. Since the ai s request contains precise user intent(e.g. searching for flights to London) the api response can include a highly relevant, structured ad object right alongside the data. That s an ad delivered at the peak moment of user intent, which is far more valuable than a simple banner impression. It gives control back to the site owner to build a business model that actually works in a world full of ai agents.
> It proposes a new standard for AI-web interaction that moves beyond fragile screen scraping and DOM manipulation towards a robust, secure, and efficient machine-readable layer for the internet.
This is nothing like robots.txt, it is much more like a sitemap. In fact, this design goal is almost word for word the point of the semantic web in general. You may find that there are existing working groups for similar resource description frameworks. Given how poor adoption of semantic tagging has been, I somewhat doubt sites start doing it just for LLMs.
Incidentally, I thought the whole point of an AI agent was that it could read and understand things by itself. I welcome any improvement in the semantic content of the web, but isn't scraping kind of the point?
You made good points. Aura is more like a sitemap for actions than robotstxt. I just wanted to make Aura much simpler, using only plain JSON and HTTP. I think the Semantic Web was too complex for people to use.
So why is now a good time for this idea? I think it s because of AI agents. Today big companies spend a lot of money on web scraping and their tools break all the time. This gives them a real reason to want a better way.
And for your last question, can't AI just read the page? Yes it can, but it s very slow and it breaks easily. It s the difference between using a clean API versus guessing where to click on a page. Aura is just a way for a website to offer that clean API-like path. It's faster for everyone and doesnt break when a small thing like a button s color changes. Thanks for the feedback.
As with what others had said, this is less of a robots.txt and more of a sitemap.
The issue with this is that website owners don't want to do this. Take Reddit removing the API for example. Everyone just switched to scraping the Reddit website instead for any remaining third party clients.
Yes, APIs were supposed to be a compromise to lower the resources needed on both sides, but Reddit's stock price is linked to the value of "their" data, so...
Alternatively, malicious website owners may make incorrect Aura files to mislead user agents. Then we're back to screen scraping as the ground truth, because behaving like a human is the best way to avoid discrimination.
If the website owner is malicious, the site's content is untrustable either way.
The Reddit example gets right to the heart of it: when a platform loses control over its data and how it makes money, it just shuts the door on open access.
And that s really what aura is trying to address. If we dont figure something out the web could easily end up just being a handful of big ai sites, and all the smaller, independent sites might just fade away.
The goal with aura is to give control back to the people running the websites. It s not about blocking ai but about giving sites a clear, standard way to say "here s how you can work with me meaningfully...". This means an agent can do something specific and useful without costly, aimless scraping, and it lets site owners build cool, new features just for ais.
and you re right to worry about malicious manifests but that s a trust problem. A site that lies in its aura file would get a bad reputation fast just like a phishing site does now.
At the end of the day aura is a bet that we can build an open, capability based web where site creators can join in on the AI revolution on their own terms. Time will tell if it's the right technical answer, but it s a conversation i think we absolutely need to have to keep the web diverse and creative.
How is it different than llms.txt? https://llmstxt.org/
Are any websites actually using it (for llms.txt: https://llmstxt.site/)? Why do I need to npm install anything instead of writing a text file?
> While websites serve both human readers and LLMs, the latter benefit from more concise, expert-level information gathered in a single, accessible location.
This honestly made me laugh out loud.
The main difference is -llms.txt is for reading content --while aura is for performing actions
-llms.txt tells an AI here is a clean, simple version of this page for you to read --Aura tells an AI here is a capability called create_post, it needs a title and content, and you can use it by POSTing to the /api/posts endpoint.
llms.txt is for reading, aura is for doing.
You also asked why you need npm install. You dont! To add Aura to your own website, you only need to create one static file - public/.well-known/aura.json. That's it. It s just as simple as creating a llms.txt file.
The npm install command in my project is for people who want to download and run my full reference implementation (the example server and client). It is not a requirement for the protocol itself.
And for your last question, Aura is new so no other websites are using it yet.
Honestly I love the llmstxt idea if only because it implies that I can get an unbloated version of the web for my own use. Reader mode gets way easier if a .md is available everywhere.
Which of course means this is never going to fly with any site that needs to show you ads.
That s a reallyy sharp observation and you re right it s the core challenge for the web as we know it. For any site that depends on ads hiding the UI is a complete nonstarter. The key is that aura isnt really about creating another reader mode. Its goal is to fundamentally separate a website s functions from its visual presentation paving the way for a truly intent based web.
Think of it this way -instead of an ai trying to find and click the "post comment" button it can just use the "post_comment" capability the site offers through a clean api. While this seems to sidestep the ad model it actually enables a more direct one. A site could specify that a particular action, say a premium translation feature, requires a small payment or an api key. It s a way to get paid for the actual value you provide, not just for ad views.
This could even change how search works like a search engine that indexes what sites can do, not just what they say. Your personal ai could then find and execute the "book a flight" capability from multiple airlines to find the best deal for you all without you ever having to load a webpage. It s a different way of thinking about the web s economy moving from attention to action.
I’m not sure this is targeted correctly. As a general rule, new protocols for the web work best when they are associated with individual URLs, not with the somewhat nebulous concept of a site.
Have you considered something like <script type="text/llm"> or Link: <https://api.example.com/llms/foo>; rel="llm:foo", or just normal content negotiation on individual pages?
I chose a single site-wide /.well-known/aura.json manifest primarily for agent efficiency. The goal is for an agent to understand the website s entire capability map with a single, predictable request, rather than having to crawl and parse individual pages to piece that map together. Aura is conceptually similar to robots.txt one file that defines the rules and possibilities for the whole site.
However you are right that context is critical for individual URLs. I handle that with the dynamic aura-state http header. While the manifest is the static map of everything possible, the aura-state header in each response tells the agent what's available right now for that specific page or state. (e.g "you are logged in so create_post is now available").
So i get the best of both worlds efficient site-wide discovery dynamic *state-aware execution
Like robots.txt, it has the fatal flaw of being unenforceable.
You're right... Aura.json file itself is completely voluntary. A badly behaved agent can just ignore it. But this is where the model differs from robots.txt. Aura isnt the fence, it s the official map to the gates. The real enforcement happens on the backend at the API endpoint, just like any normal web app.
For example, aura manifest says the create_post capability needs auth. If an agent ignores that and POSTs to /api/posts without a valid cookie, our server's API will reject it with a 401. The manifest doesnt do the blocking, the backend does. It just tells the cooperative agents the rules of the road ahead of time.
So the real incentive for an agent to use Aura isnt about avoiding punishment, it s about the huge upside in efficiency and reliability. Why scrape a page and guess at DOM elements when you can make a single, clean API call that you know will work? It saves the agent developer time, compute resources, and the headache of maintaining brittle scrapers.
So;
robots.txt tells good bots what they shouldn't do.
aura.json tells them what they can do, and gives them the most efficient way to do it, all backed by the server's actual security logic.
The primary purpose of robots.txt isn't to deny access. That's just a sideline. The intended purpose is to do exactly what this aura proposal does: to provide guidance to crawlers as to what parts of the site are valuable to crawl. That's why it's voluntary: it's main reason for existing is to benefit the crawlers in the first place.
In that light, I guess your proposal makes a certain amount of sense. I don't think it addresses what a lot of web sites want, but that's not necessarily a bad thing. Own your niche.
You re right that not every website needs this today. My bet is that this becomes essential for any site that wants to be a verb (a place to do actions), not just a noun (a place to read content), in the emerging agent driven web. Thanks for the thoughtful discussion.
Are you thinking of sitemap?
No, sitemap serves a similar but different purpose. The design goal of robots.txt is to guide crawlers to the parts of the site that are worth crawling. That it's used as a very weak method of access control is a hack.
I will purposefully implement this with a bunch of incorrect information for every website I make
Fair point.. If a website's manifest is wrong any agent trying to use it will just get an error. The agent will learn not to trust that site and the site owner just ends up making their own features unusable. The real incentive is to be useful. Websites that provide an honest manifest are the ones that will actually work with agents.
…that’s the point
If someone s goal is simply to break the interaction with agents they can certainly do that by publishing a broken manifest. But that s no different from how the web works today. There will always be a divide between those who build useful services and those who act in bad faith. The solution isnt in the protocol itself, but in the community that uses it. Just as we now have services that warn us about malicious websites a healthy AURA ecosystem would naturally lead to open, community run reputation services. An agent could check a site s aura.json against such a service before trusting it. If a manifest is found to be intentionally misleading its reputation would drop, and agents would collectively learn to ignore it. The system corrects itself through shared information. Trust has to be earned, not just declared in a file.
You should add a section to explain why OpenAPI isn't enough (narrator's voice: it is).
Otherwise, it just seems you vibecoded the wheel.
The comparison to OpenAPI is the main thing to address and you re right to ask why it isn t enough.
OpenAPI is fantastic for describing a static API for a developer to read. But the web is more than that its a dynamic stateful environment built for human interaction. The current trend of forcing AI agents to navigate this human-centric web with screen scraping and DOM manipulation is brittle and I believe, unsustainable. Its like sending a robot into a grocery store to read the label on every single can instead of just asking the manager for the inventory list.
This is where Aura tries to be different in two key ways
Control & Permission:not just Documentation: Aura is designed from the website owner's perspective. It's a way for a site to say "This is my property and here are the explicit rules for how an automated agent can interact with it." The aura.json file is a handshake a declaration of consent. It gives control back to the site owner.
Statefulness(This is the big one): An OpenAPI spec is stateless. It cant tell an agent what it can do right now based on its current context. This is what the AURA-State header solves. So for example before you log in the AURA-State might only show you list_posts and login capabilities. After you successfully call login the very next response from the server includes a new AURA-State header that now unlocks capabilities like create_post and update_profile. The agent discovers its new powers dynamically. This state management is core to the protocol and doesn't really have a parallel in OpenAPI.
You re right to be skeptical and as I said in my post maybe Aura isnt the final answer. But I strongly believe the web needs a native capability-aware layer for the coming wave of AI agents. The current path of brute force interaction feels like it will break the open, human-centric web we ve all built.
"vibecoded the wheel". Where did you get this from‽
> I actually used AI extensively to help me write the reference implementation for AURA.
Why reinvent the wheel poorly when you have a hundred of solutions like https://jsonapi.org/?
The key distinction is that im not trying to reinvent the API format wheel, im building a new steering wheel for a new kind of driver the AI agent. The actual "poorly reinvented wheel" today is screen scraping and brittle DOM manipulation which is how most agents are forced to interact with websites. AURA is the direct answer to that chaos.
A spec like jsonapi is excellent for structuring the payload of a single, known api endpoint. AURA operates at a higher level of abstraction, site-wide capability discovery and state management.
An agent doesnt just need to know how to format a request it first needs to know what is possible to do and whether it can do it right now. This is the problem json:api isnt designed to solve. For example, when an agent first visits a site it fetches the aura.json manifest and sees the login and list_posts capabilities. The create_post capability isnt even visible to it. After the agent successfully uses the login capability, the server's next response includes the dynamic aura-state header, which now advertises that create_post and logout are available. This state-aware dynamic map of available actions which changes with the user s context, is the core of the protocol. It s a formal language for the if-then logic of a user interface, built on established standards like rfc 6570 uri templates so it s a stable foundation. It s not another way to format json, it s a way for a website to declare its interactive grammar to the web.
How about we don't cater the web for the handful of companies hell bent on killing it?
> I actually used AI extensively to help me write the reference implementation for AURA.
So that's why. You drank the kool aid.
> The project is MIT licensed. I'm here all day to answer questions and listen to your feedback—especially the critical kind. Let's discuss it.
You can tell that OP is a big AI believer by the final sentence. That's gotta be one of the most ChatGPT lines I've ever read.
The current way agents interact with the web is a problem.My view isnt that we should "cater" to them but that we should define the terms of engagement before they define it for us. Right now they re using scraping. That gives site owners zero control. Aura is an attempt to hand control back to the site owner by providing a clear aura.json manifest. It s about consent. As for using AI to build it, I believe you have to deeply understand a technology to help steer it.
Hm, I don't see much reason to use it unless the website in question is paid. Web today is run on ad revenue and not only won't the site extract any revenue from the agent, it even won't look as a person for ad purposes.
The way I see it the choice for websites is quickly becoming not ads vs. no ads but getting scraped for free by ai companies vs. offering a clean managed front door. Right now, when an ai scrapes a site, it bypasses the entire user experience that site owners have carefully designed to guide visitors and keep them engaged. The ai just extracts the data and leaves completely ignoring the layout, the related content, and any other part of the site s strategy. Aura is a way for a site to manage that interaction, and this opens up a couple of possibilities. For paid services the aura.json manifest can technically define which capabilities require payment. Your ai agent connected to your payment method, could then directly pay for a api call on your behalf to complete a task. But perhaps more interestingly for adsupported sites aura enables a new kind of advertising. Since the ai s request contains precise user intent(e.g. searching for flights to London) the api response can include a highly relevant, structured ad object right alongside the data. That s an ad delivered at the peak moment of user intent, which is far more valuable than a simple banner impression. It gives control back to the site owner to build a business model that actually works in a world full of ai agents.
> It proposes a new standard for AI-web interaction that moves beyond fragile screen scraping and DOM manipulation towards a robust, secure, and efficient machine-readable layer for the internet.
This is nothing like robots.txt, it is much more like a sitemap. In fact, this design goal is almost word for word the point of the semantic web in general. You may find that there are existing working groups for similar resource description frameworks. Given how poor adoption of semantic tagging has been, I somewhat doubt sites start doing it just for LLMs.
Incidentally, I thought the whole point of an AI agent was that it could read and understand things by itself. I welcome any improvement in the semantic content of the web, but isn't scraping kind of the point?
You made good points. Aura is more like a sitemap for actions than robotstxt. I just wanted to make Aura much simpler, using only plain JSON and HTTP. I think the Semantic Web was too complex for people to use. So why is now a good time for this idea? I think it s because of AI agents. Today big companies spend a lot of money on web scraping and their tools break all the time. This gives them a real reason to want a better way. And for your last question, can't AI just read the page? Yes it can, but it s very slow and it breaks easily. It s the difference between using a clean API versus guessing where to click on a page. Aura is just a way for a website to offer that clean API-like path. It's faster for everyone and doesnt break when a small thing like a button s color changes. Thanks for the feedback.