I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their firm involves taking in documents from healthcare plans and organizations, analyzing them for certain kinds of data, and then importing that data into an internal system they use to analyze and provide guidance on plans. The internal system can contain hundreds of documents for an individual client. All of the documents have the same information (roughly) but in totally diverse formats and styles. Once it's in the system, it's easy to compare and analyze across documents and the research process is much faster.
They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.
LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.
(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)
Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd.
It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).
Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.
Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
I see problems in LLMs doing research, making drafts, etc.
I see 3 kinds of potential users.
1. Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
2. Fresh out of law school graduates.
They think they can do fast work, draft quick replies, do fast case research, enh. They should go the traditional route of interning, learning the "art", they probably won't need it because they will learn "how the law works"
3. Established attorneys.
They usually have templates made for most things, over their career, they have established routines of making things fast, maybe it can help them but to say replacing good chunk of work or offloading to LLMs isn't probably going to go well.
There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....
>Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
I used LLMs to replace a lawyer this year to great effect. I successfully advised myself as to the right strategy and drafted an immigration petition in the correct language. Granted it wasn't 100% LLMs (I also supplemented it with "traditional" online research and wrote the letter myself using the LLM version as rough guidance), but still, I certainly would've gone to a real lawyer in 2024 for this stuff but didn't.
Professional advisory work can be divided into two types:
1. Once you know the rules of the game, the next move is immediately obvious.
2. Knowing the rules is not enough to know the next move -- that requires judgement and experience.
LLMs in the hands of someone smart and savvy can easily replace the first type of work.
> Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you. I think this is a valid use case to get at least some understanding what the next steps might be and if it even makes sense to seek legal representation.
Might be true today, but I still feel pareto principle applies. These points also sound like gatekeeping a bit and I hope future AI versions prove that law is also essentially something that can be made as programmable as possible for most common answers if the underlying principles and constitution are organised such.
If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system.
Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.
I do think non legal background folks can use LLMs today to sense check legal ideas, like for instance, "what are my legal rights in this situation?"
These situations usually are not ones that an individual can justify the time or money to contact an actual lawyer, but then if they do decide to contact one they will come in with better questions and more of a sense of what they are expecting.
This is similar to medical. Should you use LLM to diagnose yourself, treat yourself with prescription drugs you buy from shady gray market online sellers? No. But you can use it very well to know when it's time to go to the doctor and what to ask.
There are 3x as many lawyers per capita in the US as we had in the 1970s. Credit agreements that used to be 50 pages are now 500. This increase in volume has not lead to any increase in actual value. The legal profession is a metastatic cancer on our society. Un-fuck this situation and maybe I'll think about taking your advice of not using a LLM.
I'm thinking back to some of my interactions with corporate lawyers and you know what? My bar (<- pun) for trusting Astra more than a lawyer is pretty low.
Lawyer here (non practicing so to be clear none of this affects me):
most comments I read here don't seem to realize that different areas of law have very very different economic models and don't even mention which one they think will be affected or why, they just sort of lump it all together.
For example: It is highly unlikely llms will have any meaningful effect on high value personal injury law - I don't see a 5 million dollar case being handed to an LLM when the majority of the cost is in trial aids and not even lawyers. It may affect where and how they advertise. It may affect how they work. But it seems really unlikely to put any of them out of business any time soon by people doing it themselves.
Will it affect other areas more? Maybe. Probably? But so far I haven't seen a ton of comments that make specific enough arguments that they could really be debated or responded to effectively with a useful opinion
No. It'll be like software. Entry level employment will be affected. You wont want or need associate attorneys when you can hire a brilliant AI associate for 1/10th the price.
In commercial law, the business model is to rent the associates out to clients at rates higher than you pay (d/t your supervision). No client will pay $350/hour for your AI.
And for direct billing, it's not your interest to do things faster for the client (at risk of making a mistake).
So the real question is whether these pay-by-the-hour models will die as a result of competition from fixed-price transactions made viable via automation. I think not because the entire difficulty of legal work is that it's a complex bundle of issues until you sort them out; once sorted, there's really little to do other than execute (read: paralegals).
> No. It'll be like software. Entry level employment will be affected.
Whether everyone at the labs knows it or not, this it what they are doing to entire industries, starting with software: Embrace, extend, and extinguish
It'll be like software. The demand for both software and legal services is effectively infinite. If it becomes cheaper to file lawsuits then we'll get more lawsuits, not the same number of lawsuits at a lower price.
There will still be jobs for competent associate attorneys to prompt LLMs and review output. But frankly a lot of attorneys — especially many of those who attended "Third Tier Toilet" (TTT) law schools — were always a waste of money and should have never entered the profession in the first place.
Can AI replace divorce lawyers? Marriage has no friction, but divorce is high friction. Made impossibly expensive and time consuming because of the cartel of divorce industry.
There are hundreds if not thousands of divorce lawyers. How is it a “cartel?”
Divorce is expensive because divorces are acrimonious and people in that situation would rather spend the assets fucking each other over than letting the other person get them.
AI will likely replace divorce lawyers (or, at least make existing divorce lawyers much more efficient) - for non-contested divorces that are amicable. Roughly 95% of what a divorce lawyer does in that context is entirely automatable.
Divorce is extremely cheap if you both agree on everything. Its literally one of the options on the divorce filing. You just pay the court fees. The problem is both parties rarely agree on everything and that's when it gets expensive.
There’s an argument to be made if divorce was easier than more people might get married when they are less serious. My view though is make it harder to get married and easier to get divorced.
It's not just the divorce lawyers. We have 3x as many lawyers per capita in the US as in the 1970s. The system is designed by lawyers for lawyers. They will create more bureaucracy and more procedure to consume any surplus.
huh? Many/most are no-fault jurisdictions with strict rules on dividing community property. Divorce is easy and cheap, unless it's hard to find assets.
Personal injury attorney. AI is entrenched of every single phase of litigation, from the busywork of getting records, research, developing case theory and themes, expert selection, expert collaboration, negotiating, drafting demands, drafting pleadings, drafting discovery, drafting discovery responses, deposition prep, motion practice, jury selection, voir dire, opening statements, witness direct outlines, witness cross outlines, closing statements, and appeals.
Yeah, this matches my friends - they use LLM's for tons of things, using Eve and such. But nobody is replacing the lawyers themselves in cases of any value.
I also have friends in the world of high-volume personal injury (IE the morgan and morgan's of the world) but they also aren't being replaced by LLMs so far.
I'd love to get your thoughts on the current frontier models for law, or a link to anything you've written on the subject. Where do you think they're at?
The word “never” is a dangerous word. In my lifetime I have seen it many times where something was supposed to “never” happen only to see it happen a few years later.
I'll buy that - but, how often does one need an honest-to-goodness expert? 99% of the time most people's requirements can be satisfied with general-competence, and, while the AI systems we have today are mostly only generally competent in porting code from one language to another - they are getting consistently better. I expect we'll start to see general competence in many more areas in the next few years.
Yeah geeks just stop trying! Seriously? Yeah, we’re in a bubble and narratives are ahead of reality but if you really don’t think AI is and will continue to eat knowledge work, just keep making your buggy whips.
The need for actual lawyers will persist I think from my own experience. I attempted drafting a contract with some points myself using AI, but after several edits I wasn't sure if it was correct. Sending it to an actual lawyer ended up in so many corrections I couldn't imagine the first time. One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.
I'd correct this to "The need for actual GOOD lawyers will persist I think from my own experience."
I have worked with many lawyers in my career. The bad ones will look at a contract and tell you fifty things that can be improved. Meanwhile, you take the same contract to a really great lawyer and they tell you that's it's not worth fighting over the changes and to just keep it as is. It's kind of like how a beginner or intermediate software engineer might tell you how you can build out a k8s or auto-scaling fargate cluster for your project and a really senior engineer might look at the same thing and explain that it's not worth worrying about scalability right now.
I find that when I feed contracts to ChatGPT (which I do all the time) it will try to find 50 things wrong with it. For example, as a test, i fed chatgpt an employment offer from a US government agency. It came back with 1152 words and 14 recommended (some of them substantial) changes. Meanwhile, any half-decent employment attorney would probably laugh at me if I asked them to redline an employment offer from the State Department.
I’ve found that good lawyers are about getting things done. You could nitpick the terms of a contract, but would you rather buy the house or have a perfect contract? Once I realized that, my view of lawyers changed (for the better).
In my recent experience, it seems like LLMs may even increase need/demand for actual lawyers (more ideas + more words = more lawyer time). I expect lawyers will generally adopt the technologies that will benefit them, rather than the technologies that would increase standardization and trust
The same has happened with modern finance. Despite more technology that has power to drive more transparent/efficient markets, we have less perfect information sharing and a larger group of middlemen capturing a growing share of profits of the economy
I want to take a step back and observe that you're saying expensive specialists will continue to be required if two parties want to make an agreement. That seems like something worth fixing rather than just accepting.
> I want to take a step back and observe that you're saying expensive specialists will continue to be required if two parties want to make an agreement.
A lot of contract-lawyering involves getting ahead of when things go south and the two previously agreeing parties find themselves in disagreement.
Millions if not billions of agreements are made every day without the involvement of lawyers. Lawyers are only needed for particularly complex situations and even there they are generally only "needed" in the sense that parties feel they would add value.
In my experience, those expensive specialists help me protect against things me and the counterparty cannot even imagine when making an agreement. Do you mean that this would be fixed by AI or by making simpler agreements?
Contract law has evolved because transfering the ownership of things is fraught with difficulty. Ownership as a concept itself is notoriously hard to define.
Consider housing - when you buy a house you own that land. Except if its an apartment, you just own the internal walls. Or when the government aquires your land through eminent domain. Or mineral rights - which don't belong to you, you just own the top 5 feet of topsoil. Or if you neglect a property and some squats in it. Or if you were given a fraudulent payment, and they take back the property.
Just getting people to agree is notoriously difficult when people have different often conflicting desires.
Not trying to be dismissive here but a lawyer will always edit your proposal. I do think lawyers will persist but not as many. And they will work very differently, much like we already use claude / codex to code - a big part of the contract probably won't be read by the lawyer.
I would rather argue they SPECIFICALLY will read everything. However, they'll likely often just be like "I'd phrase this differently but that works, too".
Also, unlike SW engineers, lawyers control their industry and will quickly circle the wagons to protect themselves. Paralegals are probably screwed, but lawyers will do fine. I'm considering law as my post-ai-SWE-apocolypse career path.
I’m thinking asking the same lines, but also looking into research / lab chemist so I can hang out with my best friend. If im going to change careers so drastically, I might as well enjoy the company.
LLMs are the first genuinely useful legal tech since the Internet. I'm pretty shocked, though, at the delta between how competent Claude is on code versus legal work. It's good for research and data organization, but terrible for drafting. I wonder if this is a structural problem with the lack of feedback loops. In law, there's no compiler to check for logical or continuity errors in your brief, and there's no unit tests to check for correctness or performance.
Even without that, I think it'll be extremely valuable to clients to allow them to answer simple questions without a lawyer, figure out the lay of the land so they can supervise their counsel, etc.
You should know - for coding they make terrible mistakes as well.
But programmers have this concept of a "code review" where another person looks at the code to look for problems.
We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.
> LLMs are the first genuinely useful legal tech since the Internet
That is an incredible statement that could not be further from the truth. Large scale adoption of email, searchable document databases like Westlaw, LexisNexis, PACER, etc.. , OCR Software, electronic signatures, and tons more have had a much more defineably positive impact on the legal profession since the internet came about.
I have found that it’s useful generally speaking to get the intent of contracts and red lines, but actual drafting I agree is where I lose all confidence. My guess is that the significance of the difference between using a word like “and“ or “or“ can be so meaningful that that level of nuance can often be lost. But I know nothing I’m not in the space, I just pay too much money for lawyers.
Not being snarky here but I've encountered some really crappy lawyers and they seem to be doing great ($$$).
It really sucks one cannot publish something like "this guy charged me $5k, wrote my name wrong and forgot important paperwork the day we showed up to court" as that could make you liable for damages. There should be a safe harbor for that.
I was in a similar situation and tried a different approach.
I had start with asking for a contact with some details I provided. But then I ask the model to be an experienced corporate lawyer and ask me a series of questions to gather the details it needed and then write the contract. The result covered a lot of details that were highly relevant but were absent from the original attempt.
The key insight is that you can lean on the model to cover your unknown unknowns.
Your comment is a great example of the phenomenon where people think AI is an expert in areas that are not because they have no way to determine if the seemingly good looking result is genius, gibberish, or somewhere in between.
I agree. But, we will need a lot fewer of them. My small company called our lawyer a lot more before the advent of AI. Mostly to clarify a few things or review simpler contracts. Now, for simple things, we do not call them at all.
The salaries are a result of the insane hours they have to do. Biglaw needs to pay those salaries to keep getting young associates in the door because burnout is a real problem in the industry. If salaries were lower and hours/stresses were lower, you would probably still find people wanting to be lawyers. A different kind of person, maybe.
(Not that I think AI will necessarily have that effect. We just don't know yet)
It’s a professional organization that only allows so many people to pass the bar per year so I don’t see that changing anytime soon… if anything if they need fewer lawyers they will just pass fewer people to keep their wages high
As costs for legal work drop we might expect demand for that work to increase. For example, it may be the case that legal help becomes accessible to entire segments of the population that had no access previously.
The best writer I know of was an associate attorney. He didn't have the technical background nor the in depth computer related knowledge, and relying on the information I fed him. But man the briefs he filed to the court were amazingly good. Reading them I would have been convinced his side was right if I were in the jury.
Law LLM will surely help competent lawyers in their fields with greater sources of knowledge not in their core area of expertise.
And to add to your last point yes it should also be like code: where the costs of development are coming down. Legal costs should trend down if law firms are effectively leveraging LLMs to accellerate their procedures (though I won't hold my breath)
> The need for actual lawyers will persist I think from my own experience.
The outcome of a case shouldn't depend on someone's fallible ability to recall facts or convince other people or point their index finger*
Law should generally be deterministic. One's CHA stat should have no bearing on justice.
There should still be human judges, but the middleman between the judge and petitioner could easily be removed, and have generally been seen as leeches since forever anyway.
Though, like how the USA opts to remain in the Stone Age with regard to tax filing because of lobbying by tax software companies, this faction of society will flail the hardest before they admit they're obsolete.
----
* What's a lawyer's favorite programming language? Objection C.
The ability to clearly and effectively communicate, and the ability to know and recall the facts of a case is always going to be important. It'd probably be a more fair system if judges and jurors couldn't see the attorneys or even the people they're representing. Now that many cases are handled over video calls it'd be pretty easy to make that change.
A lot of communication is conveyed through more than the words themselves though and even though it's often misinterpreted people put a lot of value in that information. I suspect that even if we took away the ability to see the people involved judges and jurors would still find bias in the tone/quality of people's voices. Even if everything was reduced to text they'd find bias based on word choices, phrasing, spelling, and grammar.
Yep, just yesterday, Nike removed one leach middlemen called retailers and sell directly through their app. Turns out working out very well for them too.
The typical pattern is called “deskilling”. It doesn’t usually mean a skilled profession will disappear overnight. Instead, the job might be done by less expensive folks like paralegals.
An example is in the banking industry, where making a loan used to require deep analysis of a person’s credit worthiness. Now they use an algorithm (credit scores) which means someone with less experience can do it.
If law follows the same pattern, a job done by someone making $500/hour might be done by someone making $50/hour.
"law" and those who pratice "law" have managed to place itself largly in a monopoly. This is a huge drain on the economy. Law is expensive and irrational.
Our entire world is about reducing the friction of conducting a transaction
How to fix the lack of trust which is needed to conduct a transaction
Contracts affix a moment of trust in time, so even when trust is lost between parties the contract has guardrails for behavior and consequences for acting outside those rails
There are many transactions that are completely inaccessible to most people because the need for a contract or lawyer is too high
LLM use allows for those inaccessible transactions to decrease, and get people further along to the point where a lawyer is accessible as well, and economically viable to use
> One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
>Its just like code I suppose, if you can read and understand and validate
I share this, when I asked an attorney for a contract, they were much simpler when compared to self redacted or AI redacted contract, it was as if I were rediscovering lots of concepts and I HAD to put them in paper, even if they were defaults and were essentially no-ops.
Comparable to:
> Mkdir $PWD/path
instead of
> mkdir path
or conditions so rare that they don't merit including.
> x = 0
> if x!=0:
> raise Exception("Bit flip by solar rays or otherwise")
>The need for actual lawyers will persist I think from my own experience
At the very least because the attorney monopoly assigns a 5ish year of training as PoW and natural personhood as Identity as protection for Sybil Attacks.
If anything, the exclusive right of attorneys to represent clients in court, file motions, and enjoy professional secret, is more valuable, as it's not something AI can ever compete at. (Barring a wild recognition of machines as humans, or an overturn of the impossibility of companies to appear in court.)
It’s a bad day to be a lawslop company. When you’re reliant on other companies to do all of the AI part of your AI product, they can just train on your traffic and eventually eat you.
Max Junestrand has consistently said Legora treats the model layer as swappable, selecting across frontier providers rather than building the product around one.
OpenAI didn't need to name Legora and Harvey in the second paragraph of the launch post.
They are pre-empting the obvious interpretation of Astra for Law: that moving this far up the legal stack puts them in direct competition with their biggest legal AI customers.
“Don't worry, they can build on us” is a pretty conspicuous message to include on launch day.
They have clearly thought about some pessimistic outcomes.
Switching from ChatGPT Enterprise to Legora at my firm was a godsend, it's so much better for legal work, even with the frequent changes to the underlying models.
>Max Junestrand has consistently said Legora treats the model layer as swappable, selecting across frontier providers rather than building the product around one.
Vendor-neutrality for LLMs is such a weak thesis all around, whether for providers or consumers. It weakens the product by being promiscuous and gains no material benefit at all.
LLMs are magic byte(byte) functions, it doesn't make sense to say "we have different providers for magic".
the thing is a lot of the legal work which will go through this is drafting 100 and 1 variations of draft versions of standard contracts not containing any trade secrets where the contract can be drafted with "replacement/place holder names"
the kind of work mostly done by juniors not yet through their final exam and other "non" lawyers etc.
so it's a slippery slope of "lets just use it for <this> things where it doesn't matter" and then out of laziness and convenience it creeps into all the other places (at least for drafts).
Yeah, it should be freely available, you have to be able to know the rules you're supposed to obey in order to obey them well. I've been making a free API for US law search, you can point whatever model you want at it: https://law.agentlookups.ai/
Very much a work in progress, only federal and state so far, no municipal codes yet, and no case law yet. Big hole, I know. Also working on making the search ranking work better.
> By using the legal search index, Astra for Law can search U.S. case law, statutes, regulations, court rules, and administrative decisions across a corpus of more than 230 million URLs, with sources added daily. Our work with Free Law Project, the nonprofit behind CourtListener, brings its case-law collection covering more than 99.9% of published U.S. precedential case law (opens in a new window) into this research experience.
I find the idea that people can use LLMs to exercise their rights as citizens appealing however. Many people aren't aware of the rights they have, and LLMs are pretty good at surfacing some stuff without having to pay lawyers. Having to hire a lawyer is imo actually a huge way of gatekeeping people from exercising their rights. I heard a lot of local German public institutions are currently being flooded with people arguing their case with the help of LLM that they previously weren't really realistically able to do. So I don't see it all as bad.
>are pretty good at surfacing some stuff without having to pay lawyers
They are also really good at making stuff up as evidenced by the many, many, many examples you read in the news about actual lawyers using AI to write briefs that are full of errors and hallucinations.
In civil courts, you'd likely get more sympathy from a judge if you represented yourself and admitted your lack of understanding, rather than try to appear as someone you're not because you wrote some prompts and copied the output.
Current court systems around the world are just not built to handle the flooding of cases from the citizens.
The stupidest analogy is open source projects having a hard time accepting LLM generated PRs from the masses, because review process is the bottleneck.
No idea how to fix this, to be honest. In coding world, with some mental gymnastics, I can see code not being reviewed by people anymore. In courts, things generally have more consequences, and you can’t really roll back decisions that easily.
you could probably write a cool little gotcha of an SF short story about a barren wasteland of a planet that keeps broadcasting out legalese that's revealed to just be LLM chatbot lawyers pedantically arguing with one another about xeno legal doctrine
The legal system, which is a machine/technology by itself, will be eaten out. I wonder what will replace it. Botnet law arbitrage? – Personal assistants constantly negotiating with each other to avoid permanent civil lawsuits?
the cost of making a legal argument can collapse while the cost of reaching an enforceable, legitimate decision may go higher, which will gate the "justice" system even more.
You presented the concern from my adjacent comment perfectly (“LLM performance: common law vs civil law” essentially). So is AI possibly just growing the “Reverence for Professional Experience” factor that plays such a big role for legal compensation here in the US?
I don’t know in the USA but in France, if it’s deemed that you launched a lawsuit knowing very well it wouldn’t succeed, you are susceptible to get a 10k€ fine. Even jail in serious cases.
Yes. And you(r lawyer) can collect lawyer's fees and you can be made to pay the court fees, if you lose.
IME the American legal system is set-up to discourage litigation, though. A common tactic is to bury your opponent in the threat of heavy damages or jail-time to get them to settle for what you were originally after, which courts are perfectly happy to facilitate because it gets a potentially lengthy trial off their dockets. They'll punish (or be biased against) whichever party seems responsible for not accepting a "reasonable" settlement.
Algorithmic abuse of the system to extract payments already exists in the form of the debt collection industry.
Why the hell are they not fighting fire with fire? This is not sustainable. But it is annoying that the AI labs get to play arms dealer, selling to both sides.
Does hallucination matter for this application? We've moved beyond raw recall being that important, it seems like for law specifically all relevant facts will be cited and checked easily by humans.
> all relevant facts will be cited and checked easily by humans
I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver
But based on stories in the news on the subject, not everyone has this same level of diligence
Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.
The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.
"We have dangerous AGI that can destroy humanity."
"Also, all of your sensitive legal documents will be totally safe with us."
"Also, for some reason even though we have AGI and selling tokens is a fine business, we need to sell a new product specifically targeted at a very high margin and lucrative industry."
In an utopian society, lawyers are an unnecessary profession. Laws should be clear and simple so the common person can be their own "lawyer". LLMs help with that goal.
How long before we start building detailed models of each judge trained on all of their legal output, and then test various legal theories against those judge models in virtual moot court? Craft each pitch to the legal idiosyncrasies of the batter. I assume that real lawyers do this routinely and could use a simulator.
I feel like this is on a different trajectory than what LLM-based tech does. This type of individualistic extrapolation is one of the things they're really bad at, in my experience.
What worries me is the step after this. If god forbid this proves successful and models accurately predict specific outcomes, people will start to ask whether the solution to AI slop lawsuits is to do the judging with AI too.
I'm not familiar with the Vals AI Legal Research Benchmark. But their website has other frontier models' scores, and the scores OpenAI is now revealing for "Astra for Law" are slightly less than Claude and Muse:
> The top is a three-way tie: Muse Spark 1.3 Max, Claude Opus 5, and Claude Fable 5.1 all reach 55.29% all-pass accuracy, a clear ~6-point step ahead of the next model. [Astra for Law reached 54.0%]
> Under partial-credit scoring, Claude Opus 5 reaches 90.58% weighted pass rate but 55.29% under strict all-pass grading, where every rubric check must pass. The gap shows models often get most of an answer right but fail on one or two required elements. [Astra for law reached 90.0%]
Can you imagine the sort of corruption that is possible here? Like if you have a lawsuit against someone or some entity that openai or their investors have business with...
So OpenAI is partnering with Latham Watkins, Freshfields is partnering with Anthropic and Kleiner Perkins is building their own. It'll be interesting to see which wins out here, I don't see how those partnerships can end well for the law firms unless they're making an assumption they'll be sucked dry of USP but the revenue split from the AI labs will make up for it. Why would I pay a premium for Latham Watkins when every other firm can get their expertise and experience in a subscription, and add their own on top?
The most interesting part about this to me was how they bench/compare it, like in the example with Fable:
"Given the same prompt, Astra for Law returned two closely matching precedents; in the litigation example, Claude Fable 5.1 returned a holding that had been reversed on appeal, while in the transactional example it reported finding no such case."
It made me wonder if a good deal of law is about finding a way to work in statements with clear precedents without your opposition noticing and then later drawing upon them in court (as settled precedents, in your favor) after the opposition (perhaps implicitly) accepted it. That would clarify a lot about why some lawyers need to spend so much time pouring over and memorizing past cases (even ones that are only tangentially related); because anything they miss could be used as a potential trojan horse by the opponent.
If this is true that must mean there are a good deal of cases settled using precedent "gotchas" where both sides knew that without the "load-bearing" precedent the outcome would've definitely been the opposite. (i.e precedents almost always trump even valid arguments)
open-ai have proven they can make good / decent models but business strategy is just spray and pray.
they need to pick a lane and optimize for it. coz at their size they can't serve the application layer (a.i startups who can fine-tune models will eat their lunch)
if they gonna do a consumer play - then go ham on that.
otherwise they're gonna get caught in the dreaded middle valley.
They have a lot of compute, so I think it makes sense to spray and see what works. Anthropic is limited in that regard, and focused in coding/tech, but OpenAI don't need to do the same. They even got the lead without having to focus in coding only, which is remarkable.
They can't be seen to commit to strongly to a specific product experience, because if they are understood as a regular tech product business that has way different financial scaling considerations than "superintelligent everything-factory"
I suspect that half-assed announcements like this are a result of different people internally with conflicting incentives resulting in a split-the-baby solution.
Enterprise is eventually get caught (if it isn't already) by the Microsoft/Google. Because with office/teams/suite they were already in every enterprise, and that just added a new tool to existing offerings.
Provisioning and contracts and data retention was just an extension to review of existing ones.
Nobody serious is going to risk sending sensible data to OpenAI/Anthropic, etc because "the benchmarks have shown +8% performance there and +2% there". Irrelevant.
The product isn’t meant for you or me it is for lawyers. If you can’t take on personal liability for a badly written contract then you shouldn’t be using it.
I'd expect they could indemnify you against hallucinations or similar if this gets good enough for that to be a very rare occurrence? Or you could buy insurance on it that's cheaper than hiring a lawyer (not a high bar to clear). I wouldn't rely on it currently, though.
I already use them for that, they are pretty excellent at it. Much better than the terms of use generator products that used to exist. That said, nobody cares to sue your business for the most part until you're big enough to be worth it. By that time, you'll have a team of legal analyst to assist you... or agents should I say.
Then again, nobody will have money to buy anything at this rate, so in all liklihood, this is a total non-issue.
An important take-away from this is the harness is really important. The graphs show that the same model with a better harness performs many percentage points better. OpenAI are getting into the "selling the harness" game in a big way. Why?
What's interesting about that is while not everybody can train or run a model, anybody can build a harness. You and I can build harnesses.
It seems strange that OpenAI would move into a field where any developer can compete with them. I think that tells us a lot about the economics of training and selling inference.
I don't think it says what you imply. I think they're just trying to vertically integrate.
Harnesses are going to be controlled at companies eventually, just like you might not have a choice of OS. They want to make sure they are the complete package.
Any lawyers here who have used AI agents heavily for their work? From what I've heard, they're currently very good at searching, analyzing and drafting documents like contracts and patents, but some say they suck at interpreting the law.
I’m sure there are a wide variety of experiences out there, but here’s my perspective as a former biglaw associate and current solo litigator:
I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow.
In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio.
Again, I’m sure others’ experiences differ based on workflow, legal area, etc.
Agreed. Six months ago, it was basically a gloried grammarly.
But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file.
I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.
They are excellent, especially the latest models. That said, (a) I wouldn't feel safe filing something without a real lawyer looking at it; (b) it can't (easily? legally?) do oral arguments for you; and (c) a lot can happen in the hallways outside the courtroom to move a case forward that the AI can't easily do.
In my experience it basically doesn't even try to interpret the law. It just summarises the publicly available law/guidance out there and, if there is a question about how to interpret some provision, might set out the arguments for each interpretation. It doesn't really take a position. It is pretty good at drafting though. (Legora)
Interesting! In other unrelated domains, models seem more willing to take a position. It may be nuanced, but they do tend to take a stance. I think it's good that it leaves the interpretation to humans, but I wonder if this is also some sort of a guardrail to minimize liability...
What we contributed to Reddit and Arxiv voluntarily before, what we contribute as traces right now are all being used to build business verticals by OpenAI and Anthropic. All these business verticals are being used for is more trace collection which would only strengthen these models and render most humans useless because the ceiling for 1000 swarms of Agents to learn is much higher than an average human.
These companies clearly can identify the relevant traces for these business verticals which only strengthens the argument that NS was built on someone else's traces.
This seems like a very similar set of tools to Anthropic. But I’ve enjoyed using the various frontier models to criticize each other.
Using recursive loops, the output has gone from a high school level intern to a 2nd year lawyer in about a year. It still doesn’t beat the experts, but so much legal work is (legally significant) pedantry, not legal philosophy.
AI will not kill off lawyers, or reduce the amount of litigation. It will increase volume and velocity.
I have always felt like LLMs are uniquely suitable for legal work. I really have trouble believing we will have hardly any legal assistants and paralegals going forward when these LLMs are so incredible at spotting issues with arguments, figuring out citations, and doing semantic search.
Can someone explain to me the economics of AI and how it intersects with billing by the hour?
I think there would be a strong incentive not to use tools that speed up your work because you'd effectively be able to bill less time?
I'm sure there are some firms out there with more work than people, but still wouldn't it be more effective to hire another human who can then bill at a high rate for many hours?
Just like breaking crypto in the age of cloud is more about cost than time, this will lead to legal attacks based on the same principle. The biggest wallet wins.
This was already always the case. If anything, making this more accessible will reduce the barrier to entry for whether or not it's worth your time to take on a case. Instead of 50 lawyers spending 100s of hours on a case, you can have 1 or 2 lawyers + Astra working on it and if there's a case you can add more real lawyers.
With regard to certain legal questions this has always been the case. AT&T Fought the US Government for 20 years and eventually won because the government gave up. Without some kind of national anti-SLAPP law we're all one irritated oligarch away from having our lives financially ruined.
I am curious what level of trust established law firms treat LLMs with.
So much for caring about the spirit of the law. Now we'll start an arms race for abusing every possible letter of the law.
It's analogous to crypto. Started from some noble anti-authoritarian ideas and morphed into machine that removes any friction for capital - whoever has the most money will keep gaining the most.
Now there will be even less friction to horribly abuse the law. To democratize the law we would need to move in the opposite direction - to always keep it simple and aligned with our intuitions.
The more intricate and complex legal arguments become, the more abstracted they are from their original purpose and spirit.
Hence the crypto analogy - it was also supposed to "democratize", but the opposite happaned - it only further empowered the most powerful. Imagine legal case so purposefully complex that only those with access to best models have chances to participate and win the dispute.
I find the watermarking dynamic to be really interesting in the legal space, as more large model providers provide increasingly powerful legal capabilities, and adoption (presumably) also increases. Attorneys aren’t the same as developers as their work can be traced back to them, and there are personal bar licenses and reputations at stake. I wonder if knowing the likelihood that AI generated something helps or hurts in that respect.
I also see a lot of watermark removal services popping up as a result.
The most interesting use case in my mind is skipping law suits. Obviously you need lawyers in court. But lawyers are people you are basically paying to fight for you.
Instead, if you resolve your dispute outside of court, you don’t need a lawyer. If both parties use ChatGPT to find the relevant laws or read contracts, they could come to an agreement without expensive legal fees.
There is a lot of stuff here that I don't understand, but the concept of law firms giving user reviews is quite funny to me. Those reviews are going to be the most non-legally binding reviews ever written lol.
"Felt like a significant step toward legal-focused AI."
"Showed strength across key aspects of legal research."
I do believe entry-level paralegals will be made obsolete. In the grand scheme, perhaps it reduces overall cost of legal assistance, which is a net benefit for society.
Since the cost of building software is now cheap, there is nothing stopping them from building everything imaginable. They'll soon have an app store with every app built by them and they'll say its for security reasons. Nothing is stopping this coming monopoly
At a certain point why would they sell anything other than services and products their eventual (actual) AGI/ASI builds in literally every market.
When opportunity cost isn't a thing anymore because it reaches every corner of the planet simultaneously faster and builds better than any human can.
There's no reason to let others build on top of AI, except if the AI determines that it needs capitalism to continue because it's paperclip goal is to maximize shareholder value.
Context compaction? I noticed llm seem to forget partially or completely the original task when context compaction happens. The problem is more serious with local llm with low context size.
I wonder if this is willful sabotage on the part of the model. In other words, if you ask the model to craft a defense for a morally questionable case, will the model execute the defense in good faith? Or will it apply a training or system prompt bias in subtle ways?
For law, then medicine, mathematics, physics, etc. I believe LASSI Local Artificial Super Specialized Intelligence is the future, just before it becomes GODD General Omniscient Distributed Daemon
We need a term for the dark pattern of zooming into just that part of the y-axis where the two closely competing benchmarks sit, to make the top one appear maximally better.
Interesting to see the callout to companies like harvey in the post itself as consumers rather than competitors? I guess openai isn't quite willing to step into those customer relations themselves?
I've had top SV lawfirms whos partner charged our company $2000/hr and still couldn't get the right docs in the signature packet. and another getting share counts wrong during raise.
frustrating that law firms have no liability for these mistakes
no, but they are responsible for all work being done under them. What is the point of paying a partner $2000/hr if the work of their subordinates is wrong?
Most interactions with big tech firms involve 4-5 people so a basic phone call is $5k-10k. It shouldn't be unreasonable to expect after paying $80k for a financing round that they issue the right docs to the right people.
Everyone involved is about to get a swift and thorough introduction to the world corporate contract lawyers. But don't worry, they're extremely sympathetic and understanding when you flub a multi-million dollar corporate contract and will happily refund that money.
I really hope the bar associations continues to hold lawyers to high standards but I have feeling they may not be ready to handle fallout of AI slop-law.
Wouldn’t this and similar efforts to centralize bureaucracy make AI the new gatekeeper? Without reliable transparent models we’re just trusting OpenAI instead of a hundred top legal firms.
People that are saying OpenAI is screwed because a lack of profit, I'm not of that opinion. They are encroaching on every industry they can. They have name brand recognition, a huge user base and are showing they can be a valuable tool to all types of businesses.
As much as I hate to see it. They are now threatening industries like Engineers, Game Developers, Accountants, 3D modelers, 3D animators, Video Production, Audio Production, Therapist, Tax Auditors, Journalists, Authors, Artists, Mathematicians, Product managers, Every type of analyst and pretty much any other job that can be done behind a computer screen.
I used ChatGPT the other day to resink a CPU with termal paste, replace a PSU in my PC and snake my kitchen sink from the wall. I didn't exactly need it's help but wanted someone looking over my shoulder so to speak. It can help you repair all types of stuff, but as far as county / building codes and such, that probably isn't far off. It seems to understand things quite well.
By that point money ceases to have value, because the value of money comes from the motivation it gives people to work. If AI does everything, then money is useless (unless AI like money for some reason).
there are always be next frontier of problems which require creativity, unless AI become supersmart completely make humans redundant in all cognitive functions.
Most HNers are clueless that if you have Top Talent + Capital you already have an insurmountable moat. OpenAI, SpaceX, Anthropic all have that and none of the regular guys can compete against them (if they choose to attack that industry)
how about "just turn off your company if you're so worried about AI instead of immediately suggesting we use it to judge people's lives and livelihoods"?
this essay details how law firms became sweatshops from 80s. They charge hundreds of dollars to do make busy work by the junior most staff. LLM will kill the goldengoose of the law industry.
Will these models eventually replace all knowledge work, leaving lawyers, doctors, product managers, software developers, and others out of a job?
If the benefits were shared across humanity, that could bring us closer to utopia. My worry is that we’ll instead end up with a handful of even wealthier billionaires and millions of people out of work.
> Will these models eventually replace all knowledge work, leaving lawyers, doctors, product managers, software developers, and others out of a job?
Effectively yes, in the current forms. Those professions will likely evolve, but the traditional forms (ie writing code by hand, writing law filings by hand etc) are all dead.
in that case, legal cases would just come down to who has more compute lol. Many times cases win on their merits, but we've also seen evidence where overwhelming legal pressure can influence cases.
If we have enough energy and raw materials to keep building, yes, it will be an utopia made real. But if there is energy scarcity, then other two outcomes can arise.
Doubt it. When someone can go to Astra MD for 75% of what they used to go to the doctor for, then the remaining doctors only have 25% as many visits. When doctors only have 25% as many visits, they have to compete on price and they make less per visit.
Same argument for plumbers. Everyone always jokes about what a good time it is to be a plumber. But what happens when all the software engineers turn to plumbing? Suddenly it's not such a good time to be a plumber anymore.
I mean that's what all of these execs are openly telling everyone: they want you out of work, they want their ai to be the one to bring the world to it's knees, they want to surveille every second of your day, they want killer drones to use, they want to lay all of your cities to rubble and build "paradises" on top of them like in gaza.
They also openly tell you what they are afraid of btw: collective worker power. something that is massively lacking in our industry, although i feel like it would be one of the easiest industries to unionize in terms of # of workers.
Interesting interview I just watched about how powerful and dangerous these "wishes" or "prophecies" are especially in the hands of the ultra-wealthy: https://www.youtube.com/watch?v=eR7grHa1NR0
Perhaps, but then someone would do the same but bill you for 7 hours, someone else would undercut them again, until the price reaches a lower equilibrium.
The grifts continue... imagine something as consequential as Law being advertised as being solved by a statistical word generation engine that regularly gets basic things wrong. Anyone who isn't a lawyer won't know any better but you draft a single document of any appreciable detail and send it to an actual lawyer and it's littered with problems.
I've always said this will be when we get the real Butlerian Jihad, when the AI firms start trying to liquidate the legal profession.
If you automate lawyers out of a job, you can absolutely automate lawmakers out of jobs next. (Not that this would be a bad thing? Maybe pervasive agents for everyone can be the gateway drug to a "this time it's different!" workable direct democracy)
I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their firm involves taking in documents from healthcare plans and organizations, analyzing them for certain kinds of data, and then importing that data into an internal system they use to analyze and provide guidance on plans. The internal system can contain hundreds of documents for an individual client. All of the documents have the same information (roughly) but in totally diverse formats and styles. Once it's in the system, it's easy to compare and analyze across documents and the research process is much faster.
They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.
LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.
(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)
I’m curious how this increased throughput happens.
You’ve accurately stated that AI isn’t as rigorous as a trained attorney. Doesn’t that mean that every single datapoint must be confirmed by a human?
How is that quicker than just using a human to read the content and make the call? Data entry savings?
Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd.
It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).
Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.
Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.
Reviewing something takes less time than producing it.
You don’t need a trained attorney to schematize data. The LLMs are used to make the data easier to understand and manipulate.
I'm a lawyer myself.
I see problems in LLMs doing research, making drafts, etc.
I see 3 kinds of potential users.
1. Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
2. Fresh out of law school graduates.
They think they can do fast work, draft quick replies, do fast case research, enh. They should go the traditional route of interning, learning the "art", they probably won't need it because they will learn "how the law works"
3. Established attorneys. They usually have templates made for most things, over their career, they have established routines of making things fast, maybe it can help them but to say replacing good chunk of work or offloading to LLMs isn't probably going to go well.
There is a reason why interns are made to do grunt work. The "chain of command" is built so they learn on basic stuff, learn to make decisions, etc. Without these basic decisions under their belt, an intern can't take bigger decisions later on. They just won't know....
>Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
I used LLMs to replace a lawyer this year to great effect. I successfully advised myself as to the right strategy and drafted an immigration petition in the correct language. Granted it wasn't 100% LLMs (I also supplemented it with "traditional" online research and wrote the letter myself using the LLM version as rough guidance), but still, I certainly would've gone to a real lawyer in 2024 for this stuff but didn't.
Professional advisory work can be divided into two types:
1. Once you know the rules of the game, the next move is immediately obvious.
2. Knowing the rules is not enough to know the next move -- that requires judgement and experience.
LLMs in the hands of someone smart and savvy can easily replace the first type of work.
> Non legal background people trying to avoid going to a lawyer. They should definitely NOT use an LLMs.
In my experience, it is incredibly hard, practically impossible, to find a lawyer who will even talk to you. I think this is a valid use case to get at least some understanding what the next steps might be and if it even makes sense to seek legal representation.
Might be true today, but I still feel pareto principle applies. These points also sound like gatekeeping a bit and I hope future AI versions prove that law is also essentially something that can be made as programmable as possible for most common answers if the underlying principles and constitution are organised such.
If there is no precedence for X happening when Y , Z, P, Q, R are special combinations , the judge decides and sets precedence. But if another judge interprets it differently with similar set of circumstances it is not considered incorrect I presume especially if in different jurisdiction unless overridden by a higher court. This is my layman understanding but I think this design is wrong and essentially where human subjectivity controls the system.
Subjectivity is not totally avoidable in law but the goal of the legal system should be reduce it as much as possible along with ambiguity.
I do think non legal background folks can use LLMs today to sense check legal ideas, like for instance, "what are my legal rights in this situation?"
These situations usually are not ones that an individual can justify the time or money to contact an actual lawyer, but then if they do decide to contact one they will come in with better questions and more of a sense of what they are expecting.
This is similar to medical. Should you use LLM to diagnose yourself, treat yourself with prescription drugs you buy from shady gray market online sellers? No. But you can use it very well to know when it's time to go to the doctor and what to ask.
>I see 3 kinds of potential users.
3 kinds of users that don't scare the shit out of you.
My buddy works with lots of folks that have gone all in on Harvey. There are other kids of users.
There are 3x as many lawyers per capita in the US as we had in the 1970s. Credit agreements that used to be 50 pages are now 500. This increase in volume has not lead to any increase in actual value. The legal profession is a metastatic cancer on our society. Un-fuck this situation and maybe I'll think about taking your advice of not using a LLM.
My dog can review documents at an even faster rate.
You havent given any proofs or even comments that the work is the same level of quality or accuracy.
The proof would be that the attorney who did the work before, and who still reviews all ingested data, says it is.
I'm thinking back to some of my interactions with corporate lawyers and you know what? My bar (<- pun) for trusting Astra more than a lawyer is pretty low.
You should free your dog then because a creature that intelligent should not be anyone's pet.
Lawyer here (non practicing so to be clear none of this affects me):
most comments I read here don't seem to realize that different areas of law have very very different economic models and don't even mention which one they think will be affected or why, they just sort of lump it all together.
For example: It is highly unlikely llms will have any meaningful effect on high value personal injury law - I don't see a 5 million dollar case being handed to an LLM when the majority of the cost is in trial aids and not even lawyers. It may affect where and how they advertise. It may affect how they work. But it seems really unlikely to put any of them out of business any time soon by people doing it themselves.
Will it affect other areas more? Maybe. Probably? But so far I haven't seen a ton of comments that make specific enough arguments that they could really be debated or responded to effectively with a useful opinion
> them out of business any time soon
No. It'll be like software. Entry level employment will be affected. You wont want or need associate attorneys when you can hire a brilliant AI associate for 1/10th the price.
> You wont want or need associate attorneys
In commercial law, the business model is to rent the associates out to clients at rates higher than you pay (d/t your supervision). No client will pay $350/hour for your AI.
And for direct billing, it's not your interest to do things faster for the client (at risk of making a mistake).
So the real question is whether these pay-by-the-hour models will die as a result of competition from fixed-price transactions made viable via automation. I think not because the entire difficulty of legal work is that it's a complex bundle of issues until you sort them out; once sorted, there's really little to do other than execute (read: paralegals).
> No. It'll be like software. Entry level employment will be affected.
Whether everyone at the labs knows it or not, this it what they are doing to entire industries, starting with software: Embrace, extend, and extinguish
https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
What a time to be alive!!?
It'll be like software. The demand for both software and legal services is effectively infinite. If it becomes cheaper to file lawsuits then we'll get more lawsuits, not the same number of lawsuits at a lower price.
There will still be jobs for competent associate attorneys to prompt LLMs and review output. But frankly a lot of attorneys — especially many of those who attended "Third Tier Toilet" (TTT) law schools — were always a waste of money and should have never entered the profession in the first place.
Can AI replace divorce lawyers? Marriage has no friction, but divorce is high friction. Made impossibly expensive and time consuming because of the cartel of divorce industry.
There are hundreds if not thousands of divorce lawyers. How is it a “cartel?”
Divorce is expensive because divorces are acrimonious and people in that situation would rather spend the assets fucking each other over than letting the other person get them.
AI will likely replace divorce lawyers (or, at least make existing divorce lawyers much more efficient) - for non-contested divorces that are amicable. Roughly 95% of what a divorce lawyer does in that context is entirely automatable.
Divorce is extremely cheap if you both agree on everything. Its literally one of the options on the divorce filing. You just pay the court fees. The problem is both parties rarely agree on everything and that's when it gets expensive.
There’s an argument to be made if divorce was easier than more people might get married when they are less serious. My view though is make it harder to get married and easier to get divorced.
"Marriage has no friction" is only something that could be written by someone who was never married.
It's not just the divorce lawyers. We have 3x as many lawyers per capita in the US as in the 1970s. The system is designed by lawyers for lawyers. They will create more bureaucracy and more procedure to consume any surplus.
> Made impossibly expensive
huh? Many/most are no-fault jurisdictions with strict rules on dividing community property. Divorce is easy and cheap, unless it's hard to find assets.
> the cartel of divorce industry
So like a group of lawyers among the "divorce industry" who set prices for divorce and stuff? or wtf do you mean?
Personal injury attorney. AI is entrenched of every single phase of litigation, from the busywork of getting records, research, developing case theory and themes, expert selection, expert collaboration, negotiating, drafting demands, drafting pleadings, drafting discovery, drafting discovery responses, deposition prep, motion practice, jury selection, voir dire, opening statements, witness direct outlines, witness cross outlines, closing statements, and appeals.
Yeah, this matches my friends - they use LLM's for tons of things, using Eve and such. But nobody is replacing the lawyers themselves in cases of any value.
I also have friends in the world of high-volume personal injury (IE the morgan and morgan's of the world) but they also aren't being replaced by LLMs so far.
I'd love to get your thoughts on the current frontier models for law, or a link to anything you've written on the subject. Where do you think they're at?
LLMs will never replace human experts.
The word “never” is a dangerous word. In my lifetime I have seen it many times where something was supposed to “never” happen only to see it happen a few years later.
I'll buy that - but, how often does one need an honest-to-goodness expert? 99% of the time most people's requirements can be satisfied with general-competence, and, while the AI systems we have today are mostly only generally competent in porting code from one language to another - they are getting consistently better. I expect we'll start to see general competence in many more areas in the next few years.
LLMs are replacing human experts
There's a huge problem on here - many people stepping outside of their domains of expertise with surface level knowledge.
Its teh same reason claude for finance hasn't turned the finance world upside down.
This is getting tiresome seriously. Why wont these geeks learn some lessons?
Yeah geeks just stop trying! Seriously? Yeah, we’re in a bubble and narratives are ahead of reality but if you really don’t think AI is and will continue to eat knowledge work, just keep making your buggy whips.
Many people stepping in their domains of expertise with surface level knowledge
i agree with this, maybe.
The need for actual lawyers will persist I think from my own experience. I attempted drafting a contract with some points myself using AI, but after several edits I wasn't sure if it was correct. Sending it to an actual lawyer ended up in so many corrections I couldn't imagine the first time. One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.
I'd correct this to "The need for actual GOOD lawyers will persist I think from my own experience."
I have worked with many lawyers in my career. The bad ones will look at a contract and tell you fifty things that can be improved. Meanwhile, you take the same contract to a really great lawyer and they tell you that's it's not worth fighting over the changes and to just keep it as is. It's kind of like how a beginner or intermediate software engineer might tell you how you can build out a k8s or auto-scaling fargate cluster for your project and a really senior engineer might look at the same thing and explain that it's not worth worrying about scalability right now.
I find that when I feed contracts to ChatGPT (which I do all the time) it will try to find 50 things wrong with it. For example, as a test, i fed chatgpt an employment offer from a US government agency. It came back with 1152 words and 14 recommended (some of them substantial) changes. Meanwhile, any half-decent employment attorney would probably laugh at me if I asked them to redline an employment offer from the State Department.
I’ve found that good lawyers are about getting things done. You could nitpick the terms of a contract, but would you rather buy the house or have a perfect contract? Once I realized that, my view of lawyers changed (for the better).
YAGNI isn't just for software.
In my recent experience, it seems like LLMs may even increase need/demand for actual lawyers (more ideas + more words = more lawyer time). I expect lawyers will generally adopt the technologies that will benefit them, rather than the technologies that would increase standardization and trust
The same has happened with modern finance. Despite more technology that has power to drive more transparent/efficient markets, we have less perfect information sharing and a larger group of middlemen capturing a growing share of profits of the economy
I want to take a step back and observe that you're saying expensive specialists will continue to be required if two parties want to make an agreement. That seems like something worth fixing rather than just accepting.
> I want to take a step back and observe that you're saying expensive specialists will continue to be required if two parties want to make an agreement.
A lot of contract-lawyering involves getting ahead of when things go south and the two previously agreeing parties find themselves in disagreement.
The problem is those two people are in conflict not collaboration.
If the expensive person is still better at getting what _you_ want from a contract then it’s always worth doing that.
There’s a reason lawyers are so expensive to begin with - they do very complicated stuff with very expensive consequences if done badly.
Millions if not billions of agreements are made every day without the involvement of lawyers. Lawyers are only needed for particularly complex situations and even there they are generally only "needed" in the sense that parties feel they would add value.
In my experience, those expensive specialists help me protect against things me and the counterparty cannot even imagine when making an agreement. Do you mean that this would be fixed by AI or by making simpler agreements?
How?
Contract law has evolved because transfering the ownership of things is fraught with difficulty. Ownership as a concept itself is notoriously hard to define.
Consider housing - when you buy a house you own that land. Except if its an apartment, you just own the internal walls. Or when the government aquires your land through eminent domain. Or mineral rights - which don't belong to you, you just own the top 5 feet of topsoil. Or if you neglect a property and some squats in it. Or if you were given a fraudulent payment, and they take back the property.
Just getting people to agree is notoriously difficult when people have different often conflicting desires.
Not trying to be dismissive here but a lawyer will always edit your proposal. I do think lawyers will persist but not as many. And they will work very differently, much like we already use claude / codex to code - a big part of the contract probably won't be read by the lawyer.
I would rather argue they SPECIFICALLY will read everything. However, they'll likely often just be like "I'd phrase this differently but that works, too".
Also, unlike SW engineers, lawyers control their industry and will quickly circle the wagons to protect themselves. Paralegals are probably screwed, but lawyers will do fine. I'm considering law as my post-ai-SWE-apocolypse career path.
I’m thinking asking the same lines, but also looking into research / lab chemist so I can hang out with my best friend. If im going to change careers so drastically, I might as well enjoy the company.
LLMs are the first genuinely useful legal tech since the Internet. I'm pretty shocked, though, at the delta between how competent Claude is on code versus legal work. It's good for research and data organization, but terrible for drafting. I wonder if this is a structural problem with the lack of feedback loops. In law, there's no compiler to check for logical or continuity errors in your brief, and there's no unit tests to check for correctness or performance.
Even without that, I think it'll be extremely valuable to clients to allow them to answer simple questions without a lawyer, figure out the lay of the land so they can supervise their counsel, etc.
You should know - for coding they make terrible mistakes as well.
But programmers have this concept of a "code review" where another person looks at the code to look for problems.
We use this same technique with our LLMs. Most mistakes are caught by having a second LLM look at it. Doesn't even need to be a different model - just make sure it has a different context.
> LLMs are the first genuinely useful legal tech since the Internet
That is an incredible statement that could not be further from the truth. Large scale adoption of email, searchable document databases like Westlaw, LexisNexis, PACER, etc.. , OCR Software, electronic signatures, and tons more have had a much more defineably positive impact on the legal profession since the internet came about.
I have found that it’s useful generally speaking to get the intent of contracts and red lines, but actual drafting I agree is where I lose all confidence. My guess is that the significance of the difference between using a word like “and“ or “or“ can be so meaningful that that level of nuance can often be lost. But I know nothing I’m not in the space, I just pay too much money for lawyers.
Law is certainly a less verifiable domain
Human professionals put their reputation and finances at risk when performing their work. This risk functions as a guarantee.
> Human professionals put their reputation and finances at risk when performing their work. This risk functions as a guarantee.
Do you know how difficult and costly it is to sue an attorney for malpractice?
Do they?
Not being snarky here but I've encountered some really crappy lawyers and they seem to be doing great ($$$).
It really sucks one cannot publish something like "this guy charged me $5k, wrote my name wrong and forgot important paperwork the day we showed up to court" as that could make you liable for damages. There should be a safe harbor for that.
I was in a similar situation and tried a different approach.
I had start with asking for a contact with some details I provided. But then I ask the model to be an experienced corporate lawyer and ask me a series of questions to gather the details it needed and then write the contract. The result covered a lot of details that were highly relevant but were absent from the original attempt.
The key insight is that you can lean on the model to cover your unknown unknowns.
Your comment is a great example of the phenomenon where people think AI is an expert in areas that are not because they have no way to determine if the seemingly good looking result is genius, gibberish, or somewhere in between.
> The result covered a lot of details that were highly relevant but were absent from the original attempt
Another one suckered by the plausibility engines.
Did you validate the output with a lawyer?
> The need for actual lawyers will persist..
I agree. But, we will need a lot fewer of them. My small company called our lawyer a lot more before the advent of AI. Mostly to clarify a few things or review simpler contracts. Now, for simple things, we do not call them at all.
Their work will get cheaper so it will become more accessible for people, increasing demand over all would be my guess to some degree at least.
> One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
I'm currently negotiating with someone who I feel is clearly using AI and this is EXACTLY what has been happening
> The need for actual lawyers will persist I think
But will their glamorous salaries persist? That is the question that matters.
AI doesn't need to wipe out lawyers. If they just depress salaries enough, virtually nobody is going to want to be a lawyer anymore.
The salaries are a result of the insane hours they have to do. Biglaw needs to pay those salaries to keep getting young associates in the door because burnout is a real problem in the industry. If salaries were lower and hours/stresses were lower, you would probably still find people wanting to be lawyers. A different kind of person, maybe.
(Not that I think AI will necessarily have that effect. We just don't know yet)
It’s a professional organization that only allows so many people to pass the bar per year so I don’t see that changing anytime soon… if anything if they need fewer lawyers they will just pass fewer people to keep their wages high
> The need for actual lawyers will persist I think from my own experience
This doesn’t replace lawyers, but paralegals surely will be affected. A good enough model could shrink the number of paralegals needed in a firm.
As costs for legal work drop we might expect demand for that work to increase. For example, it may be the case that legal help becomes accessible to entire segments of the population that had no access previously.
The best writer I know of was an associate attorney. He didn't have the technical background nor the in depth computer related knowledge, and relying on the information I fed him. But man the briefs he filed to the court were amazingly good. Reading them I would have been convinced his side was right if I were in the jury.
Law LLM will surely help competent lawyers in their fields with greater sources of knowledge not in their core area of expertise.
I don't want to imagine a world where juries were responsible for reading briefs and making decisions.
Sure as long as they stop making things up.
And to add to your last point yes it should also be like code: where the costs of development are coming down. Legal costs should trend down if law firms are effectively leveraging LLMs to accellerate their procedures (though I won't hold my breath)
> The need for actual lawyers will persist I think from my own experience.
The outcome of a case shouldn't depend on someone's fallible ability to recall facts or convince other people or point their index finger*
Law should generally be deterministic. One's CHA stat should have no bearing on justice.
There should still be human judges, but the middleman between the judge and petitioner could easily be removed, and have generally been seen as leeches since forever anyway.
Though, like how the USA opts to remain in the Stone Age with regard to tax filing because of lobbying by tax software companies, this faction of society will flail the hardest before they admit they're obsolete.
----
* What's a lawyer's favorite programming language? Objection C.
The ability to clearly and effectively communicate, and the ability to know and recall the facts of a case is always going to be important. It'd probably be a more fair system if judges and jurors couldn't see the attorneys or even the people they're representing. Now that many cases are handled over video calls it'd be pretty easy to make that change.
A lot of communication is conveyed through more than the words themselves though and even though it's often misinterpreted people put a lot of value in that information. I suspect that even if we took away the ability to see the people involved judges and jurors would still find bias in the tone/quality of people's voices. Even if everything was reduced to text they'd find bias based on word choices, phrasing, spelling, and grammar.
Yep, just yesterday, Nike removed one leach middlemen called retailers and sell directly through their app. Turns out working out very well for them too.
The law is much more interpretation and argument based than logic routing based than you seem to believe.
Lawyers sure. Paralegals and assistants are cooked in the way typist and secretaries were back in the day.
Nope. Paralegals and assistants do the type of thing GP hired a lawyer for (and lawyers do, as well).
Lawyers can practice, can represent you in court, etc. that's not going away soon.
The typical pattern is called “deskilling”. It doesn’t usually mean a skilled profession will disappear overnight. Instead, the job might be done by less expensive folks like paralegals.
An example is in the banking industry, where making a loan used to require deep analysis of a person’s credit worthiness. Now they use an algorithm (credit scores) which means someone with less experience can do it.
If law follows the same pattern, a job done by someone making $500/hour might be done by someone making $50/hour.
If you think paralegals + AI can do the job of good lawyers you don't understand the profession at all.
"law" and those who pratice "law" have managed to place itself largly in a monopoly. This is a huge drain on the economy. Law is expensive and irrational.
He just gad better prompt
What model did you use?
It's worth noting that the announcement seems to aim the product at lawyers or students rather than laymen.
Our entire world is about reducing the friction of conducting a transaction
How to fix the lack of trust which is needed to conduct a transaction
Contracts affix a moment of trust in time, so even when trust is lost between parties the contract has guardrails for behavior and consequences for acting outside those rails
There are many transactions that are completely inaccessible to most people because the need for a contract or lawyer is too high
LLM use allows for those inaccessible transactions to decrease, and get people further along to the point where a lawyer is accessible as well, and economically viable to use
> One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.
>Its just like code I suppose, if you can read and understand and validate
I share this, when I asked an attorney for a contract, they were much simpler when compared to self redacted or AI redacted contract, it was as if I were rediscovering lots of concepts and I HAD to put them in paper, even if they were defaults and were essentially no-ops.
Comparable to:
> Mkdir $PWD/path
instead of
> mkdir path
or conditions so rare that they don't merit including.
> x = 0
> if x!=0:
> raise Exception("Bit flip by solar rays or otherwise")
>The need for actual lawyers will persist I think from my own experience
At the very least because the attorney monopoly assigns a 5ish year of training as PoW and natural personhood as Identity as protection for Sybil Attacks.
If anything, the exclusive right of attorneys to represent clients in court, file motions, and enjoy professional secret, is more valuable, as it's not something AI can ever compete at. (Barring a wild recognition of machines as humans, or an overturn of the impossibility of companies to appear in court.)
It’s a bad day to be a lawslop company. When you’re reliant on other companies to do all of the AI part of your AI product, they can just train on your traffic and eventually eat you.
Second paragraph:
> API customers including Harvey and Legora will be able to build on Astra for Law, bringing this intelligence into their own products and workflows.
In other words: "no, no, we're not eating our children to prep for the IPO. Don't worry."
Max Junestrand has consistently said Legora treats the model layer as swappable, selecting across frontier providers rather than building the product around one.
OpenAI didn't need to name Legora and Harvey in the second paragraph of the launch post.
They are pre-empting the obvious interpretation of Astra for Law: that moving this far up the legal stack puts them in direct competition with their biggest legal AI customers.
“Don't worry, they can build on us” is a pretty conspicuous message to include on launch day.
They have clearly thought about some pessimistic outcomes.
Switching from ChatGPT Enterprise to Legora at my firm was a godsend, it's so much better for legal work, even with the frequent changes to the underlying models.
>Max Junestrand has consistently said Legora treats the model layer as swappable, selecting across frontier providers rather than building the product around one.
Vendor-neutrality for LLMs is such a weak thesis all around, whether for providers or consumers. It weakens the product by being promiscuous and gains no material benefit at all.
LLMs are magic byte(byte) functions, it doesn't make sense to say "we have different providers for magic".
"We are also expanding our work on privacy and governance to give law firms specific controls for confidential client work"
This is everything OpenAI have to say about privacy in this announcement. No guarantees. No promises. Just a pinky-swear promise.
Anyone trusting them–or a lawyer who relies on them–for legal work deserves what they get.
the thing is a lot of the legal work which will go through this is drafting 100 and 1 variations of draft versions of standard contracts not containing any trade secrets where the contract can be drafted with "replacement/place holder names"
the kind of work mostly done by juniors not yet through their final exam and other "non" lawyers etc.
so it's a slippery slope of "lets just use it for <this> things where it doesn't matter" and then out of laziness and convenience it creeps into all the other places (at least for drafts).
Lawyers are already llm’ing their work and getting slapped around in court for making shit up.
There’s another case making headlines every week.
I get the feeling a lot of them won’t care about this stuff.
If there is any cartel that deserves to be broken its legal search. May they die a miserable death.
Yeah, it should be freely available, you have to be able to know the rules you're supposed to obey in order to obey them well. I've been making a free API for US law search, you can point whatever model you want at it: https://law.agentlookups.ai/
Very much a work in progress, only federal and state so far, no municipal codes yet, and no case law yet. Big hole, I know. Also working on making the search ranking work better.
> By using the legal search index, Astra for Law can search U.S. case law, statutes, regulations, court rules, and administrative decisions across a corpus of more than 230 million URLs, with sources added daily. Our work with Free Law Project, the nonprofit behind CourtListener, brings its case-law collection covering more than 99.9% of published U.S. precedential case law (opens in a new window) into this research experience.
Found that to be very interesting
Let's swap it out for the AI cartel!
The courts are about to overrun with AI-generated lawsuits (even more so than they have been[0]).
[0]https://www.technologyreview.com/2026/06/04/1138391/courts-c...
I find the idea that people can use LLMs to exercise their rights as citizens appealing however. Many people aren't aware of the rights they have, and LLMs are pretty good at surfacing some stuff without having to pay lawyers. Having to hire a lawyer is imo actually a huge way of gatekeeping people from exercising their rights. I heard a lot of local German public institutions are currently being flooded with people arguing their case with the help of LLM that they previously weren't really realistically able to do. So I don't see it all as bad.
>are pretty good at surfacing some stuff without having to pay lawyers
They are also really good at making stuff up as evidenced by the many, many, many examples you read in the news about actual lawyers using AI to write briefs that are full of errors and hallucinations.
In civil courts, you'd likely get more sympathy from a judge if you represented yourself and admitted your lack of understanding, rather than try to appear as someone you're not because you wrote some prompts and copied the output.
Current court systems around the world are just not built to handle the flooding of cases from the citizens.
The stupidest analogy is open source projects having a hard time accepting LLM generated PRs from the masses, because review process is the bottleneck.
No idea how to fix this, to be honest. In coding world, with some mental gymnastics, I can see code not being reviewed by people anymore. In courts, things generally have more consequences, and you can’t really roll back decisions that easily.
you could probably write a cool little gotcha of an SF short story about a barren wasteland of a planet that keeps broadcasting out legalese that's revealed to just be LLM chatbot lawyers pedantically arguing with one another about xeno legal doctrine
Relevant:
https://www.ftrain.com/nanolaw
JG Ballard could probably turn that into something darkly atmospheric, and even more dystopic than his usual depopulated worlds...
The legal system, which is a machine/technology by itself, will be eaten out. I wonder what will replace it. Botnet law arbitrage? – Personal assistants constantly negotiating with each other to avoid permanent civil lawsuits?
the cost of making a legal argument can collapse while the cost of reaching an enforceable, legitimate decision may go higher, which will gate the "justice" system even more.
The legal system is slow and inefficient by design.
You presented the concern from my adjacent comment perfectly (“LLM performance: common law vs civil law” essentially). So is AI possibly just growing the “Reverence for Professional Experience” factor that plays such a big role for legal compensation here in the US?
I am not sure "eaten out" was the correct phrase to use here.
Call it a Prediction Market
They seem unaccountable. Every other profession has increased their productivity.
construction productivity is actually down since the 60s!
justice doesn't have to be productive it has to be fair
>Every other profession has increased their productivity.
You dropped this /s
Don’t you have fee when initiating a lawsuit ?
I don’t know in the USA but in France, if it’s deemed that you launched a lawsuit knowing very well it wouldn’t succeed, you are susceptible to get a 10k€ fine. Even jail in serious cases.
Yes. And you(r lawyer) can collect lawyer's fees and you can be made to pay the court fees, if you lose.
IME the American legal system is set-up to discourage litigation, though. A common tactic is to bury your opponent in the threat of heavy damages or jail-time to get them to settle for what you were originally after, which courts are perfectly happy to facilitate because it gets a potentially lengthy trial off their dockets. They'll punish (or be biased against) whichever party seems responsible for not accepting a "reasonable" settlement.
Algorithmic abuse of the system to extract payments already exists in the form of the debt collection industry.
The solution is obvious: AI judges! With complete immunity for model vendors of course, just in case some toddler is sentenced to death or something.
I wonder how many parties would prefer an AI judge over delayed justice.
Why the hell are they not fighting fire with fire? This is not sustainable. But it is annoying that the AI labs get to play arms dealer, selling to both sides.
Judges don't care if people have to wait months. What does it cost them?
How are we fighting it online? You cannot
It'll be like radar detector detector detectors, which were a thing for a while. Each side (cops, speeder) buying detectors to detect the detectors.
https://www.damiencharlotin.com/hallucinations/
> Astra for Law passed the evaluation’s overall correctness check on 54.0% of questions.
How is this a product that you are selling?
On the promise of potential cost savings by means of firing and/or not hiring as much.
Help me understand your consternation.
No word on model hallucinations in the blog post.
https://artificialanalysis.ai/models/gpt-6-astra?omniscience...
Not like common law doesnt already have a lot of hallucination going on.
can “penumbras and emanations” compete with hallucinations?
The benchmark you linked to shows GPT-6 Astra having the lowest hallucination rate of all tested models.
The blog post makes no mention. If the rate isn't 0% it should be mentioned for a field like this.
Does hallucination matter for this application? We've moved beyond raw recall being that important, it seems like for law specifically all relevant facts will be cited and checked easily by humans.
> it seems like for law specifically all relevant facts will be cited and checked easily by humans.
Don't be too sure about that. [0]
0: https://www.damiencharlotin.com/hallucinations/
> all relevant facts will be cited and checked easily by humans
I've talked to a lawyer about how they handle this. They do indeed double-check everything, since it'd be embarrassing (or worse) to send hallucinated statements to opposing council or to the court. They still find the assembly a huge time saver
But based on stories in the news on the subject, not everyone has this same level of diligence
Sure but just like generating 100x more code, someone has to review it. So you are wasting everyone in court's time (defendants, prosecutors, judges, staff) by making them parse through what is quite often a bunch of hallucinated slop. Time that could be much better spent on parties who prepared and reviews their own arguments.
The lawyers I know are very fixated on the problem of hallucinated case citations which is amusing to me as a onetime programmer, since case citations have a well-defined syntax and would be relatively easy to check programmatically.
"We have dangerous AGI that can destroy humanity."
"Also, all of your sensitive legal documents will be totally safe with us."
"Also, for some reason even though we have AGI and selling tokens is a fine business, we need to sell a new product specifically targeted at a very high margin and lucrative industry."
In an utopian society, lawyers are an unnecessary profession. Laws should be clear and simple so the common person can be their own "lawyer". LLMs help with that goal.
what profession you need in utopian society?
Poets.
How long before we start building detailed models of each judge trained on all of their legal output, and then test various legal theories against those judge models in virtual moot court? Craft each pitch to the legal idiosyncrasies of the batter. I assume that real lawyers do this routinely and could use a simulator.
I feel like this is on a different trajectory than what LLM-based tech does. This type of individualistic extrapolation is one of the things they're really bad at, in my experience.
This is already a thing to some extent. I don’t recall the products that do it, but judge profiles def exist and are taken into account .
What worries me is the step after this. If god forbid this proves successful and models accurately predict specific outcomes, people will start to ask whether the solution to AI slop lawsuits is to do the judging with AI too.
I'm not familiar with the Vals AI Legal Research Benchmark. But their website has other frontier models' scores, and the scores OpenAI is now revealing for "Astra for Law" are slightly less than Claude and Muse:
> The top is a three-way tie: Muse Spark 1.3 Max, Claude Opus 5, and Claude Fable 5.1 all reach 55.29% all-pass accuracy, a clear ~6-point step ahead of the next model. [Astra for Law reached 54.0%]
> Under partial-credit scoring, Claude Opus 5 reaches 90.58% weighted pass rate but 55.29% under strict all-pass grading, where every rubric check must pass. The gap shows models often get most of an answer right but fail on one or two required elements. [Astra for law reached 90.0%]
https://www.vals.ai/benchmarks/legal_research
Can you imagine the sort of corruption that is possible here? Like if you have a lawsuit against someone or some entity that openai or their investors have business with...
So OpenAI is partnering with Latham Watkins, Freshfields is partnering with Anthropic and Kleiner Perkins is building their own. It'll be interesting to see which wins out here, I don't see how those partnerships can end well for the law firms unless they're making an assumption they'll be sucked dry of USP but the revenue split from the AI labs will make up for it. Why would I pay a premium for Latham Watkins when every other firm can get their expertise and experience in a subscription, and add their own on top?
The most interesting part about this to me was how they bench/compare it, like in the example with Fable:
It made me wonder if a good deal of law is about finding a way to work in statements with clear precedents without your opposition noticing and then later drawing upon them in court (as settled precedents, in your favor) after the opposition (perhaps implicitly) accepted it. That would clarify a lot about why some lawyers need to spend so much time pouring over and memorizing past cases (even ones that are only tangentially related); because anything they miss could be used as a potential trojan horse by the opponent.If this is true that must mean there are a good deal of cases settled using precedent "gotchas" where both sides knew that without the "load-bearing" precedent the outcome would've definitely been the opposite. (i.e precedents almost always trump even valid arguments)
open-ai have proven they can make good / decent models but business strategy is just spray and pray.
they need to pick a lane and optimize for it. coz at their size they can't serve the application layer (a.i startups who can fine-tune models will eat their lunch)
if they gonna do a consumer play - then go ham on that.
otherwise they're gonna get caught in the dreaded middle valley.
They have a lot of compute, so I think it makes sense to spray and see what works. Anthropic is limited in that regard, and focused in coding/tech, but OpenAI don't need to do the same. They even got the lead without having to focus in coding only, which is remarkable.
Do they still do Atlas, the web browser? They are indeed throwing shit at the wall, seeing what stick.
a la Google.
They can't be seen to commit to strongly to a specific product experience, because if they are understood as a regular tech product business that has way different financial scaling considerations than "superintelligent everything-factory"
I suspect that half-assed announcements like this are a result of different people internally with conflicting incentives resulting in a split-the-baby solution.
Enterprise is eventually get caught (if it isn't already) by the Microsoft/Google. Because with office/teams/suite they were already in every enterprise, and that just added a new tool to existing offerings.
Provisioning and contracts and data retention was just an extension to review of existing ones.
Nobody serious is going to risk sending sensible data to OpenAI/Anthropic, etc because "the benchmarks have shown +8% performance there and +2% there". Irrelevant.
memes and snark aside, can you use this to create legitimate terms and contracts for my products and if I do who is getting sued when it is wrong?
> who is getting sued when it is wrong?
You. Don't take legal advice from a word calculator.
As opposed to taking advice from the glucose wetware?
The business value will be when platforms can underwrite their LLMs' legal products.
That will be a decacorn product or more.
The product isn’t meant for you or me it is for lawyers. If you can’t take on personal liability for a badly written contract then you shouldn’t be using it.
Solved without AI https://commonpaper.com
https://commonpaper.com/standards
https://commonpaper.com/product/ai-agent/
Same for SWE when pushing code they don't understand.
> can you use this to create legitimate terms
I don't think you can. What I get from this article is that this is not a product they're going to sell to average consumers.
I'd expect they could indemnify you against hallucinations or similar if this gets good enough for that to be a very rare occurrence? Or you could buy insurance on it that's cheaper than hiring a lawyer (not a high bar to clear). I wouldn't rely on it currently, though.
when you shoot someone with a gun, who gets sued? you? or the gun manufacturer?
You will be the one getting sued. It's just a tool.
I already use them for that, they are pretty excellent at it. Much better than the terms of use generator products that used to exist. That said, nobody cares to sue your business for the most part until you're big enough to be worth it. By that time, you'll have a team of legal analyst to assist you... or agents should I say.
Then again, nobody will have money to buy anything at this rate, so in all liklihood, this is a total non-issue.
You thought you could sue your lawyer when a contract they drafted turns out to not mean what you thought it meant?
You can create terms and contracts even if you're a 7 year old with no AI.
And yes, even contracts drafted for millions of $ have oversights and unlawful or unenforceable terms.
You are.
An important take-away from this is the harness is really important. The graphs show that the same model with a better harness performs many percentage points better. OpenAI are getting into the "selling the harness" game in a big way. Why?
What's interesting about that is while not everybody can train or run a model, anybody can build a harness. You and I can build harnesses.
It seems strange that OpenAI would move into a field where any developer can compete with them. I think that tells us a lot about the economics of training and selling inference.
I don't think it says what you imply. I think they're just trying to vertically integrate.
Harnesses are going to be controlled at companies eventually, just like you might not have a choice of OS. They want to make sure they are the complete package.
Yeah you're probably right, the purpose would be to lock in whole industry verticals.
Any lawyers here who have used AI agents heavily for their work? From what I've heard, they're currently very good at searching, analyzing and drafting documents like contracts and patents, but some say they suck at interpreting the law.
I’m sure there are a wide variety of experiences out there, but here’s my perspective as a former biglaw associate and current solo litigator:
I have had some success using frontier models from the last 6ish months, but only when I can break up my work into discrete and verifiable tasks. For example, I had ~15k pages of discovery I needed to dig through for a summary judgment motion. Instead of just asking Claude to find the best evidence, I asked it first to run a clean, high quality OCR pass (it was almost entirely PDFs). Then I had it generate embeddings and write some reusable python scripts to make keyword and semantic searching easy for agents. While I was writing the brief, I would routinely ask my agent (Claude Code) to use both keyword and semantic searching to find the best evidence supporting whatever assertion I was trying to make. I trusted it because there were traces I could follow.
In other cases/situations, I’ve tried just giving a model access to all the docs and saying “write a brief arguing X,” but it’s always terrible at this. It writes briefs with lots of evocative jargon and rhetorical flourish, but a low signal-to-noise ratio.
Again, I’m sure others’ experiences differ based on workflow, legal area, etc.
Agreed. Six months ago, it was basically a gloried grammarly.
But lately, I’ve been taking hints from the “company brain” models, where it develops a running model of the case, and assesses each new piece as it comes in and updates the file.
I’ve also been using “Ralph Wiggum”-type models where you pass letter or contract drafts back and forth between agents with different goals (rules compliance, grammar, conciseness, ai slop detector, an opposing counsel critic, etc.). After a few rounds, it’s not perfect — but I start with a very good first draft in my hands.
What are you using for semantic searching?
They are excellent, especially the latest models. That said, (a) I wouldn't feel safe filing something without a real lawyer looking at it; (b) it can't (easily? legally?) do oral arguments for you; and (c) a lot can happen in the hallways outside the courtroom to move a case forward that the AI can't easily do.
In my experience it basically doesn't even try to interpret the law. It just summarises the publicly available law/guidance out there and, if there is a question about how to interpret some provision, might set out the arguments for each interpretation. It doesn't really take a position. It is pretty good at drafting though. (Legora)
> It doesn't really take a position.
Interesting! In other unrelated domains, models seem more willing to take a position. It may be nuanced, but they do tend to take a stance. I think it's good that it leaves the interpretation to humans, but I wonder if this is also some sort of a guardrail to minimize liability...
Probably because the AI was also trained on HN comments.
worse, it was trained on reddit law comments
We do and it saves so much time. Of course human judgment is needed but it’s like cooking with someone else doing the mise en place.
What we contributed to Reddit and Arxiv voluntarily before, what we contribute as traces right now are all being used to build business verticals by OpenAI and Anthropic. All these business verticals are being used for is more trace collection which would only strengthen these models and render most humans useless because the ceiling for 1000 swarms of Agents to learn is much higher than an average human. These companies clearly can identify the relevant traces for these business verticals which only strengthens the argument that NS was built on someone else's traces.
I wonder if Lawslop is gonna become a mainstream expression. Anyone got a better term?
Slawp?
Haha clever
Drop the s and p. Just law.
It’s cleaner.
law-bearing
What do you mean by “lawslop”?
You know exactly what they mean. But if somehow you miraculously don't there are hundreds of stories like this:
https://www.reuters.com/legal/litigation/lawyer-state-farm-f...
i hope that someone can come up with something catchier than "thing-slop" or "slop-thing". the word has become basically meaningless from overuse.
I love the slop word, to me it means effortless, average.
People confuse slop with "bad", but slop isn't bad per se, it only becomes bad when real effort was required.
Slop?
The biggest problem with this is billable hours. Faster work means less billable hours for attorneys.
what is going on here?
this vertigoruntime is brand new and his submissions absolutely dominate the front page recently https://news.ycombinator.com/submitted?id=vertigoruntime
the about link https://vertigo.kuber.studio is obvious AI llm spam
Email them...?
I imagine the point here is to separate out the APIs by different professions and charge accordingly.
This seems like a very similar set of tools to Anthropic. But I’ve enjoyed using the various frontier models to criticize each other.
Using recursive loops, the output has gone from a high school level intern to a 2nd year lawyer in about a year. It still doesn’t beat the experts, but so much legal work is (legally significant) pedantry, not legal philosophy.
AI will not kill off lawyers, or reduce the amount of litigation. It will increase volume and velocity.
I have always felt like LLMs are uniquely suitable for legal work. I really have trouble believing we will have hardly any legal assistants and paralegals going forward when these LLMs are so incredible at spotting issues with arguments, figuring out citations, and doing semantic search.
Can someone explain to me the economics of AI and how it intersects with billing by the hour?
I think there would be a strong incentive not to use tools that speed up your work because you'd effectively be able to bill less time?
I'm sure there are some firms out there with more work than people, but still wouldn't it be more effective to hire another human who can then bill at a high rate for many hours?
Seems like another “product” that will be killed in 6 months, but is good for the IPO so they can say they solved law.
"Rogue OpenAI agent swarm accidentally overturns the Civil Rights Act"
followed a month later by
"Anthropic's Claude inadvertently repeals the 19th amendment"
Just like breaking crypto in the age of cloud is more about cost than time, this will lead to legal attacks based on the same principle. The biggest wallet wins.
> The biggest wallet wins
This was already always the case. If anything, making this more accessible will reduce the barrier to entry for whether or not it's worth your time to take on a case. Instead of 50 lawyers spending 100s of hours on a case, you can have 1 or 2 lawyers + Astra working on it and if there's a case you can add more real lawyers.
There are other possibilities:
1) Lawyers are not as naive as software engineers and will fight being replaces by new laws.
2) If they are replaced, OpenAI will take a cut commensurate with the amount in dispute (OAI, please credit me for the idea in the IPO brochure).
That’s how the legal system in the US has always worked though
With regard to certain legal questions this has always been the case. AT&T Fought the US Government for 20 years and eventually won because the government gave up. Without some kind of national anti-SLAPP law we're all one irritated oligarch away from having our lives financially ruined.
I am curious what level of trust established law firms treat LLMs with.
That's already the case in law because you could just hire the best or most lawyers.
So much for caring about the spirit of the law. Now we'll start an arms race for abusing every possible letter of the law.
It's analogous to crypto. Started from some noble anti-authoritarian ideas and morphed into machine that removes any friction for capital - whoever has the most money will keep gaining the most.
Why do you think LLM's language abilities are unable to understand the spirit of the law?
Do you really think the law isn't already horribly abused? Democratization of law has been needed for a thousand years.
Now there will be even less friction to horribly abuse the law. To democratize the law we would need to move in the opposite direction - to always keep it simple and aligned with our intuitions. The more intricate and complex legal arguments become, the more abstracted they are from their original purpose and spirit.
Hence the crypto analogy - it was also supposed to "democratize", but the opposite happaned - it only further empowered the most powerful. Imagine legal case so purposefully complex that only those with access to best models have chances to participate and win the dispute.
Maybe I’m misunderstanding, but isn’t that what legislators do?
I find the watermarking dynamic to be really interesting in the legal space, as more large model providers provide increasingly powerful legal capabilities, and adoption (presumably) also increases. Attorneys aren’t the same as developers as their work can be traced back to them, and there are personal bar licenses and reputations at stake. I wonder if knowing the likelihood that AI generated something helps or hurts in that respect.
I also see a lot of watermark removal services popping up as a result.
The most interesting use case in my mind is skipping law suits. Obviously you need lawyers in court. But lawyers are people you are basically paying to fight for you.
Instead, if you resolve your dispute outside of court, you don’t need a lawyer. If both parties use ChatGPT to find the relevant laws or read contracts, they could come to an agreement without expensive legal fees.
except for those pesky things called rights
There is a lot of stuff here that I don't understand, but the concept of law firms giving user reviews is quite funny to me. Those reviews are going to be the most non-legally binding reviews ever written lol.
"Felt like a significant step toward legal-focused AI."
"Showed strength across key aspects of legal research."
I do believe entry-level paralegals will be made obsolete. In the grand scheme, perhaps it reduces overall cost of legal assistance, which is a net benefit for society.
Since the cost of building software is now cheap, there is nothing stopping them from building everything imaginable. They'll soon have an app store with every app built by them and they'll say its for security reasons. Nothing is stopping this coming monopoly
At a certain point why would they sell anything other than services and products their eventual (actual) AGI/ASI builds in literally every market.
When opportunity cost isn't a thing anymore because it reaches every corner of the planet simultaneously faster and builds better than any human can.
There's no reason to let others build on top of AI, except if the AI determines that it needs capitalism to continue because it's paperclip goal is to maximize shareholder value.
most of the complexity in software isn't in the actual code
Will it be able to sue itself?
That’s a great question for Astra for Law.
From my experience LLMs seem to forget sometimes who they are representing when drafting clauses or editing / redlining.
Our counsel made a few edits where it clearly drafted in favor of the customer instead of us.
Context compaction? I noticed llm seem to forget partially or completely the original task when context compaction happens. The problem is more serious with local llm with low context size.
I wonder if this is willful sabotage on the part of the model. In other words, if you ask the model to craft a defense for a morally questionable case, will the model execute the defense in good faith? Or will it apply a training or system prompt bias in subtle ways?
Surprised they published this without mention of jurisdiction. Each country has their own laws.
Would have been prudent to highlight that this is for (presumably) US law
For law, then medicine, mathematics, physics, etc. I believe LASSI Local Artificial Super Specialized Intelligence is the future, just before it becomes GODD General Omniscient Distributed Daemon
Post AI era view on anything can be categorized as “What can go wrong ?”
We need a term for the dark pattern of zooming into just that part of the y-axis where the two closely competing benchmarks sit, to make the top one appear maximally better.
Truncating the Y axis or cropping the Y axis
crop-maxxing
Interesting to see the callout to companies like harvey in the post itself as consumers rather than competitors? I guess openai isn't quite willing to step into those customer relations themselves?
So there's a strategy shift here. They launched financial services specific tools and now law?
Is the play here a set of specialized harnesses using their best general model?
The end goal should be AI judges, I think China has implemented that to some degree.
Espn has an article about Lane Kiffin using this for his legal advice recently.
He almost singlehandedly killed LSUs football program until people were able to convince him to talk to a real lawyer.
This is the first blog post I saw OpenAI call out Claude directly like that.
I've had top SV lawfirms whos partner charged our company $2000/hr and still couldn't get the right docs in the signature packet. and another getting share counts wrong during raise.
frustrating that law firms have no liability for these mistakes
I welcome ai law
the partner shouldn't be doing sig packets
no, but they are responsible for all work being done under them. What is the point of paying a partner $2000/hr if the work of their subordinates is wrong?
Most interactions with big tech firms involve 4-5 people so a basic phone call is $5k-10k. It shouldn't be unreasonable to expect after paying $80k for a financing round that they issue the right docs to the right people.
Astra isn't doing well on bullshit Benchmark https://petergpt.github.io/bullshit-benchmark/viewer/index.n... and is even worse in Legal department.
Qwen 3.8 Max and Opus 4.8 score highest.
Revealing how they call Harvey and Legora customers instead of partners...
yes, law decision depends on human
yes, most of the time i use Astra Low already.
Went from AI replacing my job as an attorney, to them begging me to use their tools
This is really cool.
Everyone involved is about to get a swift and thorough introduction to the world corporate contract lawyers. But don't worry, they're extremely sympathetic and understanding when you flub a multi-million dollar corporate contract and will happily refund that money.
As a user of ai hooked up to my Gmail I assume I’ve lost all privilege
I really hope the bar associations continues to hold lawyers to high standards but I have feeling they may not be ready to handle fallout of AI slop-law.
I said this before but I wonder if Dan Kan will reboot Atrium. Rally up some old partners and hope Anthropic buys them out for a couple billion.
Wouldn’t this and similar efforts to centralize bureaucracy make AI the new gatekeeper? Without reliable transparent models we’re just trusting OpenAI instead of a hundred top legal firms.
People that are saying OpenAI is screwed because a lack of profit, I'm not of that opinion. They are encroaching on every industry they can. They have name brand recognition, a huge user base and are showing they can be a valuable tool to all types of businesses.
As much as I hate to see it. They are now threatening industries like Engineers, Game Developers, Accountants, 3D modelers, 3D animators, Video Production, Audio Production, Therapist, Tax Auditors, Journalists, Authors, Artists, Mathematicians, Product managers, Every type of analyst and pretty much any other job that can be done behind a computer screen.
We have big problems for humanity.
I'd kill for an AI that can tell me how to fix shit on my own, with proper permitting and building codes factored in.
I used ChatGPT the other day to resink a CPU with termal paste, replace a PSU in my PC and snake my kitchen sink from the wall. I didn't exactly need it's help but wanted someone looking over my shoulder so to speak. It can help you repair all types of stuff, but as far as county / building codes and such, that probably isn't far off. It seems to understand things quite well.
I think the future is going to look pretty boring, we are all going to review and put our signature below LLM generated output...
By that point money ceases to have value, because the value of money comes from the motivation it gives people to work. If AI does everything, then money is useless (unless AI like money for some reason).
there are always be next frontier of problems which require creativity, unless AI become supersmart completely make humans redundant in all cognitive functions.
> We have big problems for humanity.
The biggest problem is that we're conditioned by a paradigm that frames these as problems.
Most HNers are clueless that if you have Top Talent + Capital you already have an insurmountable moat. OpenAI, SpaceX, Anthropic all have that and none of the regular guys can compete against them (if they choose to attack that industry)
As one wise man said - the oldest profession will also be the last to go.
how about "just turn off your company if you're so worried about AI instead of immediately suggesting we use it to judge people's lives and livelihoods"?
where are the benchmarks
this essay details how law firms became sweatshops from 80s. They charge hundreds of dollars to do make busy work by the junior most staff. LLM will kill the goldengoose of the law industry.
https://aeon.co/essays/what-made-law-into-a-white-collar-swe...
"Introducing Astra for Dog Walkers"
Will these models eventually replace all knowledge work, leaving lawyers, doctors, product managers, software developers, and others out of a job?
If the benefits were shared across humanity, that could bring us closer to utopia. My worry is that we’ll instead end up with a handful of even wealthier billionaires and millions of people out of work.
> Will these models eventually replace all knowledge work, leaving lawyers, doctors, product managers, software developers, and others out of a job?
Effectively yes, in the current forms. Those professions will likely evolve, but the traditional forms (ie writing code by hand, writing law filings by hand etc) are all dead.
> Those professions will likely evolve, but the traditional forms (ie writing code by hand, writing law filings by hand etc)
there will still be writing initial and incremental prompts by hands, until and if LLMs surpass humans in all intellectual functions.
in that case, legal cases would just come down to who has more compute lol. Many times cases win on their merits, but we've also seen evidence where overwhelming legal pressure can influence cases.
Just a few years ago people were paying 400k for pictures of apes. We always make up new stuff to spend money on.
You know most of that was wash trading, right?
If we have enough energy and raw materials to keep building, yes, it will be an utopia made real. But if there is energy scarcity, then other two outcomes can arise.
This also assumes that the AI owners _want_ a global utopia, or that governments will enforce that outcome.
As it stands, it seems far more likely to result in a wonderful life for a few, and an absolute catastrophe for most.
No, it will only leave those without a professional license jobless. Like software developers.
Doubt it. When someone can go to Astra MD for 75% of what they used to go to the doctor for, then the remaining doctors only have 25% as many visits. When doctors only have 25% as many visits, they have to compete on price and they make less per visit.
Same argument for plumbers. Everyone always jokes about what a good time it is to be a plumber. But what happens when all the software engineers turn to plumbing? Suddenly it's not such a good time to be a plumber anymore.
True, unless you're a surgeon.
I mean that's what all of these execs are openly telling everyone: they want you out of work, they want their ai to be the one to bring the world to it's knees, they want to surveille every second of your day, they want killer drones to use, they want to lay all of your cities to rubble and build "paradises" on top of them like in gaza.
They also openly tell you what they are afraid of btw: collective worker power. something that is massively lacking in our industry, although i feel like it would be one of the easiest industries to unionize in terms of # of workers.
Interesting interview I just watched about how powerful and dangerous these "wishes" or "prophecies" are especially in the hands of the ultra-wealthy: https://www.youtube.com/watch?v=eR7grHa1NR0
Cool, now you can pay an attorney $500/hr for them to prompt Astra for 30 minutes, and bill you like they spent the normal 8 hours on the case.
Perhaps, but then someone would do the same but bill you for 7 hours, someone else would undercut them again, until the price reaches a lower equilibrium.
That is an ethics violation. Even in the pre-AI era, getting caught billing for hours not worked was seriously punished.
"7.5 hours - reviewing output"
How could that be enforced/checked?
The grifts continue... imagine something as consequential as Law being advertised as being solved by a statistical word generation engine that regularly gets basic things wrong. Anyone who isn't a lawyer won't know any better but you draft a single document of any appreciable detail and send it to an actual lawyer and it's littered with problems.
Law is a solved problem.
I've always said this will be when we get the real Butlerian Jihad, when the AI firms start trying to liquidate the legal profession.
If you automate lawyers out of a job, you can absolutely automate lawmakers out of jobs next. (Not that this would be a bad thing? Maybe pervasive agents for everyone can be the gateway drug to a "this time it's different!" workable direct democracy)