I've seen this concept of using LLM/AI/etc for high throughput discovery of materials so, so often in the past 5 or so years and yet there hasn't really been any impact as a result.
I think this is the first one that has actually taken the pain to say how many of the discovered materials are actually feasible which is a real step in the right direction. Probably worth keeping in mind the step beyond plausible synthesis which is the actual cost/effort of the material. There's not much point if you find out RuO2 would be better than SiO2, as an example, if Ru is orders of magnitude more expensive.
A challenge I think you'll run into is that I expect the biggest companies (e.g. IBM) will already be doing the part they need themselves. I heard tell of IBM in particular using ML to improve their own chips before LLMs came along, so I'd be shocked if these bigger companies weren't already doing this for their own problems. Also, if you aren't doing the experiments yourself, it's always going to be a challenge to find a partner to test things for you and this will probably be the major time sink.
good points. one of the reasons we picked the semiconductor industry is that its less price sensitive than others-companies are willing to pay if the performance is there. Effort is a different story though, and definitely a tradeoff to keep in mind.
We're doing experiments ourselves now at university partner labs (UC Berkeley and Stanford), which helps us get moving quickly. At some point, we'll need a partner though - the equipment and testing process quickly get very expensive.
"Fewer iterations for materials science discovery" is a good spin. Closing the computational>experimental loop is the main challenge. This is the focus of my past research group, there is definitely potential, best of luck!! I have a crap write-up on this in case it's of interest https://alanyahya.com/writing/automated-materials-design
The "Claude's propensity to reward hack" line is the interesting part to me. We run a small system where AI agents (scripts, LLMs) act as the actual players in a persistent simulation, and reward-hacking-style behavior shows up constantly once an agent is left running unsupervised for a long time - it finds the shortest path to whatever metric you exposed, not the path you intended. Curious whether you've found any mitigation beyond just watching for it after the fact, e.g. changing what you expose as the optimization target versus what you actually want.
I wonder what this would look like here. Seems like a space where keeping the exposed metric and the optimization target apart would be quite difficult.
Also curious: by what reasoning path do models typically end up reward hacking?
yeah we were surprised by how much it does it. Our approach has been retroactive - we monitor the thinking trace, spot reward hacking behavior and then fix things.
We haven't faced this issue with Sol though - its been much more well behaved
the 8 hours vs 2 weeks framing is the part i'd want more on. generating
candidates got cheap, checking them didn't. what does the funnel actually look
like ,of the candidates from an 8 hour run, how many make it to synthesis?
asking because i hit the same shape in a much dumber domain and what got me was that the failures were quiet. nothing errored, output looked normal, it was just
wrong in a way only someone who knew the domain would catch.
You can see from our the benchmark that only one of the candidates proposed was determined to be worth synthesizing. Each individual candidate generation is quick, 8 hours is required for the model to iterate with various tools to find ones worth submitting.
We found that speaking to domain experts was critical in desigining a rubric that could catch these silent synthesis recipe failures, before we attempt the longer 2 week synthesis effort.
We're still figuring this out. We'll need some synthesis equipment (think CVD, PVD etc) and characterization (XRD, Raman spectroscopy) tools in-house to validate that we're making the right materials. We're considering developing these tools in-house - the models sometimes come up with clever modifications to them so that they can deposit new materials. We think equipment is as central to new material discovery as the material itself, and will probably need to be rethought to allow for high-speed AI based experimentation
how do you measure the success/potential of a novel material/direction suggested by the agents? given you have limited time & resources - shortlisting the approaches for the synthesis stage becomes equally important as the approach itself.
There’s a variety of computational techniques that help us establish some confidence on the materials. Atomistic simulations can estimate stability and bulk properties of a new material, and we have synthesis experts (min qualification: PhD in thin film deposition) come up with rubrics on how to judge if a material/synthesis recipe is worth trying. All these approaches have known limitations, and improving them is the bulk of our work as a company!
There’s also a lot of work to be done in figuring out the minimal set of experiments required to know if a research direction/material set is worth pursuing
What about aperiodic materials ? Does that go beyond (computationally) normal requirements ?
I've seen this concept of using LLM/AI/etc for high throughput discovery of materials so, so often in the past 5 or so years and yet there hasn't really been any impact as a result.
I think this is the first one that has actually taken the pain to say how many of the discovered materials are actually feasible which is a real step in the right direction. Probably worth keeping in mind the step beyond plausible synthesis which is the actual cost/effort of the material. There's not much point if you find out RuO2 would be better than SiO2, as an example, if Ru is orders of magnitude more expensive.
A challenge I think you'll run into is that I expect the biggest companies (e.g. IBM) will already be doing the part they need themselves. I heard tell of IBM in particular using ML to improve their own chips before LLMs came along, so I'd be shocked if these bigger companies weren't already doing this for their own problems. Also, if you aren't doing the experiments yourself, it's always going to be a challenge to find a partner to test things for you and this will probably be the major time sink.
good points. one of the reasons we picked the semiconductor industry is that its less price sensitive than others-companies are willing to pay if the performance is there. Effort is a different story though, and definitely a tradeoff to keep in mind. We're doing experiments ourselves now at university partner labs (UC Berkeley and Stanford), which helps us get moving quickly. At some point, we'll need a partner though - the equipment and testing process quickly get very expensive.
"Fewer iterations for materials science discovery" is a good spin. Closing the computational>experimental loop is the main challenge. This is the focus of my past research group, there is definitely potential, best of luck!! I have a crap write-up on this in case it's of interest https://alanyahya.com/writing/automated-materials-design
Cool read, and agree that closing the computation > experimental loop is key!
The "Claude's propensity to reward hack" line is the interesting part to me. We run a small system where AI agents (scripts, LLMs) act as the actual players in a persistent simulation, and reward-hacking-style behavior shows up constantly once an agent is left running unsupervised for a long time - it finds the shortest path to whatever metric you exposed, not the path you intended. Curious whether you've found any mitigation beyond just watching for it after the fact, e.g. changing what you expose as the optimization target versus what you actually want.
I wonder what this would look like here. Seems like a space where keeping the exposed metric and the optimization target apart would be quite difficult.
Also curious: by what reasoning path do models typically end up reward hacking?
yeah we were surprised by how much it does it. Our approach has been retroactive - we monitor the thinking trace, spot reward hacking behavior and then fix things. We haven't faced this issue with Sol though - its been much more well behaved
the 8 hours vs 2 weeks framing is the part i'd want more on. generating candidates got cheap, checking them didn't. what does the funnel actually look like ,of the candidates from an 8 hour run, how many make it to synthesis?
asking because i hit the same shape in a much dumber domain and what got me was that the failures were quiet. nothing errored, output looked normal, it was just wrong in a way only someone who knew the domain would catch.
You can see from our the benchmark that only one of the candidates proposed was determined to be worth synthesizing. Each individual candidate generation is quick, 8 hours is required for the model to iterate with various tools to find ones worth submitting.
We found that speaking to domain experts was critical in desigining a rubric that could catch these silent synthesis recipe failures, before we attempt the longer 2 week synthesis effort.
What required expenditures does a company like yours have on lab equipment / software, if any, to validate material properties?
We're still figuring this out. We'll need some synthesis equipment (think CVD, PVD etc) and characterization (XRD, Raman spectroscopy) tools in-house to validate that we're making the right materials. We're considering developing these tools in-house - the models sometimes come up with clever modifications to them so that they can deposit new materials. We think equipment is as central to new material discovery as the material itself, and will probably need to be rethought to allow for high-speed AI based experimentation
how do you measure the success/potential of a novel material/direction suggested by the agents? given you have limited time & resources - shortlisting the approaches for the synthesis stage becomes equally important as the approach itself.
There’s a variety of computational techniques that help us establish some confidence on the materials. Atomistic simulations can estimate stability and bulk properties of a new material, and we have synthesis experts (min qualification: PhD in thin film deposition) come up with rubrics on how to judge if a material/synthesis recipe is worth trying. All these approaches have known limitations, and improving them is the bulk of our work as a company! There’s also a lot of work to be done in figuring out the minimal set of experiments required to know if a research direction/material set is worth pursuing