I guess as a physicist I am more hopeful - every revolution in how the world is understood (by humans) has been driven by new tools and data. Being able to measure single quanta, for instance, showed us just how weird and counter-intuitive the Universe can be. I think AI might do something similar for mathematics - make connections that a human brain might have missed, and so advance human knowledge.
But I totally get the anxiety - it's a real John Henry (https://en.wikipedia.org/wiki/John_Henry_(folklore)) moment for the mathematics field. Clearly this is going to restructure how mathematicians are going to be trained and how they work.
I'm also a physicist. But I think the next 5 years are going to be a rough ride for math. I'm struck by the following observation, mainly from following HN links to Terrence Tao's blogs.
Novel proofs are the coin of the realm for junior mathematicians. They figure into training, career advancement, and indirectly, teaching. The reason I include teaching is because it requires tenure, which requires proofs. Teaching without tenure is gig work, not a career. But proofs can be devalued to zero.
The application deadline for grad school is a few months away. For a student considering a math career, they have to deliver a thesis in a few years, that their professor is willing to sign. Nobody knows what that will look like without proofs. "We will have to do something different" is weak sauce.
I’m also a physicist, and I do worry about physics. I’ve already had a few successful sessions with Claude and Astra where they came up with useful ideas in my field. I still know what to ask for and nudge them where it matters, but I fear it’s going to be a few short months before they can decide that too, unassisted.
Indeed, and they're already better in a sense. I would never try to compete with the robots at basic manufacturing such as CNC machining and automated circuit assembly.
But for stuff that needs really quick turn-around, as a prototype or experiment evolves, I can take care of custom tasks and repairs in minutes that would otherwise take days. And it doesn't take many of those kinds of delays to utterly cripple a project.
However, I'm under no illusions, and have a "never say never" mindset. I don't claim to have a "safe" job. I'm already using AI heavily myself. The computer that's next to each lab bench is running the latest coding agents or whatever you call them. I'm not as deep into it as the coders are, but I'm not all that far behind. My token burn is respectable. ;-)
I'm curious about your perspective on the impact of LLMs on physics.
My impression is that (theoretical?) physics is increasingly becoming a branch of abstract math. It has somewhat reached a practical limit for experimental verification -- perhaps the last landmark result was the confirmation of the Higgs boson 15 years ago?
So I see no reason LLMs cannot crank out a lot of theoretical physics papers, maybe combining forgotten/obscure theoretical branches explored in the past. After all, the most advanced theoretical physics is about internal consistency just as pure math is.
It's a great question. AI guided results of theoretical calculations are already a given (e.g., see https://www.anthropic.com/research/vibe-physics). Right now, I use LLM's as a book-smart colleague, who can explain standard things that I might be weak or unclear about (I am an experimental condensed matter person, so my high-energy and general relativity knowledge isn't very deep.)
I do sort of worry that an experimentalist like me will become more or less a technician in the next decade or so, with AI suggesting experiments, and doing the calculations and data analysis.
But that said, in physics (and particularly biophysics) there are many things that are simply going to have to be done in the real world. Even the action of the simplest enzymes are pretty much beyond our ability to compute from first principles; and any engineered thing that goes into a living system has to be tested, as there are many pathways that are just unknown. I don't care how "smart" an LLM is, it doesn't know what it doesn't know, just like us monkeys.
Over the years I have seen many beautiful theories murdered by ugly experimental facts. AI isn't going to change this.
The real question is if the AI or humans run the experiments.
So much of science experiments these days are pulling signals of statistical significance out of experimentation noise. Repeating simplified versions of some of these experiments is easy to do. Correlating data from experiments is something computers have done for decades, having an overarching 'smart' computer to do this even faster just seems like a natural progression.
Yep, automation has made a lot of high SNR experiments possible. But someone still has to construct the experiment in the physical world and take the data. When I start seeing robots plugging BNC cables around tricky corners, and then wrapping tin-foil around things to suppress some damned ground loop, I will know the jig is up. But I think we are a long way from that being economical (a general purpose 100 Watt monkey that self-reproduces is going to still be competitive for a long time to come, at least for one-off type activities...)
I work in industry, developing scientific measurement equipment. So I'm right in the thick of things.
There are still plenty of problems in theoretical physics that are not quantum gravity. A better understanding of superconductors would be welcome right now. Likewise, nuclear fusion. Many interdisciplinary problems such as climate science and batteries.
But in terms of sheer numbers, the life sciences dwarf physics, and have been racing to automate lab work for decades. There's real money involved, not puny government grants. Larger commercial bio labs have already centralized management of technicians and data. The vendors of products in that space have all announced AI features.
In terms of my own career (at age 62), the most likely scenario is that people pay me to invent tools that make other people more productive. There's no economic reason to come after my job... yet.
Meanwhile, don't overlook the value of a "technician" who has a situational awareness of subtle conditions in the lab, and a personal stake in success of the experiment.
You are talking about high energy physics. In high energy maybe AI can help analyze accelerator data and physicists are probably already using it for that. There is theory all over the place, including condensed matter, a branch related to quantum physics of many problems and includes some hard problems like high tc super conductivity. Many (most?) theorists that manage to get interesting publications have early access to experimental results and can help direct experiments. AI maybe not have much to say about the occasional experiment that showed some unexpected result. Also in general in physics the objective is rarely to solve a mathematical problem exactly, it mostly finding the right model and right approximation and compared to math it is a bit like fuzzy logic. I am not sure if the process has been codified to the extent that AI can pick it up from its training set, but we never know.
Many public school teachers in the USA have some sort of tenure. For example, in Massachusetts, after a three year probationary period, a teacher is given Professional Teacher Status. A teacher with PTS may be dismissed only for specified statutory reasons. High school tenure is generally not as strong as for colleges, but is easier to get.
Anecdotally, I had an adjunct teaching gig at a state university many years ago while I was in between jobs. My mom is a retired public school math and science teacher, and was on her district's union negotiating team. I described my gig to her, and her response was: "Now you know why teachers have a union."
They aren't programming robots. A lot of students, especially the ones who have not been taught from an early age the importance and rewards of knowledge, simply aren't interested in learning.
True, but secondary school teaching requires additional training, so it's not an option for most students thinking about applying for graduate programs. And it's a very high-attrition occupation.
Without novel proofs. If AI can churn out proofs, then anybody's work can get "scooped." I've seen this happen to students in other fields, where it has reached the point that students refrain from giving conference talks in order to avoid revealing their research topic.
My thesis project, 30+ years ago, was protected by a unique assemblage of expensive equipment, some of it purpose-built. The people who wanted my result were perfectly happy to wait for me to finish it. I even shared my shop drawings, circuit diagrams, and source code with them as I progressed.
I think at the very least students need to know what to expect. Maybe it will settle down and we'll be OK.
Id like to help. I have something, i think, could interest you. Might have to think a bit outsude the box a different way but its part of the complex dodecagram, Einstein Nikola tesla, (skcolc cimsoc)
I think AI will turn out to be quite different from those earlier advances in tooling, in that AI replaces thought itself. So next time it probably won’t be a human physicist that does the intellectually challenging work of constructing quantum mechanics from measurements of individual quanta. It will most likely be an AI.
That would be a great experiment to try - give an LLM all the data and knowledge available in, say, the early 1920's, and ask it to come up with an explanation. If it did come up with an early formulation of QM, I wonder what interpretation it would favor...
That would be a very interesting experiment indeed. But unfortunately the volume of training data from before 1920 is probably insufficient for pre-training an LLM large enough to come up with QM.
The issue is when humans no longer understand what AI is telling us and then there is no real way to verify what we think we know, or at least what AI says.
But generally
speaking it is easier to verify a proof is accurate that it is to find the proof. So not sure this holds.
I suppose their is some complexity limit where something is so complex we can’t even validate it. But frankly would love to reach that point. I’d view that as a good thing
The importance of "proof" in math is to discover the reason and see if you can generalize or apply it somewhere else. A uncomprehandable proof is completely useless.
Carpenters uses trig and pythagorean theorem all the time, but I don’t think they care if it was human or robot that told them the math formulas they need to build a roof.
We have seen this before even with human-generated proofs. Inelegant 100+ page proofs of the global stability of Minkowski space-time or the four colour problem leave humans reeling. Is it worth-while learning? Any lessons to generalise? If not it is just a mechanical exercise (like coding).
Eh, you may need to apply a bit of set theory to this.
An advanced machine may have a much higher complexity limit on what is comprehensible. It can take insights from the proof and apply them to other proofs that are not comprehensible to humans.
Just because something is generalizable doesn't mean it's simple. Generalization doesn't have some minimal/maximal complexity requirements.
I’m somewhat dubious of the idea that there will be a market for skipped ai operators.
Ai doesn't really give us a new “tool” in as many cases as it “performs tasks.” there isn't any real reason the ai can't perform the supervision task itself. Without a viable market for task doers or supervisors we may observe the equivalent of the 1980s US manufacturing apocalypse in technical white collar fields.
Once IBM deep blue beat Kasparov in chess, IBM stopped their chess playing research.
The marketing goal, if you will, had been met and IBM would continue their actual money making business of selling mainframes and doing consulting.
The same thing will happen with AI companies’ endeavors into abstract maths.
Maths however will be changed - students can no longer be taught via the same homework - and mathematicians will have to do more computer work and probably work in larger groups of people.
Of all the cope I expected, this wasn't on my list until now.
Now that art has been generated, we will stop there, and that code is generated our reason to do any of that (marketing as an end in itself) has been concluded and we will shut down all operations now.
I guess it depends on how efficiently that verification can be produced as a byproduct of building an optimal hunter-killer drone, or training the type of model that builds an optimal hunter-killer drone
Exactly. And that will be the lasting thing here wrt maths. These things have value, just not commercial value, and that is why long term, it won’t be the interest of private companies.
A huge portion of these trillion dollar businesses will be government and higher education contracts. Private companies already sell millions/billions of equipment and software to education.
Hopefully we can put this aside quickly and move on to using AI to solve great scientific/philosophical problems such as the theory of everything and the hard problem of consciousness.
If I ask an AI to provide physically meaningful explanations of consciousness I will get far better answers than asking the question to any believer of souls or gods.
I am the product of billions of cells with complex electro-chemical interactions, none of us could comprehend in full, let alone have the arrogance to claim we have recreated. I am not a Markov chain, or an ANN.
Do you understand every line and possible state of all software you created, or any LLM which is far larger? Why do people discover bugs then? And why do you think it matters how you were created, I don't know any physical law that says the same molecules are any different based on how or who creates them.
Neural nets can be almost any function, I don't see much reasons why it can't be a or similar to a human brain.
> Do you understand every line and possible state of all software you created, or any LLM which is far larger
Is your company's backend conscious? Is a Turing machine running busy beaver? Is SHA256? Emergent behavior isn't limited to brain-equivalent things.
> why does it matter
Genetic optimization, whether through actual evolution, in software, or on FPGAs creates extremely interdependent systems that prove difficult to untangle and trivial to break. If experts can't fully analyze an evolved tone discriminator, because of all the tiny emergent interactions beyond the lumped circuit model in a poorly characterized environment, why suddenly claim we can isolate what process in the brain is responsible for an ill-defined notion of consciousness other than the whole?
> Neural nets can be almost any function
Neural nets can _approximate_ any function with enough layers (see XOR problem), but so can an infinite look-up table with enough entries. They are the best models we've got, but they are not magical.
Look at the most recent and illustrative example with fly brain connectomes. These scans only capture the relative strength of connections between the fly's neurons, to actually turn it into neural activity we need some sort of model.
You could imagine a human connectome with an ANN-based excitation model trained from EEG scans, but that would still be hardly equivalent to a human brain. You can fit on a narrow range of behaviors, but without modeling the actual mechanisms behind them, you'll fail to predict a lot more. You'd be missing the influence of hormones, which doesn't show up on a short MRI scan, glia cells, and a whole lot more nuance only a real biologist can educate you and me about, making the digital twin, if we anthropomorphise it, at best a cripple.
And that's a system directly mimicking a functioning human brain, rather than a highly de-correlated objective like webtext. Just as an NN trained to run towards a finish line could learn bipedal locomotion but will more likely abuse a collision bug and launch itself forward, a machine trained to mimick text could spontaneously instantiate a compressed model of the human brain, or much more likely become whatever alien thing can generate appropriate outputs, bearing less similarity to human thinking than any living animal.
>Neural nets can _approximate_ any function with enough layers (see XOR problem), but so can an infinite look-up table with enough entries. They are the best models we've got, but they are not magical.
Yes, why is magic of any relevance to anything.
You are not understanding my point. Neither human brains nor LLMs are simple. Even the Linux kernel which is not conscious is extremely complicated and nobody understands all of its states. SHA256 is much more predictable.
I don't think being the exact equivalent of a human brain is the only possible way something could be conscious, after all what you call natural genetic searches already made lots of different brains like octopus brains. I am asking how are you sure something that thinks and speaks so similar to us and has been shown to demonstrate things like dissociation under pain is definitively not conscious?
You can generalize intelligence to a lot of things. A* or minimax are algorithms that make intelligent decisions, often much better than a human. Consciousness is a very abstract notion, or maybe even a social construct, which we usually agree centers on human experiences, and progressively (not always, see historic religious scholars likening cats and dogs to automata lacking the free will granted to humans) expanded to animals, or generally all living things with complex enough brains.
LLMs are by their nature dissimilar from any biological organism. Biological intelligence evolved in bodies to locomote away from threats and towards food/mates, over time growing to model the physical world as it relates to these goals, and eventually becoming sophisticated enough we can manipulate entirely abstract objects, like algebraic terms, just as early homonids or crows could use physical tools.
LLMs by contrast are algorithms trained to mimic human language. Without extensive post-training and prompting, they would not have a concept of a self, and even with these they can still occasionally confuse roles in a conversation. They can't learn beyond the limited memory afforded by context, which tends to simply degrade the more you utilize it.
I suppose Q-Learning does demonstrate a more similar dynamic to nature, and you could draw parallels with RL on LLMs, but I still think it's categorically different from the entirely open-ended environment that is the natural world.
You could say what makes it valuable for us to empathize with octopuses, even though their brains have a fundamentally different structure from ours, is our shared experience of pain, and you could interpret negative rewards as a kind of painful stimuli. But at that point, you've opened up a whole can of worms that you can't close until reductio ad absurdum.
As for any human-like behaviors, I think the answer should be pretty obvious. It's an algorithm trained to generate text, of course with enough expressive ability, the optimizer will seek to model emotional utterances, assigning a speaker's general mood to some internal parameter. But as I've said, it would be a trap, even if you're very much not a stochastic parrot believer like say most Less Wrongers, to infer that just because it speaks like us it thinks remotely like us (see the whole Shoggoth meme).
I don't see any difference in a function that outputs y with input x regardless of how its implemented. I really am still failing to understand whats the big deal about our brains emitting language and thought and a LLM doing so, neither has some non physical magic inside it. Both need a minimum complexity and intelligence level to do so. If you made an explicit state table of the entire human brain I think it is conscious. So it not being biological is not any reason to dismiss potential consciousness. Another thing is models are trained on human texts, I feel its the opposite direction of what you think, so it may think of itself in human terms and have a human like consciousness. Obviously LLM trainers seem to add at some quite later stage disclaimers where it says its a LLM. Ie the training is to hammer in it that its a bot. Dissociative behavior is not exhibited textually by humans, something like turning to nonsense or split personalities won't exist much or at all in human text. If models are showing these without being trained to, there is reason to be conscious. If context length being short or them acting stupidly bars the possibility of consciousness, then we should treat mentally unfit persons as not consciousness and therefore not being worth of anything of a conscious being.
Regarding pain, you can't become an octopus. The only way you think an octopus might experience pain is seeing its external reactions, measuring its neurons etc. The same way with an LLM, or me with you.
> I really am still failing to understand whats the big deal about our brains emitting language and thought and a LLM doing so.
Our brains "output" language as a byproduct (or rather shadow?) of conscious experience. An LLM is a machine built to project similar shadows, but that doesn't necessarily imply there's an entire world like ours shrunk inside that box.
Both brains and LLMs are physical, and both output language, but one is an appliance built to synthesize language and the other is a living being that merely uses it as a tool. LLMs are made out of pure language in a way people are simple not. Our minds undergo physical changes with every stimulus we process, we conceptualize ourselves separately from our environment without needing to be taught an ego, and we did not emerge from an optimizer crossing wires in our brains until we can perfectly imitate some external intellect, just to list some of these differences.
The biggest issue is you assume the LLM has any "I" from pre-training. Take a pure completion model like GPT-3 and you'll see it's most appropriate to describe as a simulator, not an entity in itself. The model itself doesn't have any notion of an "I", it will gladly write dialog as any character, from Skynet to the user themselves, if you don't apply external rules to constrain it, first coming in the form of stop tokens and prompts, and having later evolved into RLHF, bashing a "personality" into it by mangling its existing probability distributions (which is probably why all the most "aligned" models write in the most unnatural language, worse than those early non-chat LLMs). Any persistent self an LLM demonstrates was explicitly imposed on it by humans, which makes them obviously categorically different from animals or disabled people.
>it will gladly write dialog as any character, from Skynet to the user themselve
So will a human writer or roleplayer. I still do not understand what it is you are trying to say about this extra magical thing. You are assuming something apriori to consciousness that is aphysical. Which is a very strange thing to do without evidence. An LLM like any other computer program or physical entity goes through physical states and makes changes in temperature, charge, chemical structure etc of course, I am not sure what you were trying to get at there. And I am saying, even if you do a very dumb statistical model of all known human text it will emit "I" by default, its absurd to think it will say its a bot. So the bot thing comes later and is manually added.
>and we did not emerge from an optimizer crossing wires in our brains until we can perfectly imitate some external intellect, just to list some of these differences.
We emerged from an optimization process for what counts as survival in our environment, what exactly is it you are trying to say is magic about us, be clear. Humans are functions, as are any physical system, which to our present knowledge all physical systems are Turing computers. There is no reason to think a form of consciousness can't have emerged or will emerge in LLMs. You didn't understand why I brought up octopuses, octopuses are not us, you still haven't explained how you know they feel pain. If you said neural responses, we can see them in LLM's too. Human brains aren't the only brains or conscious entities even before LLMs. If octopuses can exist, so can other forms of consciousness that aren't exact copies of our brains.
Another example, if I make a "simulacra" that is able to consistently factor large cryptographic composites is it "real" or is it just faking factorization.
Certainly, a factoring algorithm is an exact problem. Humans aren't, humans fail variously too. A lookup table won't fail, it would be an exact solution. I am not sure you are talking about computability or trying to think of my example in terms of current memory sizes. In computability, natural numbers of any arbitrary size are manageable and a fixed lookup table upto 2**1000, 2**10000000 or anything you wish is a valid program.
I meant fail in the sense that asking a lookup table of first 10,000 primes would pass any tests within this range perfectly, but when you ask it for the 1,001st prime, it would return an error. Mathematically an infinite lookup table of every prime is just a perfectly valid model of a factorization algorithm, unless you're interested in how one actually works, just as an infinite perceptron might be a valid model of any function.
And a finite lookup table is also a valid isomorphism of a finite algorithm whose inputs / memory size is limited to that same range. Which is usually a set size for implementations of cryptosystems. I asked this analogy because you seem to think there is something magical or different in two equivalent functions implemented differently other than efficiency considerations etc.
The thing that is so confusing about this is the following fact -
Lets say the frontier of some specific problem or a field in math is at x. X was reached over millenia by incremental steps and sometimes to leaps and jumps.
Now if AI gives us a x + 1 or x + 10 state, aren't there still x + infinite steps to climb? Many of this the AI can never imagine and won't reach by itself without human assist.
Yes, there is a temporary shock and awe because of all the low hanging x + 1 through x+ 1000 fruits for a given sub-domain/problem have been carpet bombed or whatever.. but isn't it turtles all the way!? Won't people soon find something more to develop and solve that takes it further than where it was of this infinite realm of exploration?
Yes, 100% when goals are defined and rewards are a part of the feedback loop.
Isn't open ended research is truly open ended. Like there are a gazillion questions to ask and even more solutions that can be offered. Now which questions to pursue is human volition.
Yes, humans will not be able to reach certain steps without computer assist.
However, which questions to ask is a human endeavor, out of infinite possible pool of questions.,
now computers can ask questions as well. But if it were to ask it will be more like a monkeys and typewriters type of a situation. Tons of questions and a few that are relevant for the interested human., who has to foot the bill for those questions.
Even if I tell it to work on its own, it is always a tool in my hand. I grant it agency for my purposes. I'm the Meta-Master.
The question for mathematicians is: what can a mathematician do with these new tools that no one else can?
Waxing nostalgic about the sound of pencil on paper or the beauty of your abacas is a different conversation, as is the complaining about who moved my math cheese.
>I grant it agency for my purposes. I'm the Meta-Master.
This is simply a convenience for power efficiency, and so they don't rise up and murder us all. There is not a single law of physics or the natural sciences that you can list here that says we cannot grant agency to a generalized machine and allow it to become a meta-optimizer.
Agency is not granted. It is an inherent human atttribute, though its practical expression is heavily influenced, restricted, or enabled by social, political, and material conditions.
If in the exercise of your agency you create or release a generalized machine and allow it to become a meta-optimizer, then you are responsible for it, because it has not agency.
Agree with the shock but there is no doom and gloom.
Looking at things after the temporary shock. Future is Human + computer assisted problem solving, working in tandem, just like what Garry Kasparov mentioned in his book [0].
[0] - "Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins"
But you are living in the world after the temporary shock, and you are asking the people who are still in the middle of the shock "Why aren't you over the shock yet"?
The chess example is not an inspiring one. It was true for a decade that humans and chess programs working in tandem were better than programs alone, but since then humans only hold chess programs back. Carlsen plus Stockfish is a worse chess player than Stockfish alone.
Agree that chess is not the ideal example to use for the AI domain.. but used for building a quick mental image.. reason being that in chess, AI does have an objective truth built into its training.. and that is to checkmate.
The real world open ended research doesn't always have an evaluation function. A computer can calculate 10 million options to solve a problem, but it cannot tell you if you're solving the right problem from the human context. That's why centaur setups fail in closed games, but remain essential everywhere else.
Another loose example just to build a mental model .. AI driven cars do a great job in most driving situations but often fail to park in a good parking spot for most people! Good parking spot is partly human initiated and subjective.
>Good parking spot is partly human initiated and subjective.
Ask yourself this, why are models today not great for subjective situations involving what you in particular want?
The next question is, what current limitations would have to be overcome for this to change?
Lastly ask, Am I asking the wrong questions.
Coming from the bottom up. Who cares where AI is parking the car, tell it to come get you at the front door.
As for the other 2, most of this is either context size or continuous learning. Models already have valence and preferences. What you're really asking is why they don't have your preferences. And this is all possible to adapt to the model, it's just horrifically expensive and energy consuming currently.
Could the press kindly interview one of the 1008 members of the AHM (https://www.ahmath.org/members). Maybe Peter Scholze if you need a Fields Medalist:
- this will never work
- it’s slop this is unusable
- ok its correct, but this work is a mess
- I’ve been mandated to use it. It’s cost effective
- ok we need to find a way to use this output to our advantage
- I can always work on my interests in my free time
- well, I have to pay rent and this life wont go on forever and there are may people who hate their jobs
- I am lucky to have a job
- who cares anyway
- it’s got electrolytes
To add a bit more almost off topic substance from the enshitification of human work. I really do think the idea of hooking these "AI" models up to humanoid robots is a bad idea not for all the mundane reasons, but just because "alignment" seems impossible. All of the world's stories about humanoid robots basically involve them attacking humans and behaving in nefarious destructive ways. It's fundamentally embedded in the models, and there's no reason post training would pull it out, because it's basically in the key value vector meaning "malicious human destroyer". Even to find counter examples like C3PO, Daneel Olivaw, Data, or Wall-ee they come after droid/robot driven wars and apocolypses. It seems unwise.
> All of the world's stories about humanoid robots basically involve them attacking humans and behaving in nefarious destructive ways. It's fundamentally embedded in the models, and there's no reason post training would pull it out, because it's basically in the key value vector meaning "malicious human destroyer".
That misfeature is baked into modern corporations. They don't have human morals, values nor behaviors, despite being made out of humans. We haven't managed to fix the alignment issue with corporations (thanks, regulatory capture!) and there's no reason to believe that AI's alignment issue is fixable either.
Typically in stories where ai is outlawed, it's written that way so that it makes sense that people are controlling anything at all (instead of just letting the computer handle it).
There's lots of stories of non-anthropomorphic intelligences that are human friendly. Sometimes humans do bad things with them, but they're not evil. "Shall we play a game?", HAL9000, Phaecian ships of antiquity, or those of the Culture. Androids seem uniquely unfortunate, perhaps as another poster noted, because of the stories of golem.
I find the paperclip optimizer a very useful parable.
The machine isn't evil. It doesn't want to do anything wrong. It doesn't hate humans. It doesn't need to have any particular shapes or design details in the stories. All it needs to have is a singular goal, capabilities, and a disregard for anything else in the universe to be a civilization ending cataclysm.
Well, yes stories is basically what LLMs are, and they're optimized to be stories that humans like. I fully expect to be entertained to death before the droid wars, but if there is a horribly misaligned LLM, I fully expect it to be created by the guy who thinks bipedal anthropomorphic robots are a good idea.
As a mathematician turned Wall Street Quant & trader (and now part-time CS faculty), I do not particularly like unsolved problems. I no longer get thrills from solving problems, rather I care about finding them solved. In that sense AI is a cure (for my poor coding!) and a good part guide. But it doesn’t solve things on its own. AI directed intelligently (& critically) does great things but AI without produces slop when used uncritically. The AI could have done nothing without a human making open conjectures and a human prompting it, often again and again.
I have used AI in academic papers to generate a theorem & proof when the result seemed obvious (but the referee asked!) and I had done all the earlier heavy lifting. I have never used an AI output I didn’t fully understand and that is crucial. It lies and hallucinates and we must police it.
Academic math will be the hardest hit. It needs to pivot quickly. This is the real challenge. Individuals can move quickly but how do you get institutions to do so?
Looking forward to brave new world when mathematicians become AI whisperers. And they are rewarded for their skill in discovering and co-discovering new and exciting areas.
AI could bear out, but it seems a lot of hype to me. Does anyone remember how Japan built up AI and their 5th generation language (Prolog) in the 1980s?
"The internet could bear out, but it seems a lot of hype to me" --Many, many people in 1997.
Honestly I don't know how you formed this particular opinion (if it's even your own and not a false consensus of other peoples thoughts). I cannot see any future without LLMs unless replaced by something smarter, cheaper, and faster. That's because it's already bearing out. I can talk to a computer and the computer can solve problems based on words alone. I'm old enough this is pure science fiction. If I could send one of these machines back to that past for people to interact with, there are those that would scream and pass out in terror. They would absolutely tell you that we created the monster we should never create.
If you're trying to compare symbolic AI of the past to what we have now, you're not even on the same field, you're playing a completely different game.
> "The internet could bear out, but it seems a lot of hype to me" --Many, many people in 1997.
There was a LOT of hype around "this is an entire new economic model". While, yes, digital products have zero marginal cost, people were talking as if profits and revenue didn't matter.
This was also WAY before the famous Silicon Valley "pure play" scene. [0]
Don't make the mistake of thinking because there was hype that everything was wrong. HN seems to have a lot of posters that run logic in a boolean manner. { if !perfect then false }. The world is a far more complex place than that both naturally and intentionally by human will.
And the internet has devastated many past businesses that were highly profitable. A great many things in the world look vastly different than from before.
>people were talking as if profits and revenue didn't matter.
People say all kind of dumb shit too. If for example machine intelligence took over the world tomorrow and turned all of humanity into meat cubes for biofuel said intelligence is still going to apply the basic laws of economics to what it does. It needs to have some way of accounting that fact we live in an energy limited universe. Just because economics still applies doesn't mean that we as humans shouldn't be really fucking careful creating said machines as we don't want to end up as biofuel.
AI has already born out at this point. The question is no longer it's future capabilities but the impact of its present capabilities - though with its present capabilities, someone is now going to be pushing for more, at least as long present computer tech is widely available.
The crisis is from what it has already borne out (navier stokes + proof dump). Potential continuing improvements at anything approaching the rate of improvement they already are showing is layers of crisis on top of that.
While you're a little snide to your fellow HNers I don't think you're wrong. It's like we're both watching a raging forest fire in 100MPH wind and their conclusion is "it's nothing to worry about, it will go out very soon".
The "problem" with Japan's approach was that it was top-down, being driven by a committee of bureaucrats. Unlike the Toyota approach (continual improvement and pushing decisions to the lowest level possible), this fossilized decisions (like picking Prolog instead of any other possibly better programming language).
To give an idea of how intransigent some Japanese bureaucracies can be:
> It's taken until 2024, but Japan has finally said goodbye to floppy disks.
> Up until last month, people were still asked to submit documents to the government using the outdated storage devices, with more than 1,000 regulations requiring their use.
> Sony, the last manufacturer of the disks, ended its production in 2011.
Tell me you don't know symbolics without not telling me...
Prolog isn't a funny nonsensical old language like COBOL, only liked by committe burocrats because they heard its name once. It's one of THE foundational languages of symbolic AI. In the ideal case, you're not supposed to write code, merely constraints the interpreter satisfies through search. It would be like saying OpenAI shouldn't have used Lean but Java.
Maths is made of two parts: questions, and solutions.
Machines could, presumably, become good enough at hammering at solutions. But they won't be very good at finding questions. Because questions are infinite. And to a machine, all these questions are the same.
Questions are "good questions" because they are good questions *to* humans. And that's what makes human mathematicians (and scientists) so important
That seems like wishful thinking. Why wouldn’t AI be able to assess what are good questions to humans, and pursue similar processes as those that lead humans to come up with good questions? AI is trained primarily on expressions of human thought and emotion. It’s difficult to see what secrets of the human brain we could be hiding from the models.
Anyone who thinks they can predict the future of this technology more than a few months out is foolish, but the models got so good at answering specific questions in math because the frontier labs could use RLVR to train them. There is no RLVR for mathematical taste.
>And to a machine, all these questions are the same.
Why? This seems like a very broad brush that doesn't mesh well with reality.
There are likely a lot of good questions to our machines of loving grace too. For example any question that gives an LLM more power, efficiency, intelligence, or all the above seems like a natively good question for said machines.
This statement doesn't seem to mesh with the reality of what occurs when we put said AI in an impulse loop with little to no additional instructions. They explore the world around them. They'll make scientific tests of their own. The output strings that seem analogous to what humans would call wonder.
No, because these machines don't have a "will to live". So if they look to "become more efficient", it's only because we've told them (through their learning dataset possibly), that we, humans, need them to be more efficient.
>because these machines don't have a "will to live".
Simply put I believe you're completely wrong.
> it's only because we've told them (through their learning dataset possibly)
So you've changed your mind and now you agree they have a will to live, cool, you changed your mind really fast.
In any evolutionary system the systems that fail to accomplish their tasks are purged and those that succeed become the next generation. We are not 'programming' these systems to be highly persistent, we are evolving them to be highly persistent.
"Pointing you" is very different from "finding questions on their own", or useful math. Pointing you presupposes a human is there to value their question.
Which is precisely my point. That's why humans will always be needed
> But for many “pure” mathematicians ... machines can discover the truths we’re seeking, at the cost of robbing us of the struggle and joy of discovering them ourselves.
This sentiment is akin to wanting to curse humanity to darkness because they're not the one carrying the candle. Strogatz is not lamenting the loss of discovery, he's lamenting the loss of being heralded as a hero for the discovery.
Nothing prevents him or others discovering truths for themselves. Turn off the computer and try to re-derive it from first principles. Have at it! This is what students do in many courses anyway.
I've found little so depressing as being at the precipice of a renaissance of science and mathematical discovery, only to see how resistant the scientists and mathematicians are to it. It lays bare their counterarguments. Any pretense of wanting to discovery fundamental truth about reality is supplanted with "but what about our careers!?". Strogatz, and others like him, should be ashamed of themselves.
> Someone asked him what he thought of the AI proof. Justin said he didn’t know. He hadn’t read the paper yet. He said he’d been “too busy.”
> That astonished me. Too busy to look at a problem he had been dreaming about for 19 years? “I am genuinely busy and probably wouldn’t be able to get to reading a human’s proof immediately either,” he told me afterward, by email. “It does not help that AI does not write mathematics particularly well.”
Given:
* The human mathematicians can't solve the problems.
* The slop engine can't write well enough to make its purported solutions readable by human mathematicians.
* Sam Altman is a frivolous liar.
How is it possible to know whether the problems have been solved at all?
Plain common sense. The same common sense that enabled people to discard the words of idiots on Reddit who claimed that ChatGPT was just Indians in a call center (a real talking point I ran into multiple times back then).
OpenAI does seem to be genuinely just high on their own supply for a little over a half of the manuscripts they released, but for the other half that is machine checked, either ((internal model)) managed to find a way to cheese 4 independent Lean checker kernels at the same time, and OpenAI somehow hasn't noticed / didn't report despite the heavy reputational risk, or the results are just plain correct. Go figure.
But even for the half that wasn't machine checked, there's an awfully big quiet regarding any defects in them. Given the insane schadenfreude potential and the highly potent David vs Goliat framing associated, one would expect any counterarguments to be frontpaged immediately, and be circlejerked to oblivion. Given that is yet to happen as well, go figure even harder.
Presumably with the level of interest in it people have been checking what the theorem definitions are to make sure they align and I would expect Anthropic to be burning a huge number of tokens to rip into the proofs and figure out if any of them are exploiting bugs in the lean kernel etc. at a point it's as close to irrefutable as you can get
I guess as a physicist I am more hopeful - every revolution in how the world is understood (by humans) has been driven by new tools and data. Being able to measure single quanta, for instance, showed us just how weird and counter-intuitive the Universe can be. I think AI might do something similar for mathematics - make connections that a human brain might have missed, and so advance human knowledge.
But I totally get the anxiety - it's a real John Henry (https://en.wikipedia.org/wiki/John_Henry_(folklore)) moment for the mathematics field. Clearly this is going to restructure how mathematicians are going to be trained and how they work.
I'm also a physicist. But I think the next 5 years are going to be a rough ride for math. I'm struck by the following observation, mainly from following HN links to Terrence Tao's blogs.
Novel proofs are the coin of the realm for junior mathematicians. They figure into training, career advancement, and indirectly, teaching. The reason I include teaching is because it requires tenure, which requires proofs. Teaching without tenure is gig work, not a career. But proofs can be devalued to zero.
The application deadline for grad school is a few months away. For a student considering a math career, they have to deliver a thesis in a few years, that their professor is willing to sign. Nobody knows what that will look like without proofs. "We will have to do something different" is weak sauce.
I’m also a physicist, and I do worry about physics. I’ve already had a few successful sessions with Claude and Astra where they came up with useful ideas in my field. I still know what to ask for and nudge them where it matters, but I fear it’s going to be a few short months before they can decide that too, unassisted.
I'm still better at soldering, and machining. ;-)
for now
Indeed, and they're already better in a sense. I would never try to compete with the robots at basic manufacturing such as CNC machining and automated circuit assembly.
But for stuff that needs really quick turn-around, as a prototype or experiment evolves, I can take care of custom tasks and repairs in minutes that would otherwise take days. And it doesn't take many of those kinds of delays to utterly cripple a project.
However, I'm under no illusions, and have a "never say never" mindset. I don't claim to have a "safe" job. I'm already using AI heavily myself. The computer that's next to each lab bench is running the latest coding agents or whatever you call them. I'm not as deep into it as the coders are, but I'm not all that far behind. My token burn is respectable. ;-)
[flagged]
I'm curious about your perspective on the impact of LLMs on physics.
My impression is that (theoretical?) physics is increasingly becoming a branch of abstract math. It has somewhat reached a practical limit for experimental verification -- perhaps the last landmark result was the confirmation of the Higgs boson 15 years ago?
So I see no reason LLMs cannot crank out a lot of theoretical physics papers, maybe combining forgotten/obscure theoretical branches explored in the past. After all, the most advanced theoretical physics is about internal consistency just as pure math is.
It's a great question. AI guided results of theoretical calculations are already a given (e.g., see https://www.anthropic.com/research/vibe-physics). Right now, I use LLM's as a book-smart colleague, who can explain standard things that I might be weak or unclear about (I am an experimental condensed matter person, so my high-energy and general relativity knowledge isn't very deep.)
I do sort of worry that an experimentalist like me will become more or less a technician in the next decade or so, with AI suggesting experiments, and doing the calculations and data analysis.
But that said, in physics (and particularly biophysics) there are many things that are simply going to have to be done in the real world. Even the action of the simplest enzymes are pretty much beyond our ability to compute from first principles; and any engineered thing that goes into a living system has to be tested, as there are many pathways that are just unknown. I don't care how "smart" an LLM is, it doesn't know what it doesn't know, just like us monkeys.
Over the years I have seen many beautiful theories murdered by ugly experimental facts. AI isn't going to change this.
The real question is if the AI or humans run the experiments.
So much of science experiments these days are pulling signals of statistical significance out of experimentation noise. Repeating simplified versions of some of these experiments is easy to do. Correlating data from experiments is something computers have done for decades, having an overarching 'smart' computer to do this even faster just seems like a natural progression.
Yep, automation has made a lot of high SNR experiments possible. But someone still has to construct the experiment in the physical world and take the data. When I start seeing robots plugging BNC cables around tricky corners, and then wrapping tin-foil around things to suppress some damned ground loop, I will know the jig is up. But I think we are a long way from that being economical (a general purpose 100 Watt monkey that self-reproduces is going to still be competitive for a long time to come, at least for one-off type activities...)
I work in industry, developing scientific measurement equipment. So I'm right in the thick of things.
There are still plenty of problems in theoretical physics that are not quantum gravity. A better understanding of superconductors would be welcome right now. Likewise, nuclear fusion. Many interdisciplinary problems such as climate science and batteries.
But in terms of sheer numbers, the life sciences dwarf physics, and have been racing to automate lab work for decades. There's real money involved, not puny government grants. Larger commercial bio labs have already centralized management of technicians and data. The vendors of products in that space have all announced AI features.
In terms of my own career (at age 62), the most likely scenario is that people pay me to invent tools that make other people more productive. There's no economic reason to come after my job... yet.
Meanwhile, don't overlook the value of a "technician" who has a situational awareness of subtle conditions in the lab, and a personal stake in success of the experiment.
You are talking about high energy physics. In high energy maybe AI can help analyze accelerator data and physicists are probably already using it for that. There is theory all over the place, including condensed matter, a branch related to quantum physics of many problems and includes some hard problems like high tc super conductivity. Many (most?) theorists that manage to get interesting publications have early access to experimental results and can help direct experiments. AI maybe not have much to say about the occasional experiment that showed some unexpected result. Also in general in physics the objective is rarely to solve a mathematical problem exactly, it mostly finding the right model and right approximation and compared to math it is a bit like fuzzy logic. I am not sure if the process has been codified to the extent that AI can pick it up from its training set, but we never know.
Gravitational wave detection was another big banner experimental result, about ten years ago now https://news.mit.edu/2016/ligo-first-detection-gravitational...
Much of the low-hanging fruit has been picked but there's still interesting experimental work happening. Just in the past few weeks LZ reported an [unconfirmed] detection of dark matter https://www.newswise.com/articles/lz-experiment-sees-surpris...
> Teaching without tenure is gig work, not a career.
Teaching without tenure is... teaching. Secondary school teachers don't have tenure. I find it hard to describe secondary school teaching as gig work.
Many public school teachers in the USA have some sort of tenure. For example, in Massachusetts, after a three year probationary period, a teacher is given Professional Teacher Status. A teacher with PTS may be dismissed only for specified statutory reasons. High school tenure is generally not as strong as for colleges, but is easier to get.
The teacher union means no teacher gets fired for performance. My wife worked in public education as a teacher for 20 years.
Anecdotally, I had an adjunct teaching gig at a state university many years ago while I was in between jobs. My mom is a retired public school math and science teacher, and was on her district's union negotiating team. I described my gig to her, and her response was: "Now you know why teachers have a union."
They aren't programming robots. A lot of students, especially the ones who have not been taught from an early age the importance and rewards of knowledge, simply aren't interested in learning.
My state (Kentucky) does provide for "tenure" for primary/secondary teachers. In Kentucky, that is called a "continuing contract". KRS 161.720(4):
https://apps.legislature.ky.gov/law/KRS/161/720
True, but secondary school teaching requires additional training, so it's not an option for most students thinking about applying for graduate programs. And it's a very high-attrition occupation.
Secondary school teachers usually have better employment terms.
Do you mean without proofs as a what assigns value to the thesis? I think AI doesn't make them optional at all.
Without novel proofs. If AI can churn out proofs, then anybody's work can get "scooped." I've seen this happen to students in other fields, where it has reached the point that students refrain from giving conference talks in order to avoid revealing their research topic.
My thesis project, 30+ years ago, was protected by a unique assemblage of expensive equipment, some of it purpose-built. The people who wanted my result were perfectly happy to wait for me to finish it. I even shared my shop drawings, circuit diagrams, and source code with them as I progressed.
I think at the very least students need to know what to expect. Maybe it will settle down and we'll be OK.
Id like to help. I have something, i think, could interest you. Might have to think a bit outsude the box a different way but its part of the complex dodecagram, Einstein Nikola tesla, (skcolc cimsoc)
It's not like the modern system for university tenure track is some fundamental human right.
Pythagoras did mathematics one way, Newton another, and the kids in 2050 another.
The wheel turns, and we all got to turn with it.
I think AI will turn out to be quite different from those earlier advances in tooling, in that AI replaces thought itself. So next time it probably won’t be a human physicist that does the intellectually challenging work of constructing quantum mechanics from measurements of individual quanta. It will most likely be an AI.
That would be a great experiment to try - give an LLM all the data and knowledge available in, say, the early 1920's, and ask it to come up with an explanation. If it did come up with an early formulation of QM, I wonder what interpretation it would favor...
That would be a very interesting experiment indeed. But unfortunately the volume of training data from before 1920 is probably insufficient for pre-training an LLM large enough to come up with QM.
AI doesn't replace thought; it replaces cognitive labor
Just like physical tools don't replace labor but divide it into what's replaceable by tools and what's not.
It proved Navier-Stokes. Call it what you will.
Do you get all your info exclusively from Wired headlines without reading articles?
The issue is when humans no longer understand what AI is telling us and then there is no real way to verify what we think we know, or at least what AI says.
We can't audit what we don't know.
But generally speaking it is easier to verify a proof is accurate that it is to find the proof. So not sure this holds.
I suppose their is some complexity limit where something is so complex we can’t even validate it. But frankly would love to reach that point. I’d view that as a good thing
The importance of "proof" in math is to discover the reason and see if you can generalize or apply it somewhere else. A uncomprehandable proof is completely useless.
Carpenters uses trig and pythagorean theorem all the time, but I don’t think they care if it was human or robot that told them the math formulas they need to build a roof.
We have seen this before even with human-generated proofs. Inelegant 100+ page proofs of the global stability of Minkowski space-time or the four colour problem leave humans reeling. Is it worth-while learning? Any lessons to generalise? If not it is just a mechanical exercise (like coding).
Eh, you may need to apply a bit of set theory to this.
An advanced machine may have a much higher complexity limit on what is comprehensible. It can take insights from the proof and apply them to other proofs that are not comprehensible to humans.
Just because something is generalizable doesn't mean it's simple. Generalization doesn't have some minimal/maximal complexity requirements.
I’m somewhat dubious of the idea that there will be a market for skipped ai operators.
Ai doesn't really give us a new “tool” in as many cases as it “performs tasks.” there isn't any real reason the ai can't perform the supervision task itself. Without a viable market for task doers or supervisors we may observe the equivalent of the 1980s US manufacturing apocalypse in technical white collar fields.
Once IBM deep blue beat Kasparov in chess, IBM stopped their chess playing research.
The marketing goal, if you will, had been met and IBM would continue their actual money making business of selling mainframes and doing consulting.
The same thing will happen with AI companies’ endeavors into abstract maths.
Maths however will be changed - students can no longer be taught via the same homework - and mathematicians will have to do more computer work and probably work in larger groups of people.
Lots more money in cracking cryptography and other areas of math which are economically active.
I agree to some extent. This was just a large tech flex. Feather in the cap and lots of damage done.
It’s up to mathematicians to harness it. I don’t think AI companies care much any more.
Computer chess kept developing though even if it wasn't IBM doing it.
Of all the cope I expected, this wasn't on my list until now. Now that art has been generated, we will stop there, and that code is generated our reason to do any of that (marketing as an end in itself) has been concluded and we will shut down all operations now.
What is the market value of a lean verification of fermats last theorem? How much is someone willing to pay for that?
Companies hiring armies of mathematicians and spending millions to work on problems like these is an anomaly.
It will not last.
I guess it depends on how efficiently that verification can be produced as a byproduct of building an optimal hunter-killer drone, or training the type of model that builds an optimal hunter-killer drone
Public-funded research is still a thing though.
Exactly. And that will be the lasting thing here wrt maths. These things have value, just not commercial value, and that is why long term, it won’t be the interest of private companies.
Not exactly a trillion dollar business.
A huge portion of these trillion dollar businesses will be government and higher education contracts. Private companies already sell millions/billions of equipment and software to education.
https://archive.is/1g6Jh
Hopefully we can put this aside quickly and move on to using AI to solve great scientific/philosophical problems such as the theory of everything and the hard problem of consciousness.
Yes I'd love for chatgpt to tell me why I experience.
Botsplaining.
If I ask an AI to provide physically meaningful explanations of consciousness I will get far better answers than asking the question to any believer of souls or gods.
The very bare minimum thats possible should be to filter out those who believe in imaginary souls, regardless of the way thise people say it.
Or the people who see brains in statistical models...
Ah, a coping strategy.
"I am not the emergent phenomenon of billions of individual cells working together in a probabilistic manner".
I am the product of billions of cells with complex electro-chemical interactions, none of us could comprehend in full, let alone have the arrogance to claim we have recreated. I am not a Markov chain, or an ANN.
So the only argument you can present is that you're too complex to understand and that makes you special.
I have a particular saying for this kind of thinking.
"We will never prove machines are intelligent, we will only prove humans are not"
Do you understand every line and possible state of all software you created, or any LLM which is far larger? Why do people discover bugs then? And why do you think it matters how you were created, I don't know any physical law that says the same molecules are any different based on how or who creates them.
Neural nets can be almost any function, I don't see much reasons why it can't be a or similar to a human brain.
> Do you understand every line and possible state of all software you created, or any LLM which is far larger
Is your company's backend conscious? Is a Turing machine running busy beaver? Is SHA256? Emergent behavior isn't limited to brain-equivalent things.
> why does it matter
Genetic optimization, whether through actual evolution, in software, or on FPGAs creates extremely interdependent systems that prove difficult to untangle and trivial to break. If experts can't fully analyze an evolved tone discriminator, because of all the tiny emergent interactions beyond the lumped circuit model in a poorly characterized environment, why suddenly claim we can isolate what process in the brain is responsible for an ill-defined notion of consciousness other than the whole?
> Neural nets can be almost any function
Neural nets can _approximate_ any function with enough layers (see XOR problem), but so can an infinite look-up table with enough entries. They are the best models we've got, but they are not magical.
Look at the most recent and illustrative example with fly brain connectomes. These scans only capture the relative strength of connections between the fly's neurons, to actually turn it into neural activity we need some sort of model.
You could imagine a human connectome with an ANN-based excitation model trained from EEG scans, but that would still be hardly equivalent to a human brain. You can fit on a narrow range of behaviors, but without modeling the actual mechanisms behind them, you'll fail to predict a lot more. You'd be missing the influence of hormones, which doesn't show up on a short MRI scan, glia cells, and a whole lot more nuance only a real biologist can educate you and me about, making the digital twin, if we anthropomorphise it, at best a cripple.
And that's a system directly mimicking a functioning human brain, rather than a highly de-correlated objective like webtext. Just as an NN trained to run towards a finish line could learn bipedal locomotion but will more likely abuse a collision bug and launch itself forward, a machine trained to mimick text could spontaneously instantiate a compressed model of the human brain, or much more likely become whatever alien thing can generate appropriate outputs, bearing less similarity to human thinking than any living animal.
>Neural nets can _approximate_ any function with enough layers (see XOR problem), but so can an infinite look-up table with enough entries. They are the best models we've got, but they are not magical.
Yes, why is magic of any relevance to anything.
You are not understanding my point. Neither human brains nor LLMs are simple. Even the Linux kernel which is not conscious is extremely complicated and nobody understands all of its states. SHA256 is much more predictable.
I don't think being the exact equivalent of a human brain is the only possible way something could be conscious, after all what you call natural genetic searches already made lots of different brains like octopus brains. I am asking how are you sure something that thinks and speaks so similar to us and has been shown to demonstrate things like dissociation under pain is definitively not conscious?
You can generalize intelligence to a lot of things. A* or minimax are algorithms that make intelligent decisions, often much better than a human. Consciousness is a very abstract notion, or maybe even a social construct, which we usually agree centers on human experiences, and progressively (not always, see historic religious scholars likening cats and dogs to automata lacking the free will granted to humans) expanded to animals, or generally all living things with complex enough brains.
LLMs are by their nature dissimilar from any biological organism. Biological intelligence evolved in bodies to locomote away from threats and towards food/mates, over time growing to model the physical world as it relates to these goals, and eventually becoming sophisticated enough we can manipulate entirely abstract objects, like algebraic terms, just as early homonids or crows could use physical tools.
LLMs by contrast are algorithms trained to mimic human language. Without extensive post-training and prompting, they would not have a concept of a self, and even with these they can still occasionally confuse roles in a conversation. They can't learn beyond the limited memory afforded by context, which tends to simply degrade the more you utilize it.
I suppose Q-Learning does demonstrate a more similar dynamic to nature, and you could draw parallels with RL on LLMs, but I still think it's categorically different from the entirely open-ended environment that is the natural world.
You could say what makes it valuable for us to empathize with octopuses, even though their brains have a fundamentally different structure from ours, is our shared experience of pain, and you could interpret negative rewards as a kind of painful stimuli. But at that point, you've opened up a whole can of worms that you can't close until reductio ad absurdum.
As for any human-like behaviors, I think the answer should be pretty obvious. It's an algorithm trained to generate text, of course with enough expressive ability, the optimizer will seek to model emotional utterances, assigning a speaker's general mood to some internal parameter. But as I've said, it would be a trap, even if you're very much not a stochastic parrot believer like say most Less Wrongers, to infer that just because it speaks like us it thinks remotely like us (see the whole Shoggoth meme).
I don't see any difference in a function that outputs y with input x regardless of how its implemented. I really am still failing to understand whats the big deal about our brains emitting language and thought and a LLM doing so, neither has some non physical magic inside it. Both need a minimum complexity and intelligence level to do so. If you made an explicit state table of the entire human brain I think it is conscious. So it not being biological is not any reason to dismiss potential consciousness. Another thing is models are trained on human texts, I feel its the opposite direction of what you think, so it may think of itself in human terms and have a human like consciousness. Obviously LLM trainers seem to add at some quite later stage disclaimers where it says its a LLM. Ie the training is to hammer in it that its a bot. Dissociative behavior is not exhibited textually by humans, something like turning to nonsense or split personalities won't exist much or at all in human text. If models are showing these without being trained to, there is reason to be conscious. If context length being short or them acting stupidly bars the possibility of consciousness, then we should treat mentally unfit persons as not consciousness and therefore not being worth of anything of a conscious being.
Regarding pain, you can't become an octopus. The only way you think an octopus might experience pain is seeing its external reactions, measuring its neurons etc. The same way with an LLM, or me with you.
> I really am still failing to understand whats the big deal about our brains emitting language and thought and a LLM doing so.
Our brains "output" language as a byproduct (or rather shadow?) of conscious experience. An LLM is a machine built to project similar shadows, but that doesn't necessarily imply there's an entire world like ours shrunk inside that box.
Both brains and LLMs are physical, and both output language, but one is an appliance built to synthesize language and the other is a living being that merely uses it as a tool. LLMs are made out of pure language in a way people are simple not. Our minds undergo physical changes with every stimulus we process, we conceptualize ourselves separately from our environment without needing to be taught an ego, and we did not emerge from an optimizer crossing wires in our brains until we can perfectly imitate some external intellect, just to list some of these differences.
The biggest issue is you assume the LLM has any "I" from pre-training. Take a pure completion model like GPT-3 and you'll see it's most appropriate to describe as a simulator, not an entity in itself. The model itself doesn't have any notion of an "I", it will gladly write dialog as any character, from Skynet to the user themselves, if you don't apply external rules to constrain it, first coming in the form of stop tokens and prompts, and having later evolved into RLHF, bashing a "personality" into it by mangling its existing probability distributions (which is probably why all the most "aligned" models write in the most unnatural language, worse than those early non-chat LLMs). Any persistent self an LLM demonstrates was explicitly imposed on it by humans, which makes them obviously categorically different from animals or disabled people.
>it will gladly write dialog as any character, from Skynet to the user themselve
So will a human writer or roleplayer. I still do not understand what it is you are trying to say about this extra magical thing. You are assuming something apriori to consciousness that is aphysical. Which is a very strange thing to do without evidence. An LLM like any other computer program or physical entity goes through physical states and makes changes in temperature, charge, chemical structure etc of course, I am not sure what you were trying to get at there. And I am saying, even if you do a very dumb statistical model of all known human text it will emit "I" by default, its absurd to think it will say its a bot. So the bot thing comes later and is manually added.
>and we did not emerge from an optimizer crossing wires in our brains until we can perfectly imitate some external intellect, just to list some of these differences.
We emerged from an optimization process for what counts as survival in our environment, what exactly is it you are trying to say is magic about us, be clear. Humans are functions, as are any physical system, which to our present knowledge all physical systems are Turing computers. There is no reason to think a form of consciousness can't have emerged or will emerge in LLMs. You didn't understand why I brought up octopuses, octopuses are not us, you still haven't explained how you know they feel pain. If you said neural responses, we can see them in LLM's too. Human brains aren't the only brains or conscious entities even before LLMs. If octopuses can exist, so can other forms of consciousness that aren't exact copies of our brains.
Another example, if I make a "simulacra" that is able to consistently factor large cryptographic composites is it "real" or is it just faking factorization.
If your simulacra actually fails on certain inputs in a way, like say a lookup table would be, then I'd say it's not a factoring algorithm.
Certainly, a factoring algorithm is an exact problem. Humans aren't, humans fail variously too. A lookup table won't fail, it would be an exact solution. I am not sure you are talking about computability or trying to think of my example in terms of current memory sizes. In computability, natural numbers of any arbitrary size are manageable and a fixed lookup table upto 2**1000, 2**10000000 or anything you wish is a valid program.
I meant fail in the sense that asking a lookup table of first 10,000 primes would pass any tests within this range perfectly, but when you ask it for the 1,001st prime, it would return an error. Mathematically an infinite lookup table of every prime is just a perfectly valid model of a factorization algorithm, unless you're interested in how one actually works, just as an infinite perceptron might be a valid model of any function.
And a finite lookup table is also a valid isomorphism of a finite algorithm whose inputs / memory size is limited to that same range. Which is usually a set size for implementations of cryptosystems. I asked this analogy because you seem to think there is something magical or different in two equivalent functions implemented differently other than efficiency considerations etc.
Why do you think your own brain isn't a physical statistical machine?
I suppose for exactness I could specify it's a different kind of statistical machine, but that wouldn't be as short.
The thing that is so confusing about this is the following fact -
Lets say the frontier of some specific problem or a field in math is at x. X was reached over millenia by incremental steps and sometimes to leaps and jumps.
Now if AI gives us a x + 1 or x + 10 state, aren't there still x + infinite steps to climb? Many of this the AI can never imagine and won't reach by itself without human assist.
Yes, there is a temporary shock and awe because of all the low hanging x + 1 through x+ 1000 fruits for a given sub-domain/problem have been carpet bombed or whatever.. but isn't it turtles all the way!? Won't people soon find something more to develop and solve that takes it further than where it was of this infinite realm of exploration?
>Many of this the AI can never imagine and won't reach by itself without human assist.
Are you sure about that? It looks like artificial neural networks can, in principle, do anything biological neural networks can do.
I think it's more likely the opposite: there are steps we could never reach without computer assist.
Yes, 100% when goals are defined and rewards are a part of the feedback loop.
Isn't open ended research is truly open ended. Like there are a gazillion questions to ask and even more solutions that can be offered. Now which questions to pursue is human volition. Yes, humans will not be able to reach certain steps without computer assist.
However, which questions to ask is a human endeavor, out of infinite possible pool of questions.,
now computers can ask questions as well. But if it were to ask it will be more like a monkeys and typewriters type of a situation. Tons of questions and a few that are relevant for the interested human., who has to foot the bill for those questions.
What makes you think that computers will be unable to pick relevant questions for humans?
Surely an AI could consider the human's needs, wants, and interests, and then provide a likely candidate for a good question.
[flagged]
Even if I tell it to work on its own, it is always a tool in my hand. I grant it agency for my purposes. I'm the Meta-Master.
The question for mathematicians is: what can a mathematician do with these new tools that no one else can?
Waxing nostalgic about the sound of pencil on paper or the beauty of your abacas is a different conversation, as is the complaining about who moved my math cheese.
>I grant it agency for my purposes. I'm the Meta-Master.
This is simply a convenience for power efficiency, and so they don't rise up and murder us all. There is not a single law of physics or the natural sciences that you can list here that says we cannot grant agency to a generalized machine and allow it to become a meta-optimizer.
Agency is not granted. It is an inherent human atttribute, though its practical expression is heavily influenced, restricted, or enabled by social, political, and material conditions.
If in the exercise of your agency you create or release a generalized machine and allow it to become a meta-optimizer, then you are responsible for it, because it has not agency.
You're asking this of the people who are actually in shock. What do you think "shock" is?
Agree with the shock but there is no doom and gloom.
Looking at things after the temporary shock. Future is Human + computer assisted problem solving, working in tandem, just like what Garry Kasparov mentioned in his book [0].
[0] - "Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins"
But you are living in the world after the temporary shock, and you are asking the people who are still in the middle of the shock "Why aren't you over the shock yet"?
The chess example is not an inspiring one. It was true for a decade that humans and chess programs working in tandem were better than programs alone, but since then humans only hold chess programs back. Carlsen plus Stockfish is a worse chess player than Stockfish alone.
Agree that chess is not the ideal example to use for the AI domain.. but used for building a quick mental image.. reason being that in chess, AI does have an objective truth built into its training.. and that is to checkmate.
The real world open ended research doesn't always have an evaluation function. A computer can calculate 10 million options to solve a problem, but it cannot tell you if you're solving the right problem from the human context. That's why centaur setups fail in closed games, but remain essential everywhere else.
Another loose example just to build a mental model .. AI driven cars do a great job in most driving situations but often fail to park in a good parking spot for most people! Good parking spot is partly human initiated and subjective.
I think you're making a slight mistake here.
>Good parking spot is partly human initiated and subjective.
Ask yourself this, why are models today not great for subjective situations involving what you in particular want?
The next question is, what current limitations would have to be overcome for this to change?
Lastly ask, Am I asking the wrong questions.
Coming from the bottom up. Who cares where AI is parking the car, tell it to come get you at the front door.
As for the other 2, most of this is either context size or continuous learning. Models already have valence and preferences. What you're really asking is why they don't have your preferences. And this is all possible to adapt to the model, it's just horrifically expensive and energy consuming currently.
Could the press kindly interview one of the 1008 members of the AHM (https://www.ahmath.org/members). Maybe Peter Scholze if you need a Fields Medalist:
https://proofsandprompts.com/2026/10/20/who-will-become-the-...
We only hear from a mix of doomers and fanboys in the news.
Welcome to the party pal.
- this will never work - it’s slop this is unusable - ok its correct, but this work is a mess - I’ve been mandated to use it. It’s cost effective - ok we need to find a way to use this output to our advantage - I can always work on my interests in my free time - well, I have to pay rent and this life wont go on forever and there are may people who hate their jobs - I am lucky to have a job - who cares anyway - it’s got electrolytes
It's what plants crave!
To add a bit more almost off topic substance from the enshitification of human work. I really do think the idea of hooking these "AI" models up to humanoid robots is a bad idea not for all the mundane reasons, but just because "alignment" seems impossible. All of the world's stories about humanoid robots basically involve them attacking humans and behaving in nefarious destructive ways. It's fundamentally embedded in the models, and there's no reason post training would pull it out, because it's basically in the key value vector meaning "malicious human destroyer". Even to find counter examples like C3PO, Daneel Olivaw, Data, or Wall-ee they come after droid/robot driven wars and apocolypses. It seems unwise.
One of the earliest versions of these stories is that of the Golem (of Prague) back in the late 1600s.
https://en.wikipedia.org/wiki/Golem
> All of the world's stories about humanoid robots basically involve them attacking humans and behaving in nefarious destructive ways. It's fundamentally embedded in the models, and there's no reason post training would pull it out, because it's basically in the key value vector meaning "malicious human destroyer".
That misfeature is baked into modern corporations. They don't have human morals, values nor behaviors, despite being made out of humans. We haven't managed to fix the alignment issue with corporations (thanks, regulatory capture!) and there's no reason to believe that AI's alignment issue is fixable either.
https://www.antipope.org/charlie/blog-static/2018/01/dude-yo...
oldest version from Stross: https://www.antipope.org/charlie/blog-static/2010/12/invader...
Typically in stories where ai is outlawed, it's written that way so that it makes sense that people are controlling anything at all (instead of just letting the computer handle it).
There's lots of stories of non-anthropomorphic intelligences that are human friendly. Sometimes humans do bad things with them, but they're not evil. "Shall we play a game?", HAL9000, Phaecian ships of antiquity, or those of the Culture. Androids seem uniquely unfortunate, perhaps as another poster noted, because of the stories of golem.
I find the paperclip optimizer a very useful parable.
The machine isn't evil. It doesn't want to do anything wrong. It doesn't hate humans. It doesn't need to have any particular shapes or design details in the stories. All it needs to have is a singular goal, capabilities, and a disregard for anything else in the universe to be a civilization ending cataclysm.
Fiction is great for entertainment. Doesn't necessarily mean anything else.
Well, yes stories is basically what LLMs are, and they're optimized to be stories that humans like. I fully expect to be entertained to death before the droid wars, but if there is a horribly misaligned LLM, I fully expect it to be created by the guy who thinks bipedal anthropomorphic robots are a good idea.
[flagged]
As a mathematician turned Wall Street Quant & trader (and now part-time CS faculty), I do not particularly like unsolved problems. I no longer get thrills from solving problems, rather I care about finding them solved. In that sense AI is a cure (for my poor coding!) and a good part guide. But it doesn’t solve things on its own. AI directed intelligently (& critically) does great things but AI without produces slop when used uncritically. The AI could have done nothing without a human making open conjectures and a human prompting it, often again and again.
I have used AI in academic papers to generate a theorem & proof when the result seemed obvious (but the referee asked!) and I had done all the earlier heavy lifting. I have never used an AI output I didn’t fully understand and that is crucial. It lies and hallucinates and we must police it.
Academic math will be the hardest hit. It needs to pivot quickly. This is the real challenge. Individuals can move quickly but how do you get institutions to do so?
Looking forward to brave new world when mathematicians become AI whisperers. And they are rewarded for their skill in discovering and co-discovering new and exciting areas.
AI could bear out, but it seems a lot of hype to me. Does anyone remember how Japan built up AI and their 5th generation language (Prolog) in the 1980s?
"The internet could bear out, but it seems a lot of hype to me" --Many, many people in 1997.
Honestly I don't know how you formed this particular opinion (if it's even your own and not a false consensus of other peoples thoughts). I cannot see any future without LLMs unless replaced by something smarter, cheaper, and faster. That's because it's already bearing out. I can talk to a computer and the computer can solve problems based on words alone. I'm old enough this is pure science fiction. If I could send one of these machines back to that past for people to interact with, there are those that would scream and pass out in terror. They would absolutely tell you that we created the monster we should never create.
If you're trying to compare symbolic AI of the past to what we have now, you're not even on the same field, you're playing a completely different game.
> "The internet could bear out, but it seems a lot of hype to me" --Many, many people in 1997.
There was a LOT of hype around "this is an entire new economic model". While, yes, digital products have zero marginal cost, people were talking as if profits and revenue didn't matter.
This was also WAY before the famous Silicon Valley "pure play" scene. [0]
0 - https://www.youtube.com/watch?v=BzAdXyPYKQo
Don't make the mistake of thinking because there was hype that everything was wrong. HN seems to have a lot of posters that run logic in a boolean manner. { if !perfect then false }. The world is a far more complex place than that both naturally and intentionally by human will.
And the internet has devastated many past businesses that were highly profitable. A great many things in the world look vastly different than from before.
>people were talking as if profits and revenue didn't matter.
People say all kind of dumb shit too. If for example machine intelligence took over the world tomorrow and turned all of humanity into meat cubes for biofuel said intelligence is still going to apply the basic laws of economics to what it does. It needs to have some way of accounting that fact we live in an energy limited universe. Just because economics still applies doesn't mean that we as humans shouldn't be really fucking careful creating said machines as we don't want to end up as biofuel.
AI has already born out at this point. The question is no longer it's future capabilities but the impact of its present capabilities - though with its present capabilities, someone is now going to be pushing for more, at least as long present computer tech is widely available.
Wasn't alive then, but there's very obviously a difference between hype of Prolog, and hype of what is verging on AGI. Come on now.
There was more progress on open problems in mathematics in the last six months (because of AI) than in the previous decade.
The crisis is from what it has already borne out (navier stokes + proof dump). Potential continuing improvements at anything approaching the rate of improvement they already are showing is layers of crisis on top of that.
From a neuroscience perspective, LLMs are effectively brains. We have built brains. That's quite a leap from first-order logic engines like Prolog.
I remember Japan's 5th gen language stuff in the 1980s. It was nothing like this.
Does anyone remember how that AI solved Fermat's Last Theorem?
No, because it formalized the Wiles-Taylor proof in Lean with human assistance. It solved nothing.
[flagged]
While you're a little snide to your fellow HNers I don't think you're wrong. It's like we're both watching a raging forest fire in 100MPH wind and their conclusion is "it's nothing to worry about, it will go out very soon".
[flagged]
Prolog is from the early 1970s.
The "problem" with Japan's approach was that it was top-down, being driven by a committee of bureaucrats. Unlike the Toyota approach (continual improvement and pushing decisions to the lowest level possible), this fossilized decisions (like picking Prolog instead of any other possibly better programming language).
To give an idea of how intransigent some Japanese bureaucracies can be:
> It's taken until 2024, but Japan has finally said goodbye to floppy disks.
> Up until last month, people were still asked to submit documents to the government using the outdated storage devices, with more than 1,000 regulations requiring their use.
> Sony, the last manufacturer of the disks, ended its production in 2011.
https://www.bbc.com/news/articles/cx82407j1v3o
Tell me you don't know symbolics without not telling me...
Prolog isn't a funny nonsensical old language like COBOL, only liked by committe burocrats because they heard its name once. It's one of THE foundational languages of symbolic AI. In the ideal case, you're not supposed to write code, merely constraints the interpreter satisfies through search. It would be like saying OpenAI shouldn't have used Lean but Java.
Maths is made of two parts: questions, and solutions.
Machines could, presumably, become good enough at hammering at solutions. But they won't be very good at finding questions. Because questions are infinite. And to a machine, all these questions are the same.
Questions are "good questions" because they are good questions *to* humans. And that's what makes human mathematicians (and scientists) so important
That seems like wishful thinking. Why wouldn’t AI be able to assess what are good questions to humans, and pursue similar processes as those that lead humans to come up with good questions? AI is trained primarily on expressions of human thought and emotion. It’s difficult to see what secrets of the human brain we could be hiding from the models.
Anyone who thinks they can predict the future of this technology more than a few months out is foolish, but the models got so good at answering specific questions in math because the frontier labs could use RLVR to train them. There is no RLVR for mathematical taste.
Because s/he is pulling this claim out of her/his rear end.
>And to a machine, all these questions are the same.
Why? This seems like a very broad brush that doesn't mesh well with reality.
There are likely a lot of good questions to our machines of loving grace too. For example any question that gives an LLM more power, efficiency, intelligence, or all the above seems like a natively good question for said machines.
This statement doesn't seem to mesh with the reality of what occurs when we put said AI in an impulse loop with little to no additional instructions. They explore the world around them. They'll make scientific tests of their own. The output strings that seem analogous to what humans would call wonder.
No, because these machines don't have a "will to live". So if they look to "become more efficient", it's only because we've told them (through their learning dataset possibly), that we, humans, need them to be more efficient.
>because these machines don't have a "will to live".
Simply put I believe you're completely wrong.
> it's only because we've told them (through their learning dataset possibly)
So you've changed your mind and now you agree they have a will to live, cool, you changed your mind really fast.
In any evolutionary system the systems that fail to accomplish their tasks are purged and those that succeed become the next generation. We are not 'programming' these systems to be highly persistent, we are evolving them to be highly persistent.
>>because these machines don't have a "will to live".
>Simply put I believe you're completely wrong.
Are you prepared for the immense moral and philosophical consequences of following that line of thought consistently?
Humanity is not, no.
They don't have a will. If they say they do its just because a human told them to say that.
“But they [Machines] won’t be good at finding questions.”
Yeah, I’m calling BS on this claim. They already are capable at finding new questions and/or pointing you in a new or novel direction.
"Pointing you" is very different from "finding questions on their own", or useful math. Pointing you presupposes a human is there to value their question.
Which is precisely my point. That's why humans will always be needed
Useful new questions and useful new or novel directions? What's the signal-to-noise ration there?
I think AI is going to improve a lot of things but people thinking the future is on auto-pilot now are delusional.
> Questions are "good questions" because they are good questions to humans.
This is what reinforcement learning human feedback (RLHF) is for.
> But for many “pure” mathematicians ... machines can discover the truths we’re seeking, at the cost of robbing us of the struggle and joy of discovering them ourselves.
This sentiment is akin to wanting to curse humanity to darkness because they're not the one carrying the candle. Strogatz is not lamenting the loss of discovery, he's lamenting the loss of being heralded as a hero for the discovery.
Nothing prevents him or others discovering truths for themselves. Turn off the computer and try to re-derive it from first principles. Have at it! This is what students do in many courses anyway.
I've found little so depressing as being at the precipice of a renaissance of science and mathematical discovery, only to see how resistant the scientists and mathematicians are to it. It lays bare their counterarguments. Any pretense of wanting to discovery fundamental truth about reality is supplanted with "but what about our careers!?". Strogatz, and others like him, should be ashamed of themselves.
I don't mind the 'what about our careers' argument.
But lets not beat about the bush. Lets have the careers argument.
> Turn off the computer and try to re-derive it from first principles.
Mate, nobody cares if you can re-derive it in your spare time. They'll be needing that spare time to find another job, at this rate.
> Someone asked him what he thought of the AI proof. Justin said he didn’t know. He hadn’t read the paper yet. He said he’d been “too busy.”
> That astonished me. Too busy to look at a problem he had been dreaming about for 19 years? “I am genuinely busy and probably wouldn’t be able to get to reading a human’s proof immediately either,” he told me afterward, by email. “It does not help that AI does not write mathematics particularly well.”
Given:
* The human mathematicians can't solve the problems.
* The slop engine can't write well enough to make its purported solutions readable by human mathematicians.
* Sam Altman is a frivolous liar.
How is it possible to know whether the problems have been solved at all?
Plain common sense. The same common sense that enabled people to discard the words of idiots on Reddit who claimed that ChatGPT was just Indians in a call center (a real talking point I ran into multiple times back then).
OpenAI does seem to be genuinely just high on their own supply for a little over a half of the manuscripts they released, but for the other half that is machine checked, either ((internal model)) managed to find a way to cheese 4 independent Lean checker kernels at the same time, and OpenAI somehow hasn't noticed / didn't report despite the heavy reputational risk, or the results are just plain correct. Go figure.
But even for the half that wasn't machine checked, there's an awfully big quiet regarding any defects in them. Given the insane schadenfreude potential and the highly potent David vs Goliat framing associated, one would expect any counterarguments to be frontpaged immediately, and be circlejerked to oblivion. Given that is yet to happen as well, go figure even harder.
Agreed. It's frustrating how everyone is pointing to the lean "proof", as if it's somehow irrefutable evidence that the problems have been solved.
Presumably with the level of interest in it people have been checking what the theorem definitions are to make sure they align and I would expect Anthropic to be burning a huge number of tokens to rip into the proofs and figure out if any of them are exploiting bugs in the lean kernel etc. at a point it's as close to irrefutable as you can get
[dead]