How does this relate to the more recent work on the M4 ANE found at https://maderix.github.io/articles/ ? Does the M4 and later ANE expose any additional capabilities, or is it just a higher-performance iteration of the same thing?
As an aside, the introduction to this article seems to conflate the ANE with the Neural Accelerators (NAX) found in the M5+ (and A-series equivalents) GPUs. These are very different things, and Apple is still working on the ANE - the M6 and A20 will apparently feature doubled ANE blocks.
It's worth remembering that Apple is releasing a new framework (Core AI) this fall that goes beyond the Pytorch and Tensorflow workloads that the decade old Core ML framework allowed.
> Core AI allows your app to use the latest model architectures and inference techniques across the CPU, GPU, and Neural Engine.
A lot of folks are/were saying that Apple has missed the boat when it comes to AI, in some (important) aspects that is correct, but I think it's worth remembering that Apple added the Neural Engine to A-series chips in 2017, before the AI hoopla really kicked off:
"AI" has grown much more since then, and there have been important developments that Apple has not deployed (well), but I think they were looking ahead a little more than most at the time (even if events 'got away' from them subsequently).
IMHO, I think Apple is poised extremely well. They didn't blow billions of dollars chasing models that are becoming commoditized. So many interesting models can now be run locally. All the big AI players have to pay even more to run those models when Apple will happily sell you the hardware, and you pay for the electricity. There will always be a place for some many-billion parameter model but as time progresses I think fast local models that keep data on site will always be a valuable, and Apple will happily sell you something to run them.
It may not matter that they're "sleeping on it". AI is going to disrupt all tech companies.
Clearly software and SaaS has no moat. I just saw someone vibe code a fully functional Photoshop in a week of Astra use. I'm thinking even big tech is not immune to this. Even the mighty infrastructure players.
Vibe hardware is just getting started, and I'm hoping we see competitors to iPhone and Android and MacBook. It's less expensive to build and test things now, which might lead to a Cambrian explosion of new hardware startups.
The world will greatly benefit from this.
It's funny that American tech giants invented AI, because AI is going to unseat American tech giants.
SaaS' moat has always been someone else worrying about the hosting/upgrades/management part for you. Stores still sell sliced bread even though bread machines make bread for you and everyone already has a knife. People often buy convenience, even when they could do something themselves.
> It's funny that American tech giants invented AI, because AI is going to unseat American tech giants.
Ah yes, let's ignore that 40 years of research done on neural networks and give all the credit instead to the tech giants that came in as soon as there was enough of a breakthrough to start making them useful in language tasks. Definitely the American tech giants and not the Canadian and international researchers who spent their lives building the foundations. /s
Apple's Neural Engine is equivalent to the "NPU" hardware you would find on many cheap ARM SBCs. It's a dedicated coprocessor with a very low performance target, not intended for giant transformers or LLM acceleration. These NPUs are not particularly hard to design, and their critical flaw is that they don't scale very well. CUDA "won" because a bigger GPU meant having an utterly massive amount of CUDA cores to delegate ALU work to. NPUs/Neural Engine has the opposite problem, where spending $10,000 on an M1 Ultra only gets you ~2x better NPU performance versus the baseline $600 M1 chip. NPUs and Neural Engines are essentially dark silicon on the majority of devices with them, their uses are few and far between.
Considering Apple's refusal to sign Nvidia's ARM/CUDA drivers, it is pretty clear how Apple missed the boat here. Apple Silicon could have dominated the datacenter rollout if macOS supported CUDA properly. The Mac Pro would probably not have been cancelled if the PCI lanes could be used for normal datacenter GPUs and CUDA workloads. The excellent Thunderbolt bandwidth present on so many Macs is wasted supporting RDNA but not eGPU enclosures. There are several hardware features that Apple holds back for no good reason, handing Nvidia the lead in certain markets.
The only thing that stopped Apple from riding AI to the top was their own petty grudge towards Nvidia. Simple changes to macOS would have destroyed Nvidia's Grace CPU sales and made Apple Silicon the crown prince of the AI boom, at zero risk to themselves. The only thing Apple really needed was their own Mellanox equivalent.
This isn't ai slop. It's fascinating and well written.
But I learned something really basic - i didn't know that the ANE (and the data pipeline around it) was designed for CNN rather than transformers. It's always been an open loop in my head, wondering why the ANE was less impactful than i understood it should be.
>i didn't know that the ANE (and the data pipeline around it) was designed for CNN rather than transformers.
Multiple stories have reported that ANE came from Apple's self-driving car project that got canceled. (Makes sense since CNN is used for vision-related machine learning and enables cars to analyze their surroundings.) They spent 10 years and ~10 billion on research & development on a product that never got released so Apple is probably happy they're able to salvage some of that ai technology and put it in iPhones and Macs.
This is pure sunk cost fallacy. CNNs were from the deep learning ImageNet heydays, but now everything in that domain is equally done better by transformers. All you are doing is wasting area and saddling software with outdated hardware, and myopic PMs insisting on its use will create inferior products. Now that sounds a lot like the Apple AI efforts..
I designed and deployed several custom CNN-based AI models to ipad hosts around 2018-2020. It was a finicky process - CoreML didn't give you much control (or even feedback) on what parts of your model would run on the CPU, GPU, or ANE - but you could usually tell right away when they didn't, just based on big latency differences. Performance to power (and size) ratio was pretty good and I was impressed at the time, despite being generally not really happy about working with the apple ecosystem. I would've preferred something Android (or Linux)-based but there wasn't really anything comparable out there back then. (SBCs like the Jetson would've worked, but we would've had to figure out how to add an interface for human operators which would've been a big lift compared to just using off the shelf hardware).
A lot of neural engine, particularly in the embedded domain (ARM/RISC MCUs) have the same problem. Designing other models means on top of this means a lot of profiling to get convolution blocks right to get good speedups. (We optimized this in the past e.g. using Neural Architecture Search on super networks)
Also, just imagine being the group at Apple responsible for designing this section of the chip, starting probably almost a decade back – under the constant uncertainty of not knowing what direction ML workloads would develop in…
> ML research was a rather known quantity, or the separate "Neural Engine" CPU explicitly aimed at existing ML pipelines wouldn't exist.
No, not really. Transformers were just one of the possible directions. Silicon design does not have the same time scale than software. Now, everyone is using transformers so it becomes harder to do anything else, and it’s been the case long enough that hardware had some time to align (but is still lagging). But who’s to say that a different architecture published last year won’t take the world by storm 2 years from now?
It’s easy to say it in hindsight, but transformers took a bit of effort to get where they are now.
> Silicon design does not have the same time scale than software.
"Neural Engine" has been a part of iPhones since 2017. So, in development since at least 2013, possibly earlier. And it targeted the rather well established, known, and widely used ML practices.
GPT-like models didn't become even remotely useful until at least 5 years later.
I'm all for compassion, but engineers knew the NE was empty when it sat idle for 10 years on our computers.
- when you're given no usecase for your engineering piece, apart from "detour characters in pictures". It's an exageration but AI's contributions in iOS aren't visible; Meanwhile Google has features that people actually notice like removing tourists from your holidays photos — worse: it's mostly a simple collage feature working on the main CPU, and it has the same social effect as green bubbles in iMessage ("ah. Tourists on your photos. iPhone user?")
- and you tout it as "16 Neural Engine cores" during the sales, with no associated software, no listed material feature, just hand-waving,
- Siri maxxes out at "There is no contact named 'What's the weather today' in your agenda",
Then can't really claim that Apple engineers' problem was really the bad luck that ML wasn't the determining part of the future. It's more like misreading the room for 5 to 10 years straight.
Apple engineering's excellence on vertical integration and supply chain control gave them absolute power over our world (with merit), it just failed at that particular project. Which occupies 40% of our CPUs.
> an exageration but AI's contributions in iOS aren't visible
If you want to ignore them, that’s right. In the real world, they’ve been talking about ML and how it’s making pictures or such-and-such aspect of the OS better for about a decade now. It might not be flashy, but it is used throughout the OS.
> Meanwhile Google has features that people actually notice like removing tourists from your holidays photos — worse: it's mostly a simple collage feature working on the main CPU, and it has the same social effect as green bubbles in iMessage ("ah. Tourists on your photos. iPhone user?")
The feature to do this has been in the Photos application for years, what are you talking about?
How does this relate to the more recent work on the M4 ANE found at https://maderix.github.io/articles/ ? Does the M4 and later ANE expose any additional capabilities, or is it just a higher-performance iteration of the same thing?
As an aside, the introduction to this article seems to conflate the ANE with the Neural Accelerators (NAX) found in the M5+ (and A-series equivalents) GPUs. These are very different things, and Apple is still working on the ANE - the M6 and A20 will apparently feature doubled ANE blocks.
This one's authored by a human, the other one is authored by Claude.
> Does the M4 and later ANE expose any additional capabilities, or is it just a higher-performance iteration of the same thing?
IIUC, M4 introduced a fast path for INT8 weights and activations (w8a8). M5 Ultra, M6 and A20 have two ANEs.
> As an aside, the introduction to this article seems to conflate the ANE with the Neural Accelerators (NAX)
Yeah, that part is true. NAX cores are matmult accelerators, closer to tensor cores in NVIDIA GPUs.
It's worth remembering that Apple is releasing a new framework (Core AI) this fall that goes beyond the Pytorch and Tensorflow workloads that the decade old Core ML framework allowed.
> Core AI allows your app to use the latest model architectures and inference techniques across the CPU, GPU, and Neural Engine.
https://developer.apple.com/documentation/coreai
A lot of folks are/were saying that Apple has missed the boat when it comes to AI, in some (important) aspects that is correct, but I think it's worth remembering that Apple added the Neural Engine to A-series chips in 2017, before the AI hoopla really kicked off:
* https://en.wikipedia.org/wiki/Neural_Engine
* https://apple.fandom.com/wiki/Neural_Engine
"AI" has grown much more since then, and there have been important developments that Apple has not deployed (well), but I think they were looking ahead a little more than most at the time (even if events 'got away' from them subsequently).
IMHO, I think Apple is poised extremely well. They didn't blow billions of dollars chasing models that are becoming commoditized. So many interesting models can now be run locally. All the big AI players have to pay even more to run those models when Apple will happily sell you the hardware, and you pay for the electricity. There will always be a place for some many-billion parameter model but as time progresses I think fast local models that keep data on site will always be a valuable, and Apple will happily sell you something to run them.
Does anyone use it though? Apparently ANEForge got 16.8 tok/s of Qwen3-8B + 0.6B draft using private APIs. Not big enough
> but I think it's worth remembering that Apple added the Neural Engine to A-series chips in 2017
but missing the boat means you haven't succeeded not that you had some kind of initial foothold. haven't you seen the big short:
"i may have been early but i'm not wrong"
"it's the same thing"
https://www.youtube.com/watch?v=pLLgNi5UmB0&t=116s
It may not matter that they're "sleeping on it". AI is going to disrupt all tech companies.
Clearly software and SaaS has no moat. I just saw someone vibe code a fully functional Photoshop in a week of Astra use. I'm thinking even big tech is not immune to this. Even the mighty infrastructure players.
Vibe hardware is just getting started, and I'm hoping we see competitors to iPhone and Android and MacBook. It's less expensive to build and test things now, which might lead to a Cambrian explosion of new hardware startups.
The world will greatly benefit from this.
It's funny that American tech giants invented AI, because AI is going to unseat American tech giants.
SaaS' moat has always been someone else worrying about the hosting/upgrades/management part for you. Stores still sell sliced bread even though bread machines make bread for you and everyone already has a knife. People often buy convenience, even when they could do something themselves.
> It's funny that American tech giants invented AI, because AI is going to unseat American tech giants.
Ah yes, let's ignore that 40 years of research done on neural networks and give all the credit instead to the tech giants that came in as soon as there was enough of a breakthrough to start making them useful in language tasks. Definitely the American tech giants and not the Canadian and international researchers who spent their lives building the foundations. /s
Apple's Neural Engine is equivalent to the "NPU" hardware you would find on many cheap ARM SBCs. It's a dedicated coprocessor with a very low performance target, not intended for giant transformers or LLM acceleration. These NPUs are not particularly hard to design, and their critical flaw is that they don't scale very well. CUDA "won" because a bigger GPU meant having an utterly massive amount of CUDA cores to delegate ALU work to. NPUs/Neural Engine has the opposite problem, where spending $10,000 on an M1 Ultra only gets you ~2x better NPU performance versus the baseline $600 M1 chip. NPUs and Neural Engines are essentially dark silicon on the majority of devices with them, their uses are few and far between.
Considering Apple's refusal to sign Nvidia's ARM/CUDA drivers, it is pretty clear how Apple missed the boat here. Apple Silicon could have dominated the datacenter rollout if macOS supported CUDA properly. The Mac Pro would probably not have been cancelled if the PCI lanes could be used for normal datacenter GPUs and CUDA workloads. The excellent Thunderbolt bandwidth present on so many Macs is wasted supporting RDNA but not eGPU enclosures. There are several hardware features that Apple holds back for no good reason, handing Nvidia the lead in certain markets.
The only thing that stopped Apple from riding AI to the top was their own petty grudge towards Nvidia. Simple changes to macOS would have destroyed Nvidia's Grace CPU sales and made Apple Silicon the crown prince of the AI boom, at zero risk to themselves. The only thing Apple really needed was their own Mellanox equivalent.
Amazing analysis
Same author even found a bug in it https://eiln.github.io/posts/ane-dma.html
This isn't ai slop. It's fascinating and well written.
But I learned something really basic - i didn't know that the ANE (and the data pipeline around it) was designed for CNN rather than transformers. It's always been an open loop in my head, wondering why the ANE was less impactful than i understood it should be.
>i didn't know that the ANE (and the data pipeline around it) was designed for CNN rather than transformers.
Multiple stories have reported that ANE came from Apple's self-driving car project that got canceled. (Makes sense since CNN is used for vision-related machine learning and enables cars to analyze their surroundings.) They spent 10 years and ~10 billion on research & development on a product that never got released so Apple is probably happy they're able to salvage some of that ai technology and put it in iPhones and Macs.
Tesla also has its own NPUs for self-driving - and Tesla uses transformers for sensor fusion.
My guess would be that the main use case for an NPU in iPhone just used to be image processing/computational photography. Thus the CNN bent.
Also makes sense with the timing - back when iPhone first got its NPU, CV was the killer app for ML.
This is pure sunk cost fallacy. CNNs were from the deep learning ImageNet heydays, but now everything in that domain is equally done better by transformers. All you are doing is wasting area and saddling software with outdated hardware, and myopic PMs insisting on its use will create inferior products. Now that sounds a lot like the Apple AI efforts..
I designed and deployed several custom CNN-based AI models to ipad hosts around 2018-2020. It was a finicky process - CoreML didn't give you much control (or even feedback) on what parts of your model would run on the CPU, GPU, or ANE - but you could usually tell right away when they didn't, just based on big latency differences. Performance to power (and size) ratio was pretty good and I was impressed at the time, despite being generally not really happy about working with the apple ecosystem. I would've preferred something Android (or Linux)-based but there wasn't really anything comparable out there back then. (SBCs like the Jetson would've worked, but we would've had to figure out how to add an interface for human operators which would've been a big lift compared to just using off the shelf hardware).
A lot of neural engine, particularly in the embedded domain (ARM/RISC MCUs) have the same problem. Designing other models means on top of this means a lot of profiling to get convolution blocks right to get good speedups. (We optimized this in the past e.g. using Neural Architecture Search on super networks)
> But I learned something really basic
Same for me!
Also, just imagine being the group at Apple responsible for designing this section of the chip, starting probably almost a decade back – under the constant uncertainty of not knowing what direction ML workloads would develop in…
ML research was a rather known quantity, or the separate "Neural Engine" CPU explicitly aimed at existing ML pipelines wouldn't exist.
However, very few used it for anything, even within Apple. I feel like it was a huge wasted opportunity.
> ML research was a rather known quantity, or the separate "Neural Engine" CPU explicitly aimed at existing ML pipelines wouldn't exist.
No, not really. Transformers were just one of the possible directions. Silicon design does not have the same time scale than software. Now, everyone is using transformers so it becomes harder to do anything else, and it’s been the case long enough that hardware had some time to align (but is still lagging). But who’s to say that a different architecture published last year won’t take the world by storm 2 years from now?
It’s easy to say it in hindsight, but transformers took a bit of effort to get where they are now.
> Silicon design does not have the same time scale than software.
"Neural Engine" has been a part of iPhones since 2017. So, in development since at least 2013, possibly earlier. And it targeted the rather well established, known, and widely used ML practices.
GPT-like models didn't become even remotely useful until at least 5 years later.
I'm all for compassion, but engineers knew the NE was empty when it sat idle for 10 years on our computers.
- when you're given no usecase for your engineering piece, apart from "detour characters in pictures". It's an exageration but AI's contributions in iOS aren't visible; Meanwhile Google has features that people actually notice like removing tourists from your holidays photos — worse: it's mostly a simple collage feature working on the main CPU, and it has the same social effect as green bubbles in iMessage ("ah. Tourists on your photos. iPhone user?")
- and you tout it as "16 Neural Engine cores" during the sales, with no associated software, no listed material feature, just hand-waving,
- Siri maxxes out at "There is no contact named 'What's the weather today' in your agenda",
Then can't really claim that Apple engineers' problem was really the bad luck that ML wasn't the determining part of the future. It's more like misreading the room for 5 to 10 years straight.
Apple engineering's excellence on vertical integration and supply chain control gave them absolute power over our world (with merit), it just failed at that particular project. Which occupies 40% of our CPUs.
The ANE hasn't been sitting empty for 10 years. All those Photos features like face recognition and auto classification run on ANE.
Some features using the Neural Engine:
* Face ID since the iPhone X released in 2017
* fall and crash detection
* Live captions in videos, calls, and spoken audio
* facial recognition in the Photos app
* voice isolation in calls
There's more, but I'll stop there.
Dictation, text to speech, some of that computational photography, etc too!
> an exageration but AI's contributions in iOS aren't visible
If you want to ignore them, that’s right. In the real world, they’ve been talking about ML and how it’s making pictures or such-and-such aspect of the OS better for about a decade now. It might not be flashy, but it is used throughout the OS.
> Meanwhile Google has features that people actually notice like removing tourists from your holidays photos — worse: it's mostly a simple collage feature working on the main CPU, and it has the same social effect as green bubbles in iMessage ("ah. Tourists on your photos. iPhone user?")
The feature to do this has been in the Photos application for years, what are you talking about?
Ported a transformer to ANE and the whole job was pretending it was a CNN, 4D tensors with seq in the last axis and 1x1 convs instead of matmuls.
I thought random developers weren't even allowed to use ANE, and it's just do the computational photography DoF faking.
> what workloads it was designed for and accels at.
excels!
Could've been a pun as a neural processor is an accelerator, so it 'accels' at machine learning tasks!
Maybe, but then I wouldn't expect "at" :)
At least we know it wasn't written by a bot