Just made a TTS tool based on Kitten TTS, fully browser based, no Python server backend: https://quickeditvideo.com/tts/
A tts model of this size should be industry standard!
The people calling it "OK" probably tried it for themselves. Whatever model is being demoed in that video is not the same as the 25MB model they released.
It doesn't sound so good. Excellent technical achievement and it may just improve more and more! But for now I can't use it for consumer facing applications.
Speech speed is always a tunable parameter and not something intrinsic to the model.
The comparison to make is expressiveness and correct intonation for long sentences vs something like espeak. It actually sounds amazing for the size. The closest thing is probably KokoroTTS at 82M params and ~300MB.
The voices sound artificial and a bit grating. The male voices especially are lacking, especially in depth: only the ultimate voice has any depth at all, while the others sound like teenagers who haven't finished puberty. None of the voices sound quite human, but they're all very annoying, and part of that is that they sound like they're acting.
The only real questions are which Chinese gacha game they ripped data from and whether they used Claude Code or Gemini CLI for Python code. I bet one can get a formant match from output this much overfit to whatever data. This isn't going to stay up for long.
Impressive technical achievement, but in terms of whether I'd use it: oof, that male voice is like one of these fake-excited newsreaders. Like they're always at the edge of their breath. The female one is better but still someone reading out an advertisement for a product they were told they must act extra excited for. I assume this is what the majority of training data was like and not an intentional setting for the demo. Unsure whether I could get used to that
I use TTS on my phone regularly and recently also tried this new project on F-Droid called SherpaTTS, which grabs some models from Huggingface. They're super heavy (the phone suspends other apps to disk while this runs) and sound good, but in the first news article there were already one or two mispronunciations because it's guessing how to say uncommon or new words and it's not based on logical rules anymore to turn text into speech
Google and Samsung have each a TTS engine pre-installed on my device and those sound and work fine. A tad monotonous but it seems to always pronounce things the same way so you can always work out what the text said
Espeak (or -ng) is the absolute worst, but after 30 seconds of listening closely you get used to it and can understand everything fine. I don't know if it's the best open source option (probably there are others that I should be trying) but it's at least the most reliable where you'll always get what is happening and you can install it on any device without licensing issues
anyone else wants to try sherpaOnnx you can try this.. https://github.com/willwade/tts-wrapper we recently added in the kokoro models which should sound a lot better. There are a LOT of models to choose from. I have a feeling the Droid app isnt handling cold starts very well.
I hope this is the future. Offline, small ML models, running inference on ubiquitous, inexpensive hardware. Models that are easy to integrate into other things, into devices and apps, and even to drive from other models maybe.
Dedicated single-purpose hardware with models would be even less energy-intensive. It's theoretically possible to design chips which run neural networks and alike using just resistors (rather than transistors).
Such hardware is not general-purpose, and upgrading the model would not be possible, but there's plenty of use-cases where this is reasonable.
But resistors are, even in theory, heat dissipating devices. Unlike transistors, which can in theory be perfectly on or off (in both cases not dissipating heat).
It's theoretically possible but physical "neurons" is a terrible idea. The number of connections between two layers of an FF net is the product of the number of weights in each, so routing makes every other problem a rounding error.
This is what Apple is envisioning with their SLMs, like having a model specifically for managing calendar events. It doesn't need to have the full knowledge of all humanity in it - just what it needs to manage the calendar.
I'm still betting on a future AppleTV model being a full-on local LLM machine.
This way they could offload as much of the "LLM" work on a device that lives in the home, all family linked phones and devices could use it for local inference.
It's way overpowered as is anyway, why not use it for something useful.
Hmm. A pay once (or not at all) model that can run on anything? Or a subscription model that locks you in, and requires hardware that only the richest megacorps can afford? I wonder which one will win out.
Performance Results:
Initial Latency: ~315ms for short text
Audio Generation Speed (seconds of audio per second of processing):
- Short text (12 chars): 3.35x realtime
- Medium text (100 chars): 5.34x realtime
- Long text (225 chars): 5.46x realtime
- Very Long text (306 chars): 5.50x realtime
Findings:
- Model loads in ~710ms
- Generates audio at ~5x realtime speed (excluding initial latency)
- Performance is consistent across different voices (4.63x - 5.28x realtime)
Thanks for running the benchmarks. Currently the models are not optimized yet. We will optimize loading etc when we release an SDK meant for production :)
Does anybody find it funny that sci-fi movies have to heavily distort "robot voices" to make them sound "convincingly robotic"? A robotic, explicitly non-natural voice would be perfectly acceptable, and even desirable, in many situations. I don't expect a smart toaster to talk like a BBC host; it'd be enough is the speech if easy to recognize.
A robotic, explicitly non-natural voice would be perfectly acceptable, and even desirable, in many situations[...]it'd be enough is the speech if easy to recognize.
We've had formant synths for several decades, and they're perfectly understandable and require a tiny amount of computing power, but people tend not to want to listen to them:
The YouTube video [1] was published in 2019. The Blog spam posts range from Nov 2022 to July 2023.
Other than the video, the only relevant content is on the about page [2]. It says the voice is a collaboration between 5 different entities, including advocacy groups, marketing firms and a music producer.
The video is the only example of the voice in use. There is no API, weights, SDK, etc.
I suspect this was a one-off marketing stunt sponsored by Copenhagen pride before the pandemic. The initial reaction was strong enough that a couple years they were still getting a small but steady flow of traffic. One of the involved marketing firms decided to monetize the asset and defaced it with blog spam.
Huh. Sounds perfectly intelligible and definitively artificial. Feels weakly feminine to me, but only because I was primed to think about gender from the branding.
It’s a good choice for a robot voice. It’s easier to understand than the formant synths or deliberately distorted human voices. The genderless aspect is alien enough to avoid the uncanny valley. You intuitively know you’re dealing with something a little different.
In the Culture novels, Iain Banks imagines that we would become uncomfortable with the uncanny realism of transmitted voices / holograms, and intentionally include some level of distortion to indicate you're speaking to an image
Depends on the movie. Ash and Bishop in the Alien franchise sound human until there's a dramatic reason to sound more 'robotic'.
I agree with your wider point. I use Google TTS with Moon+Reader all the time (I tried audio books read by real humans but I prefer the consistency of TTS)
Slightly different there because it's important in both cases that Ripley (and we) can't tell they're androids until it's explicitly uncovered. The whole point is that they're not presented as artificial. Same in Blade Runner: "more human than human". You don't have a film without the ambiguity there.
I remember that the novelization of the fifth element describes that the cops are taught to speak as robotic as possible when using speakers for some reason. Always found the idea weird that someone would _want_ that
I got an error when I tried the demo with 6 sentences, but it worked great when I reduced the text to 3 sentences. Is the length limit due to the model or just a limitation for the demo?
"This first Book proposes, first in brief, the whole Subject, Mans disobedience, and the loss thereupon of Paradise wherein he was plac't: Then touches the prime cause of his fall, the Serpent, or rather Satan in the Serpent; who revolting from God, and drawing to his side many Legions of Angels, was by the command of God driven out of Heaven with all his Crew into the great Deep."
It takes a while until it starts generating sound on my i7 cores but it kind of works.
This also works:
"blah. bleh. blih. bloh. blyh. bluh."
So I don't think it's a limit on punctuation. Voice quality is quite bad though, not as far from the old school C64 SAM (https://discordier.github.io/sam/) of the eighties as I expected.
I tried to replicate their demo text but it doesn't sound as good for some reason.
If anyone else wants to try:
> Kitten TTS is an open-source series of tiny and expressive text-to-speech models for on-device applications. Our smallest model is less than 25 megabytes.
> Error generating speech: failed to call OrtRun(). ERROR_CODE: 2, ERROR_MESSAGE: Non-zero status code returned while running Expand node. Name:'/bert/Expand' Status Message: invalid expand shape
Thanks, I was looking for that. While the reddit demo sounds ok, even though on a level we reached a couple of years ago, all TTS samples I tried were barley understandable at all
On PC it's a python dependency hell but someone managed to package it in self contained JS code that works offline once it loaded the model? How is that done?
ONNXRuntime makes it fairly easy, you just need to provide a path to the ONNX file, give it inputs in the correct format, and use the outputs. The ONNXRuntime library handles the rest. You can see this in the main.js file: https://github.com/clowerweb/kitten-tts-web-demo/blob/main/m...
Plus, Python software are dependency hell in general, while webpages have to be self-contained by their nature (thank god we no longer have Silverlight and Java applets...)
yeah, this is just a preview model from an early checkpoint. the full model release will be next week which includes a 15M model and an 80M model, both of which will have much higher quality than this preview.
Not open source. "You will need internet connectivity to validate your AccessKey with Picovoice license servers ... If you wish to increase your limits, you can purchase a subscription plan." https://github.com/Picovoice/orca#accesskey
Going online is a dealbreaker but if you really need it you could use ghidra to fix that. I had tried to find a conversion of their model to onnx (making their proprietary pipeline useless) but failed.
Hopefully open source will render them irrelevant in the future.
Does an apk for Android exist for replacing its speech to text engine? I tried sherpa-onnx but it was too slow for real time usage it seemed, and especially so for audiobooks when sped up.
I can't test this out right now, is this just a demo or is it actually an apk for replacing the engine? Because those are two different things, the latter can be used any time you want to read something aloud on the page for example. This is the sherpa-onnx one I'm talking about.
The headline feature isn’t the 25 MB footprint alone. It’s that KittenTTS is Apache-2.0. That combo means you can embed a fully offline voice in Pi Zero-class hardware or even battery-powered toys without worrying about GPUs, cloud calls, or restrictive licenses. In one stroke it turns voice everywhere from a hardware/licensing problem into a packaging problem. Quality tweaks can come later; unlocking that deployment tier is the real game-changer.
yeah, we are super excited to build tiny ai models that are super high quality. local voice interfaces are inevitable and we want to power those in the future. btw, this model is just a preview, and the full release next week will be of much higher quality, along w another ~80M model ;)
The issue is even bigger: phonemizer is using espeak-ng, which isn't very good at turning graphemes into phonemes. In other TTS which rely on phonemes (e.g. Zonos) it turned out to be one of the key issues which cause bad generations.
And it isn't something you can fix, because the model was trained on bad phonemes (everyone uses Whisper + then phonemizes the text transcript).
> IANAL, but AFAICS this leaves 2 options, switching the license or removing that dependency.
There is a third option: asking the project for an exception.
Though that is unlikely to be granted¹ leaving you back with just the other two options.
And of course a forth choice: just ignore the license. This is the option taken by companies like Onyx, whose products I might otherwise be interested in…
----
[1] Those of us who pick GPL3 or AGPL generally do so to keep things definite and an exception would muddy the waters, also it might not even be possible if the project has many maintainers as relicensing would require agreement from all who have provided code that is in the current release. Furthermore, if it has inherited the license from one of its dependencies, an exception is even less practical.
Ah, yes, good catch, I didn't look deeper into the dependency tree at all. I'll update my footnote to include that as one of the reasons an exception may be impossible (or at least highly impractical).
A fourth option would be a kind of dual-licensing: the project as-is is available under GPL-3.0, but the source code in this repository excluding any dependencies is also available under Apache 2.0
Any user would still effectively be bound by the GPL-3.0, but if someone can remove the GPL dependencies they could use the project under Apache
That is an option for the publisher of the library, not the consumer of it. If it isn't already done then asking for it to be done is the same as asking for an exception otherwise (option three).
The use of the library is four lines. Three set up the library (`phonemizer.backend.EspeakBackend(language="en-us", preserve_punctuation=True, with_stress=True)`), the other calls it (`phonemes_list = self.phonemizer.phonemize([text])`). Plus I guess the import statements. Even ignoring Google vs Oracle I don't think those lines by themselves meet any threshold of originality.
Obviously you can't run them (with the original library) without complying with the GPL. But I don't see why I couldn't independently of that also give you this text file under Apache 2.0 to do with as you want (which for the record still doesn't allow you to run them with the original library without complying with the GPL, but that'd be phoneme forcing you to do that, not this project)
You would have to be very specific about the dual-licensing to avoid confusion about what you are allowed to do under Apache conditions though. You can't just say "it's dual-licensed"
You could even extract out the parts that do not call the GPL library into an upstream project under the Apache 2.0 licence, and pull in both that and the GPL library in the downstream project, relying on Apache 2.0 -> GPL 3.0 compatibility instead of explicit dual licensing to allow the combined work to be distributed under GPLv3.
This would only apply if they were distributing the GPL licensed code alongside their own code.
If my MIT-licensed one-line Python library has this line of code…
run([“bash”, “-c”, “echo hello”])
…I’m not suddenly subject to bash’s licensing. For anyone wanting to run my stuff though, they’re going to need to make sure they themselves have bash installed.
(But, to argue against my own point, if an OS vendor ships my library alongside a copy of bash, do they have to now relicense my library as GPL?)
The FSF thinks it counts as a derivative work and you have to use the LGPL to allow linking.
However, this has never actually been proven in court, and there's many good arguments that linking doesn't count as a derivative work.
Old post by a lawyer someone else found (version 3 wouldn't affect this) [1]
For me personally I don't really understand how, if dynamic linking was viral, using linux to run code isn't viral. Surely at some level what linux does to run your code calls GPLed code.
It doesn't really matter though, since the FSF stance is enough to scare companies from not using it, and any individual is highly unlikely to be sued.
> For me personally I don't really understand how, if dynamic linking was viral, using linux to run code isn't viral. Surely at some level what linux does to run your code calls GPLed code.
The Linux kernel has an explicit exception for userspace software:
> NOTE! This copyright does not cover user programs that use kernel services by normal system calls
And the GPL also has an explicit exception for "system" software such as kernel, platform libraries etc.:
> The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
> The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work.
> This would only apply if they were distributing the GPL licensed code alongside their own code.
As far as I understand the FSF's interpretation of their license, that's not true. Even if you only dynamically link to GPL-licensed code, you create a combined work which has to be licensed, as a whole, under the GPL.
I don't believe that this extends to calling an external program via its CLI, but that's not what the code in question seems to be doing.
(This is not an endorsement, but merely my understanding on how the GPL is supposed to work.)
This is a false analogy. It's quite straightforward.
Running bash (via exec()/fork()/spawn()/etc) isn't the same as (statically or dynamically) linking with its codebase. If your MIT-licensed one-liner links to code that's GPL licensed, then it gets infected by the GPL license.
My interpretation of their FAQ[1] on it is that shelling out and IPC are fine, while linking is not. As you say, it's ultimately up to the courts to decide on.
GPL is for boomers at this point. Floppy disks? Distribution? You can use a tool but you cant change it? A DLL call means you need to redistribute your code but forking doesn't?
Yes, but if you use open source libraries for your closed source SaaS - thats fine. People get their software _over_ the network delivered to them in a VM (your browser).
The result can only be distributed under the terms of the GPL-3. That's actually a crucial difference: there's nothing preventing Kitten TTS from being Apache licensed, soliciting technical contributions under that license, and parts of its code being re-used in other software under that license. Yes, for the time being, this limits what you can do with Kitten TTS if you want to use the software as a whole (e.g. by embedding it into your product), but the license itself is still Apache and that can have value.
Okay, what's stopping you from feeding the code into an LLM and re-write it and make it yours? You can even add extra steps like make it analyze the code block by block then supervise it as it is rewriting it. Bam. AI age IP freedom.
Morals may stop you but other than that? IMHO all open source code is public domain code if anyone is willing to spend some AI tokens.
One person reads the code and produces a detailed technical specification. Someone reviews it to ensure that there is nothing in there that could be classified as copyrighted material, then a third person (who has never seen the original code) implements the spec.
You could use an LLM at both stages, but you'd have to be able to prove that the LLM that does the implementation had no prior knowledge of the code in question... Which given how LLMs have been trained seems to me to be very dubious territory for now until that legal situation gets resolved.
AI is useful in Chinese walling code, but it’s not as easy as you make it sound. To stay out of legal trouble, you probably should refactor the code into a different language, then back into the target language. In the end, it turns into a process of being forced to understand the codebase and supervising its rewriting. I’ve translated libraries into another language using LLMs, I’d say that process was 1/2 the labor of writing it myself. So in the end, going 2 ways, you may as well rewrite the code yourself… but working with the LLM will make you familiar with the subject matter so you -could- rewrite the code, so I guess you could think of it as a sort of buggy tutorial process?
I am not sure even that is enough. You would really need to do a clean room reimplementation to be safe - for exactly the same reasons that people writing code write clean room reimplementations.
Yeah, the algorithms and program flow would have to be materially distinct to be really safe. Maybe switching language paradigms would get that for you in most cases? Js->haskell->js? Sounds like a nightmare lol.
Tell me you don't know how to use LLMs properly without telling me.
You don't give the whole codebase to an LLM and expect it to have one shot output. Instead, you break it down and and write the code block by block. Then the size if the codebase doesn't matter. You use the LLM as a tool, it is not supposed to replace you. You don't try to become George from Jetsons who is just pressing a button and doesn't touch anything, instead you are on top of it as the LLM does the coding. You test the code on every step to see if the implementation behaves as expected. Do enough of this and you have proper, full "bespoke" software.
A Festival's English model, festvox-kallpc16k, is about 6 MB, and it is a large model; festvox-kallpc8k is about 3.5 MB.
eSpeak NG's data files take about 12 MB (multi-lingual).
I guess this one may generate more natural-sounding speech, but older or lower-end computers were capable of decent speech synthesis previously as well.
I'm not blind but spoken English it's far more difficult to grasp than written one (I'm a non-native speaker), and Flite runs on n270 netbooks at crazy speeds with really good enough voices.
What about the training data? Is everyone 100% confident that models are not a derived work of the training inputs now, even if they can reproduce input exactly?
I play around with a nvidia jetson orin nano super right now and its actually pretty usuable with gemma3:4b and quite fast - even image processing is done in like 10-20 seconds but this is with GPU support. When something is not working and ollama is not using the GPU this calls take ages because the cpu is just bad.
This opens up voice interfaces for medical devices, offline language learning tools, and accessibility gadgets for the visually impaired - all markets where cloud dependency and proprietary licenses were showstoppers.
If you have `uv` installed, you can try my merged ref that has all of these PRs (and #22, a fix for short generation being trimmed unnecessarily) with
uvx --from git+https://github.com/akx/KittenTTS.git@pr-21-22-24-25 kittentts --output output.wav --text "This high quality TTS model works without a GPU"
Thanks for the quick intro into UV, it looks like docker layers for python
I found the TTS a bit slow so I piped the output into ffplay with 1.2x speedup to make it sound a bit better
uvx --from git+https://github.com/akx/KittenTTS.git@pr-21-22-24-25 kittentts --text "I serve 12 different beers at my restaurant for over 1000000 customers" --voice expr-voice-3-m --output - | ffplay -af "atempo=1.2" -f wav -
I was commiserating with my brother over how difficult it is to set up an environment to run one LLM or diffusion model, let alone multiple or a combination. It's 5 percent CUDA/ROCm difficulties and 95% Python difficulties. We have a theory that Lanyone working with generative AI has to tolerate output that is only 90% right, and is totaly fine working with a language and environment that only 90% works.
Why is Python so bad at that? It's less kludgy than Bash scripts, but even those are easier to get working.
Yeah, but it's easily solved, with directives, headers, or make files that specify which language standard it follows. Better yet, you can use different syntax with different language standards, so it's clear which to follow. If a compiler can automatically figure whether I'm compiling C or C++, why can't a Python interpreter figure out if I'm running version two or three, of the same language?
> JS/TS/npm is just as bad with probably more build tools/frameworks.
This is flat out wrong. NPM packages by default are local to a directory. And I haven't seen a package rely on a specific minor version of node in literally years. Node's back compat is also great, there was one hiccup 5 or 6 years ago where a super popular native package was deprecated ago but that's been about it.
I can take current LTS node and run just about any package from the NPM repo written within the last 4 or 5 years and it will just work. Meanwhile plenty of python packages somehow need specific point releases. What the unholy hell.
Node version manager does exist, and it can be setup to work per directory, which is super cool, but I haven't needed NVM in literal years.
I think it can install Python itself too. Though I have had issues with that - especially with SSL certificate locations, which is one of Linux's other clusterfucks.
A tool that was only released, what, a year or two ago? It simply won't be present in nearly all OS/distros. Only modern or rolling will have it (maybe). It's funny when the recommended python dependency manager managers are just as hard to install and use as the script themselves. Very python.
The project is like 80% there by having a pyproject file that should work with uv and poetry. The just aren't any package versions specified and the python version is incredibly lax, and no lock file is provided.
in this context uv works perfectly fine with poertry, if you publish a wheel from poetry uv can use it. You don't have to switch anything in your project to make it work.
PYTHON(1) General Commands Manual PYTHON(1)
NAME
python - an object-oriented programming language
SYNOPSIS
python [ -c command | script | - ] [ arguments ]
DESCRIPTION
Python is the standard programming language.
Computer scientists love Python, not just because whitespace comes first ASCIIbetically, but because it's the standard. Everyone else loves Python because it's PYTHON!
You're getting a lot of comments along the lines of "Why don't you just ____," which only shows how Stockholmed the entire Python community is.
With no other language are you expected to maintain several entirely different versions of the language, each of which is a relatively large installation. Can you imagine if we all had five different llvms or gccs just to compile five different modern C projects?
I'm going to get downvoted to oblivion, but it doesn't change the reality that Python in 2025 is unnecessarily fragile.
That’s exactly what I have. The C++ codebases I work on build against a specific pinned version of LLVM with many warnings (as errors) enabled, and building with a different version entails a nonzero amount of effort. Ubuntu will happily install several versions of LLVM side by side or compilation can be done in a Docker container with the correct compiler. Similarly, the TypeScript codebases I work with test against specific versions of node.js in CI and the engine field in package.json is specified. The different versions are managed via nvm. Python is the same via uv and pyproject.yaml.
I don't doubt it, but I don't think that situation is accepted as the default in C/C++ development. For the most part, I expect OSS to compile with my own clang.
Oof, those are poor examples. Most compilers using LLVM other than clang do ship with their own LLVM patches, and cross-compiling with GCC does require installing a toolchain for each target.
Cross-compiling is a totally different subject… I'm trying to make an apples-to-apples comparison. If you compile a lot of OSS C projects for the host architecture, you typically do not need multiple LLVMs or GCCs. Usually, the makefile detects various things about the platform and compiler and then fails with an inscrutable error. But that is a separate issue! haha
There are still people who use machine wide python installs instead of environments? Python dependency hell was already bad years ago, but today it's completely impractical to do it this way. Even on raspberries.
Yep. Python stopped being Python a decade ago. Now there are just innumberable Pythons. Perl... on the otherhand, you can still run any perl script from any time on any system perl interpreter and it works! Granted, perl is unpopular and not getting constant new features re: hardcore math/computation libs.
Anyway, I think I'll stick with Festival 1.96 for TTS. It's super fast even on my core2duo and I have exactly zero chance of getting this Python 3'ish script to run on any machine with an OS older than a handful of years.
It reminds me of the costs and benefits of RollerCoaster Tycoon being written in assembly language. Because it was so light on resources, it could run on any privately owned computer, or at least anything x86, which was pretty much everything at the time.
Now, RISC architectures are much more common, so instead of the rare 68K Apple/Amiga/etc computer that existed at the time, it's super common to want to run software on an ARM or occasionally RISC-V processor, so writing in x86 assembly language would require emulation, making for worse performance than a compiled language.
system python is for system applications that are known to work together. If you need a python install for something else, there's venv or conda and then pip install stuff.
I tried it. Not bad for the size (of the model) and speed. Once you install all the massive number of libraries and things needed we are a far cry away from 25MB though. Cool project nonetheless.
To make the setup easier and add a few features people are asking for here (like GPU support and long text handling), I built a self-hosted server for this model:
https://github.com/devnen/Kitten-TTS-Server
The goal was a setup that "just works" using a standard Python virtual environment to avoid dependency conflicts.
The setup is just the standard git clone, pip install in a venv, and python server.py.
The repository already runs an ONNX model. But the onnx model doesn't get English text as input, it gets tokenized phonemes. The prepocessing for that is where most of the dependencies come from.
Which is completely reasonable imho, but obviously comes with tradeoffs.
For space sensitive applications like embedded systems, could you shift the preprocessing to compile time?
You would need to constrain the vocabulary to see any benefits, but that could be reasonable. For example, you an enumeration of numbers, units and metric names could handle dynamic time, temperature and other dashboard items.
For something more complex like offline navigation, you already need to store a map. You could store street names as tokens instead of text. Add a few turn commands, and you have offline spoken directions without on device pre-processing.
Hmm that actually seems extremely slow, Piper can crank out a sentence almost instantly on a Pi 4 which is a like a sloth compared to that Ryzen and the speech quality seems about the same at first glance.
I suppose it would make sense if you want to include it on top of an LLM that's already occupying most of a GPU and this could run in the limited VRAM that's left.
assuming most answers will be more than a sentence, 2.25 seconds is already long enough if you factor the token generation in between... and imagine with reasoning!... We're not there yet.
While I think this is indeed impressive and has a specific use case (e.g. in the embedded sector), I'm not totally convinced that the quality is good enough to replace bigger models.
With fish-speech[1] and f5-tts[2] there are at least 2 open source models pushing the quality limits of offline text-to-speech. I tested F5-TTS with an old NVidia 1660 (6GB VRAM) and it worked ok-ish, so running it on a little more modern hardware will not cost you a fortune and produce MUCH higher quality with multi-language and zero-shot support.
For Android there is SherpaTTS[3], which plays pretty well with most TTS Applications.
Fish Speech says its weights are for non-commercial use.
Also, what are the two's VRAM requirents? This model has 15 million parameters which might run on low-power, sub-$100 computers with up-to-date software. Your hardware was an out-of-date 6GB GPU.
Hmm the quality is not so impressive. I'm looking for a really naturally sounding model. Not very happy with piper/kokoro, XTTS was a bit complex to set up.
For STT whisper is really amazing. But I miss a good TTS. And I don't mind throwing GPU power at it. But anyway. this isn't it either, this sounds worse than kokoro.
> Hmm the quality is not so impressive. [...] And I don't mind throwing GPU power at it.
This isn't for you, then. You should evaluate quality here based on the fact you don't need a GPU.
Back in the pre-Tacotron2 days, I was running slim TTS and vocoder models like GlowTTS and MelGAN on Digital Ocean droplets. No GPU to speak of. It cost next to nothing to run.
Since then, the trend has been to scale up. We need more models to scale down.
In the future we'll see small models living on-device. Embedded within toys and tools that don't need or want a network connection. Deployed with Raspberry Pi.
Edge AI will be huge for robotics, toys and consumer products, and gaming (ie. world models).
> This isn't for you, then. You should evaluate quality here based on the fact you don't need a GPU.
I know but it was more of a general comment. A really good TTS just isn't around yes in the OSS sphere. I looked at some of the other suggestions here but they have too many quirks. Dia sounds great but messages must have certain lengths etc and it picks a random voice every time. I'd love to have something self hosted that's as good as openai.
The best open one I've found so far is Dia - https://github.com/nari-labs/dia - it has some limitations, but i think it's really impressive and I can run it on my laptop.
Thanks I'll try! I like how it sounds, the quality is really good. But the limitations are really severe (shorter than 5 seconds is not ok, > 30 seconds is not ok, it will play a random voice every time, those make it pretty much unusable for an assistant to be honest).
But it might be worth setting it up and seeing if it improves over time.
Microsoft's and some of Google's TTS models make the simplest mistakes. For instance, they sometimes read "i.e." as "for example." This is a problem if you have low vision and use TTS for, say, proofreading your emails.
Well, speech synthesizers are pretty much famous for speaking all sorts of things wrong. But what I find very concerning about LLM based TTS is that some of them cant really speak numbers greater then 100. They try, but fail a lot. At least tts-1-hd was pretty much doing this for almost every 3 or 4 digit number. Especially noticeable when it is supposed to read a year number.
From the web demo this model is really good at numbers. It rushes through them, slurs them a bit together, but they are all correct, even 7 digit numbers (didn't test further).
Looks like they are sidestepping these kinds of issues by generating the phonemes with the preprocessing stage of traditional speech synthesizers, and using the LLM only to turn those phonemes into natural-ish sounding speech. That limits how natural the model can become, but it should be able to correctly pronounce anything the preprocessing can pronounce
Not entirely related but humans have the same problem.
For scriptwriting when doing voice overs we always explicitly write out everything. So instead of 1 000 000 we would write one million or a million. This is a trivial example but if the number was 1 548 736 you will almost never be able to just read that off. However one million, five hundred and forty eight thousand, seven hundred and thirty six can just be read without parsing.
Regarding humans, yes and no. If a human had constantly problems with 3 and 4 digit numbers like tts-1-hd does, I'd ask myself if they were neurodivergent in some way.
And yes, I added instructions along the lines of what you describe to my prompt. Its just sad that we have to. After all, LLM TTS has solved a bunch of real problems, like switching languages in a text, or foreign words. The pronounciation is better then anything we ever had. But it fails to read short numbers. I feel like that small issue could probably have been solved by doing some fine tuning. But I actually dont really understand the tech for it, so...
You probably mean "e.g." as "for example", not "i.e."?
This might be on purpose and part of the training data because "for example" just sounds much better than "e.g.". Presumably for most purposes, linguistic naturalness is more important than fidelity.
Sometimes I use “for example” and “e.g.” in consecutive sentences to not sound repetitive, or possibly even within the same sentence (e.g. in parentheses). In that case, speaking both as “for example” would degrade it linguistically.
In any case, I’d like TTS to not take that kind of artistic freedom.
would love to see how that turns out. the full model release next week will be more expressive and higher quality than this one so we're excited to see you try that out.
Thanks! Yeah. It definitely isn’t the absolute best in quality but it trounces the default TTS options on macOS (as third party developers are locked out of the Siri voices). And for less than the size of many modern web pages…
Most of these comments were originally posted to a different thread (https://news.ycombinator.com/item?id=44806543). I've moved them hither because on HN we always prefer to give the project creators credit for their work.
(it does however explain how many of these comments are older than the thread they are now children of)
The samples featured elsewhere seem to be from a larger model?
After testing this locally, it still sounds quite mechanical, and fails catastrophically for simple phrases with numbers ("easy as 1-2-3"). If the 80M model can improve on this and keep the expressiveness seen in the reddit post, that looks promising.
I'm curious why smallish TTS models have metallic voice quality.
The pronunciation sounds about right - i thought it's the hard part. And the model does it well. But voice timbre should be simpler to fix? Like, a simple FIR might improve it?
We change our tone based on personal style, emotion, context, and other factors. An accurate generator might need to encode all that information in the model. It will be larger than a model that doesn't do all of that.
Question for the experts here; What would be a SOTA TTS that can run on an average laptop (32GB RAM, 4GB VRAM). I just want to attach a TTS to my SLM output, and get the highest possible voice quality/ human resembleness.
thanks, but keep in mind that this model is just a preview checkpoint that is only 10% trained. the full release next week will be of much higher quality and it will include a 15M model and an 80M model.
Does your Eloquence installation include multiple languages? The one I have is only 1876 KB for US English only. And classic DECtalk is even smaller; I have here a version that's only 638 KB (again, US English only).
This feels different. This feels like a genuinely monumental release. Holy cow.
Very well done. The quality is excellent and the technical parameters are, simply, unbelievable. Makes me want to try to embed this on a board just to see if it's possible.
I'm so confused on how the model is actually made. It doesn't seem to be in the code or this stuff is way simpler than i thought. It seems to use a fancy library from japan, not sure how much it's just that
How does one build similar model, but for different languages? I was under impression that being open source, there would be some instructions how to build everything on your own.
Not bad for the size (with my very limited knowledge of this field) !
In a couple tests, the "Male 2" voice sounds reasonable, but I've found it has problem with some groups of words, specially when played with little context. I think it's small sentences.
For example, if you try to do just "Hey gang!", it will sound something like "Chay yang". But if you add an additional sentence after that, it will sound a bit different (but still weird).
If you're looking for other languages, Piper has been around in this scene for much longer and they have open-source training code and a lot of models (they're ~60MB instead of 25MB but whatever...) https://huggingface.co/rhasspy/piper-voices/tree/main
Actually I found it irritating that the readme does not mention the language at all. I think it is not good practice to deduce it from the language of the readme itself. I would not like to have German language tts models with only a German readme...
TTS is generally not multilingual. One might think a well-annotated phonetic descriptions of voices would suffice, but that's not quite how languages work nor how TTS work.
(but somehow LLMs handle multilingual input perfectly fine! that's a bit strange, if you think about that)
Awesome work! Often times in the TTS space, human-similarity is given way too much emphasis at the expense of hurting user access. Frankly as long as a voice is clear and you listen to it for a while, the brain filters out most quirks you would perceive on the first pass. Hence why many blind folks still are perfectly fine using espeak-ng. The other properties like speed of generation and size make it worth it.
I've been using a custom AI audiobook generation program [0] with piper for quite a while now and am very excited to look at integrating kitten. Historically piper has been the only good option for a free CPU-only local model so I am super happy to see more competition in the space. Easy installation is a big deal, since piper historically has had issues with that. (Hence why I had to add auto installation support in [0])
What I am still looking for is a way to clone voice locally. I have OK hardware. For example I can use Mistral Small 3.1 or what it is called locally. Premade voices can be interesting too, but I am looking for custom voice. Perhaps by providing audio and the corresponding transcript to the model, training it, and then give it a new text and let it speak that.
TL;DR: If you are interested in TTS, you should explore alternatives
I tried to use it...
Its python venv has grown to 6 GBytes in size. The demo sentence
> "This high quality TTS model works without a GPU"
works, it takes 3s to render the audio.
Audio sounds like a voice in a tin can.
I tried to have a news article read aloud and failed with
> [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Expand node. Name:'/bert/Expand'
> Status Message: invalid expand shape
If you are interested in TTS, you should explore alternatives
Nothing to do with Apple Intelligence. The speech synthesiser manager (the term manager was used for OS components in Classic Mac OS) has been around since the mid 90s or so. The change you’re hearing is probably a new/modified default voice.
> Nothing to do with Apple Intelligence
Sorry, the clarify - I mean in macOS beta 26, the `say` command's output matches the settings set in the System Settings -> Apple Intelligence & Siri -> Voice > Select... menu.
> speech synthesiser manager (the term manager was used for OS components in Classic Mac OS)
Especially fun to play with on the rainbow iMacs back then, too.
A localized version of this, and I could finally build my tiny Amazon Echo replacement. I would love to see all speech synthesis performed on a local device.
I'm doing this now with Home Assistant voice. All the TTS, STT, and LLMs involved run locally on my network. It's absurdly superior to every other voice assistant product. (Would be nice if it was just a pure multi-modal model though)
Good TTS feels like it is something that should be natively built into every consumer device. So the user can decide if they want to read or listen to the text at hand.
I'm surprised that phone manufacturers do not include good TTS models in their browser APIs for example. So that websites can build good audio interfaces.
I for one would love to build a text editor that the user can use completely via audio. Text input might already be feasible via the "speak to type" feature, both Android and iOS offer.
But there seems to be no good way to output spoken text without doing round-trips to a server and generate the audio there.
The interface I would like would offer a way to talk to write and then commands like "Ok editor, read the last paragraph" or "Ok editor, delete the last sentence".
It could be cool to do writing this way while walking. Just with a headset connected to a phone that sits in one's pocket.
On Mac OS you can "speak" a text in almost every app, using built in voice (like the Siri voice or some older voices). All offline, and even from the terminal with "say".
I tried it a few months ago to narrate an epub in Apple Books and it was very broken in a weird way. It starts out decent but after a few pages, it starts slurring, skipping words, trailing off not finishing sentences and then goes silent.
(I've just tried it again without seeing that issue within a few pages)
> Siri voice or some older voices
You can choose "Enhanced" and "Premium" versions of voices which are larger and sound nice and modern to me. The "Serena Premium" voice I was using is over 200Mb and far better that this Show HN. It's very natural but kind of ruined by diabolical pronunciation of anything slightly non-standard which sadly seems to cover everything I read e.g. people/place names, technical/scientific terms or any neologisms in scifi/fantasy.
It's so wildly incomprehensible for e.g. Tibetan names in a mountaineering book, that you have to check the text. If the word being butchered is frequently repeated e.g. main character’s name, then it's just too painful to use.
I absolutely agree, but it's really stubborn with the flowery language. I tried adding things like "DO NOT USE EMPTY PHRASES LIKE 'EVER-EVOLVING TECH LANDSCAPE'!!!!!" to the prompt, but it just can't resist.
I want to give the whole system an overhaul, maybe newer models are better at this. Or maybe a second LLM pass to de-flowerize (lol) the language.
It looks like it's Python, so it might be possible to use via https://github.com/livebook-dev/pythonx ? But the parallel huggingface/bumblebee idea was also good, hadn't seen or thought of, that definitely works for a lot of other models, curious if you get working! Some chance I'll play with this myself in a few months, so feel free to report back here or DM me!
I just decided to try this quickly and hit some issues on my Mac FYI, it might work better on Linux but I hit a compilation issue with `curated-tokenizers`, possibly from a typo in setup.py or pyproject.toml in curated-tokenizers, spotted by AI:
-Wno-sign-compare-Wno-strict-prototypes
should be
-Wno-sign-compare -Wno-strict-prototypes
so could perhaps fix with a PR to curated-tokenizers or by forking it...
Might well be other issues behind that, and unclear if need any other dependencies that kitten doesn't rely on directly like torch or torchaudio? but... not 5 mins easy, but looks like issues might be able to be worked through...
For reference this is all I was trying basically:
Mix.install([:pythonx])
Pythonx.uv_init("""
[project]
name = "project"
version = "0.0.0"
requires-python = ">=3.8"
dependencies = [
"kittentts @ https://github.com/KittenML/KittenTTS/releases/download/0.1/kittentts-0.1.0-py3-none-any.whl"
]
""")
It's just so annoying and idiotic that there aren't a few samples on the home page. It didn't occur to you that it's the very first thing people would want to hear?
Reddit post with generated audio sample: https://www.reddit.com/r/LocalLLaMA/comments/1mhyzp7/kitten_...
And a quick video with all of the different voices:
https://www.youtube.com/watch?v=60Dy3zKBGQg
Thanks. I really would not want to listen to any of these regularly.
Cool, thanks... aside: the last male voice sounds high/drunk.
thank you!
The reddit video is awesome. I don't understand how people are calling it an OK model. Under 25MB and cpu only for this quality is amazing.
Just made a TTS tool based on Kitten TTS, fully browser based, no Python server backend: https://quickeditvideo.com/tts/ A tts model of this size should be industry standard!
The people calling it "OK" probably tried it for themselves. Whatever model is being demoed in that video is not the same as the 25MB model they released.
Nope, looks like the default voice is the worst and it's not in the demo. A Reddit user generated these as well https://limewire.com/d/28CRw#UPuRLynIi7
Never thought I'd see the name LimeWire again, wow
Haha interesting pivot!
It did say this was a preview release, so I'll reserve judgement until that's out the door.
Local quality is very bad
https://vocaroo.com/1njz1UwwVHCF
It doesn't sound so good. Excellent technical achievement and it may just improve more and more! But for now I can't use it for consumer facing applications.
We are still training the model. We expect the quality to go up in the next release. This is just a preview release :)
[flagged]
Sounds very clear. For a non native english speaker like me, it's easy to understand.
Sounds slow and like something from an anine
Speech speed is always a tunable parameter and not something intrinsic to the model.
The comparison to make is expressiveness and correct intonation for long sentences vs something like espeak. It actually sounds amazing for the size. The closest thing is probably KokoroTTS at 82M params and ~300MB.
I think he meant overacting typical for English dubs.
The voices sound artificial and a bit grating. The male voices especially are lacking, especially in depth: only the ultimate voice has any depth at all, while the others sound like teenagers who haven't finished puberty. None of the voices sound quite human, but they're all very annoying, and part of that is that they sound like they're acting.
I heard a little DVa from Overwatch.
The only real questions are which Chinese gacha game they ripped data from and whether they used Claude Code or Gemini CLI for Python code. I bet one can get a formant match from output this much overfit to whatever data. This isn't going to stay up for long.
was it cross trained on futurama voices?
That would be a feature!
Sounds like Mort from Family Guy.
Lol
It was not
Impressive technical achievement, but in terms of whether I'd use it: oof, that male voice is like one of these fake-excited newsreaders. Like they're always at the edge of their breath. The female one is better but still someone reading out an advertisement for a product they were told they must act extra excited for. I assume this is what the majority of training data was like and not an intentional setting for the demo. Unsure whether I could get used to that
I use TTS on my phone regularly and recently also tried this new project on F-Droid called SherpaTTS, which grabs some models from Huggingface. They're super heavy (the phone suspends other apps to disk while this runs) and sound good, but in the first news article there were already one or two mispronunciations because it's guessing how to say uncommon or new words and it's not based on logical rules anymore to turn text into speech
Google and Samsung have each a TTS engine pre-installed on my device and those sound and work fine. A tad monotonous but it seems to always pronounce things the same way so you can always work out what the text said
Espeak (or -ng) is the absolute worst, but after 30 seconds of listening closely you get used to it and can understand everything fine. I don't know if it's the best open source option (probably there are others that I should be trying) but it's at least the most reliable where you'll always get what is happening and you can install it on any device without licensing issues
anyone else wants to try sherpaOnnx you can try this.. https://github.com/willwade/tts-wrapper we recently added in the kokoro models which should sound a lot better. There are a LOT of models to choose from. I have a feeling the Droid app isnt handling cold starts very well.
If anyone wants to test ready to install android apks: https://k2-fsa.github.io/sherpa/onnx/tts/apk.html
Thanks a lot for the detailed feedback. We are working on some models which do not use a phonemizer
RHvoice is pretty good, imho.
I hope this is the future. Offline, small ML models, running inference on ubiquitous, inexpensive hardware. Models that are easy to integrate into other things, into devices and apps, and even to drive from other models maybe.
Dedicated single-purpose hardware with models would be even less energy-intensive. It's theoretically possible to design chips which run neural networks and alike using just resistors (rather than transistors).
Such hardware is not general-purpose, and upgrading the model would not be possible, but there's plenty of use-cases where this is reasonable.
But resistors are, even in theory, heat dissipating devices. Unlike transistors, which can in theory be perfectly on or off (in both cases not dissipating heat).
It's theoretically possible but physical "neurons" is a terrible idea. The number of connections between two layers of an FF net is the product of the number of weights in each, so routing makes every other problem a rounding error.
The thing is that the new models keep coming every day. So it’s economically not feasible to make chips for a single model
This is what Apple is envisioning with their SLMs, like having a model specifically for managing calendar events. It doesn't need to have the full knowledge of all humanity in it - just what it needs to manage the calendar.
Issue is their envisioning everyone only using Apple products.
Just like Google wants everyone to use their products. That’s how companies work.
The tech is still public and the research is available
Apple's hardware is notoriously overpriced, so I don't think they're envisioning that at all.
Is it? The base $600 Mac and $150 Apple TV are easily two of the best deals in their market
I'm still betting on a future AppleTV model being a full-on local LLM machine.
This way they could offload as much of the "LLM" work on a device that lives in the home, all family linked phones and devices could use it for local inference.
It's way overpowered as is anyway, why not use it for something useful.
Hmm. A pay once (or not at all) model that can run on anything? Or a subscription model that locks you in, and requires hardware that only the richest megacorps can afford? I wonder which one will win out.
The popular one.
That is our vision too!
This is our goal too.
yeah totally. the quality of these tiny models are only going to go up.
I ran some quick benchmarks.
Ubuntu 24, Razer Blade 16, Intel Core i9-14900HX
Thanks for running the benchmarks. Currently the models are not optimized yet. We will optimize loading etc when we release an SDK meant for production :)
on my Intel(R) Celeron(R) N4020 CPU @ 1.10GHz it takes 6 seconds to import/load and text generation is roughly 1x realtime on various lengths of text.
thanks for testing on the same hardware as mine, before me.
Web version: https://clowerweb.github.io/kitten-tts-web-demo/
It sounds ok, but impressive for the size.
Does anybody find it funny that sci-fi movies have to heavily distort "robot voices" to make them sound "convincingly robotic"? A robotic, explicitly non-natural voice would be perfectly acceptable, and even desirable, in many situations. I don't expect a smart toaster to talk like a BBC host; it'd be enough is the speech if easy to recognize.
A robotic, explicitly non-natural voice would be perfectly acceptable, and even desirable, in many situations[...]it'd be enough is the speech if easy to recognize.
We've had formant synths for several decades, and they're perfectly understandable and require a tiny amount of computing power, but people tend not to want to listen to them:
https://en.wikipedia.org/wiki/Software_Automatic_Mouth
https://simulationcorner.net/index.php?page=sam (try it yourself to hear what it sounds like)
SAM and the way it works is not what people typically associate with the term "formant synthesizer."
DECtalk[1,2] would be a much better example, that's as formant as you get.
[1] https://en.wikipedia.org/wiki/DECtalk [2] https://webspeak.terminal.ink
Well, this one is a bit too jarring to the ears.
But there is no latency, as opposed to KittenTTS, so it certainly has its applications too.
Try this demo, which has more knobs:
https://discordier.github.io/sam/
I think it's charming
Huh, now I know what Airdorf used in Faith: Unholy Trinity.
Yeah blind people love eloquence
This one is at least an interesting idea: https://genderlessvoice.com/
The voice sounds great! I find it quite aesthetically pleasing, but it's far from genderless.
So, what's the gender?
Meet Q, a Genderless Voice - https://news.ycombinator.com/item?id=19505835 - March 2019 (235 comments)
It doesn't sound genderless.
Interesting concept, but why is that site filled with Top X blogspam?
The YouTube video [1] was published in 2019. The Blog spam posts range from Nov 2022 to July 2023.
Other than the video, the only relevant content is on the about page [2]. It says the voice is a collaboration between 5 different entities, including advocacy groups, marketing firms and a music producer.
The video is the only example of the voice in use. There is no API, weights, SDK, etc.
I suspect this was a one-off marketing stunt sponsored by Copenhagen pride before the pandemic. The initial reaction was strong enough that a couple years they were still getting a small but steady flow of traffic. One of the involved marketing firms decided to monetize the asset and defaced it with blog spam.
[1] https://www.youtube.com/watch?v=lvv6zYOQqm0
[2] https://genderlessvoice.com/about/
Huh. Sounds perfectly intelligible and definitively artificial. Feels weakly feminine to me, but only because I was primed to think about gender from the branding.
It’s a good choice for a robot voice. It’s easier to understand than the formant synths or deliberately distorted human voices. The genderless aspect is alien enough to avoid the uncanny valley. You intuitively know you’re dealing with something a little different.
Thanks, I hate it.
In the Culture novels, Iain Banks imagines that we would become uncomfortable with the uncanny realism of transmitted voices / holograms, and intentionally include some level of distortion to indicate you're speaking to an image
Depends on the movie. Ash and Bishop in the Alien franchise sound human until there's a dramatic reason to sound more 'robotic'.
I agree with your wider point. I use Google TTS with Moon+Reader all the time (I tried audio books read by real humans but I prefer the consistency of TTS)
Slightly different there because it's important in both cases that Ripley (and we) can't tell they're androids until it's explicitly uncovered. The whole point is that they're not presented as artificial. Same in Blade Runner: "more human than human". You don't have a film without the ambiguity there.
You're right. I should have used Marvin from Hitchhiker's Guide as an example instead. There's very light processing on his speech.
> I don't expect a smart toaster to talk like a BBC host;
Well sure, the BBC have already established that it's supposed to sound like a brit doing an impersonation of an American: https://www.youtube.com/watch?v=LRq_SAuQDec
I remember that the novelization of the fifth element describes that the cops are taught to speak as robotic as possible when using speakers for some reason. Always found the idea weird that someone would _want_ that
If you're on a Mac, you can type "say [thing to say]" into your terminal.
I personally prefer the older synthetic voices for TTS when the text is coming from software or a language model.
I got an error when I tried the demo with 6 sentences, but it worked great when I reduced the text to 3 sentences. Is the length limit due to the model or just a limitation for the demo?
Currently we don't have chunking enabled yet. We will add it soon. That will remove the length limitations.
Perhaps a length limit? I tried this:
"This first Book proposes, first in brief, the whole Subject, Mans disobedience, and the loss thereupon of Paradise wherein he was plac't: Then touches the prime cause of his fall, the Serpent, or rather Satan in the Serpent; who revolting from God, and drawing to his side many Legions of Angels, was by the command of God driven out of Heaven with all his Crew into the great Deep."
It takes a while until it starts generating sound on my i7 cores but it kind of works.
This also works:
"blah. bleh. blih. bloh. blyh. bluh."
So I don't think it's a limit on punctuation. Voice quality is quite bad though, not as far from the old school C64 SAM (https://discordier.github.io/sam/) of the eighties as I expected.
I tried to replicate their demo text but it doesn't sound as good for some reason.
If anyone else wants to try:
> Kitten TTS is an open-source series of tiny and expressive text-to-speech models for on-device applications. Our smallest model is less than 25 megabytes.
Is the demo using the not smallest model?
Perhaps, but the 25MB model is the only thing they've released
Doesn't work here. Backend module returns 404 :
https://clowerweb.github.io/node_modules/onnxruntime-web/dis...
Looks like this commit 15 minutes ago broke it https://github.com/clowerweb/kitten-tts-web-demo/commit/6b5c...
(seems reverted now)
> Error generating speech: failed to call OrtRun(). ERROR_CODE: 2, ERROR_MESSAGE: Non-zero status code returned while running Expand node. Name:'/bert/Expand' Status Message: invalid expand shape
Doesn't seem to work with thai.
You can also try on https://clowerweb.github.io/node_modules/onnxruntime-web/dis...
Thanks, I was looking for that. While the reddit demo sounds ok, even though on a level we reached a couple of years ago, all TTS samples I tried were barley understandable at all
This is just an early checkpoint. We hope that the quality will improve in the future.
On PC it's a python dependency hell but someone managed to package it in self contained JS code that works offline once it loaded the model? How is that done?
ONNXRuntime makes it fairly easy, you just need to provide a path to the ONNX file, give it inputs in the correct format, and use the outputs. The ONNXRuntime library handles the rest. You can see this in the main.js file: https://github.com/clowerweb/kitten-tts-web-demo/blob/main/m...
Plus, Python software are dependency hell in general, while webpages have to be self-contained by their nature (thank god we no longer have Silverlight and Java applets...)
It feels like it doesn't handle punctuation well. I don't hear sentence boundaries and commas. It sounds like continuous stream of words.
yeah, this is just a preview model from an early checkpoint. the full model release will be next week which includes a 15M model and an 80M model, both of which will have much higher quality than this preview.
besides issues with webgpu (it is in beta fwiw), it'd be nice to increase voice speed through the setting without affecting the voice pitch.
Using male voice 2 at 48kHz at 0.5x speed sounds a lot like Madeline's voice lines in Celeste. Seemed funny to me.
This doesn’t seem to work on Safari. Works great on Chrome, though
Hmm, we will look into it.
You should post on the NVDA email list. https://nvda.groups.io/g/nvda Or the Screen reader list: https://winaccess.groups.io/g/winaccess FYI blind people do not like any lag when reading that’s is why so many still use eloquence and espeak.
[flagged]
Not open source. "You will need internet connectivity to validate your AccessKey with Picovoice license servers ... If you wish to increase your limits, you can purchase a subscription plan." https://github.com/Picovoice/orca#accesskey
The guy is just spamming the project in a lot of comments.
Going online is a dealbreaker but if you really need it you could use ghidra to fix that. I had tried to find a conversion of their model to onnx (making their proprietary pipeline useless) but failed.
Hopefully open source will render them irrelevant in the future.
Does an apk for Android exist for replacing its speech to text engine? I tried sherpa-onnx but it was too slow for real time usage it seemed, and especially so for audiobooks when sped up.
[flagged]
I can't test this out right now, is this just a demo or is it actually an apk for replacing the engine? Because those are two different things, the latter can be used any time you want to read something aloud on the page for example. This is the sherpa-onnx one I'm talking about.
https://k2-fsa.github.io/sherpa/onnx/tts/apk-engine.html
The headline feature isn’t the 25 MB footprint alone. It’s that KittenTTS is Apache-2.0. That combo means you can embed a fully offline voice in Pi Zero-class hardware or even battery-powered toys without worrying about GPUs, cloud calls, or restrictive licenses. In one stroke it turns voice everywhere from a hardware/licensing problem into a packaging problem. Quality tweaks can come later; unlocking that deployment tier is the real game-changer.
yeah, we are super excited to build tiny ai models that are super high quality. local voice interfaces are inevitable and we want to power those in the future. btw, this model is just a preview, and the full release next week will be of much higher quality, along w another ~80M model ;)
> It’s that KittenTTS is Apache-2.0
Have you seen the code[1] in the repo? It uses phonemizer[2] which is GPL-3.0 licensed. In its current state, it's effectively GPL licensed.
[1]: https://github.com/KittenML/KittenTTS/blob/main/kittentts/on...
[2]: https://github.com/bootphon/phonemizer
Edit: It looks like I replied to an LLM generated comment.
The issue is even bigger: phonemizer is using espeak-ng, which isn't very good at turning graphemes into phonemes. In other TTS which rely on phonemes (e.g. Zonos) it turned out to be one of the key issues which cause bad generations.
And it isn't something you can fix, because the model was trained on bad phonemes (everyone uses Whisper + then phonemizes the text transcript).
https://github.com/KittenML/KittenTTS/issues/17
> IANAL, but AFAICS this leaves 2 options, switching the license or removing that dependency.
There is a third option: asking the project for an exception.
Though that is unlikely to be granted¹ leaving you back with just the other two options.
And of course a forth choice: just ignore the license. This is the option taken by companies like Onyx, whose products I might otherwise be interested in…
----
[1] Those of us who pick GPL3 or AGPL generally do so to keep things definite and an exception would muddy the waters, also it might not even be possible if the project has many maintainers as relicensing would require agreement from all who have provided code that is in the current release. Furthermore, if it has inherited the license from one of its dependencies, an exception is even less practical.
> There is a third option: asking the project for an exception.
IIUC, the project isn't at the liberty to grant such an exception because it inherits its GPL license from espeak-ng.
Ah, yes, good catch, I didn't look deeper into the dependency tree at all. I'll update my footnote to include that as one of the reasons an exception may be impossible (or at least highly impractical).
A fourth option would be a kind of dual-licensing: the project as-is is available under GPL-3.0, but the source code in this repository excluding any dependencies is also available under Apache 2.0
Any user would still effectively be bound by the GPL-3.0, but if someone can remove the GPL dependencies they could use the project under Apache
That is an option for the publisher of the library, not the consumer of it. If it isn't already done then asking for it to be done is the same as asking for an exception otherwise (option three).
The use of the library is four lines. Three set up the library (`phonemizer.backend.EspeakBackend(language="en-us", preserve_punctuation=True, with_stress=True)`), the other calls it (`phonemes_list = self.phonemizer.phonemize([text])`). Plus I guess the import statements. Even ignoring Google vs Oracle I don't think those lines by themselves meet any threshold of originality.
Obviously you can't run them (with the original library) without complying with the GPL. But I don't see why I couldn't independently of that also give you this text file under Apache 2.0 to do with as you want (which for the record still doesn't allow you to run them with the original library without complying with the GPL, but that'd be phoneme forcing you to do that, not this project)
You would have to be very specific about the dual-licensing to avoid confusion about what you are allowed to do under Apache conditions though. You can't just say "it's dual-licensed"
You could even extract out the parts that do not call the GPL library into an upstream project under the Apache 2.0 licence, and pull in both that and the GPL library in the downstream project, relying on Apache 2.0 -> GPL 3.0 compatibility instead of explicit dual licensing to allow the combined work to be distributed under GPLv3.
Once the license issues are resolved it would nice if you could install it on a distro with the normal package manager.
This would only apply if they were distributing the GPL licensed code alongside their own code.
If my MIT-licensed one-line Python library has this line of code…
…I’m not suddenly subject to bash’s licensing. For anyone wanting to run my stuff though, they’re going to need to make sure they themselves have bash installed.(But, to argue against my own point, if an OS vendor ships my library alongside a copy of bash, do they have to now relicense my library as GPL?)
The FSF thinks it counts as a derivative work and you have to use the LGPL to allow linking.
However, this has never actually been proven in court, and there's many good arguments that linking doesn't count as a derivative work.
Old post by a lawyer someone else found (version 3 wouldn't affect this) [1]
For me personally I don't really understand how, if dynamic linking was viral, using linux to run code isn't viral. Surely at some level what linux does to run your code calls GPLed code.
It doesn't really matter though, since the FSF stance is enough to scare companies from not using it, and any individual is highly unlikely to be sued.
[1] https://www.linuxjournal.com/article/6366
> For me personally I don't really understand how, if dynamic linking was viral, using linux to run code isn't viral. Surely at some level what linux does to run your code calls GPLed code.
The Linux kernel has an explicit exception for userspace software:
> NOTE! This copyright does not cover user programs that use kernel services by normal system calls
And the GPL also has an explicit exception for "system" software such as kernel, platform libraries etc.:
> The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
> The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work.
> This would only apply if they were distributing the GPL licensed code alongside their own code.
As far as I understand the FSF's interpretation of their license, that's not true. Even if you only dynamically link to GPL-licensed code, you create a combined work which has to be licensed, as a whole, under the GPL.
I don't believe that this extends to calling an external program via its CLI, but that's not what the code in question seems to be doing.
(This is not an endorsement, but merely my understanding on how the GPL is supposed to work.)
This is a false analogy. It's quite straightforward.
Running bash (via exec()/fork()/spawn()/etc) isn't the same as (statically or dynamically) linking with its codebase. If your MIT-licensed one-liner links to code that's GPL licensed, then it gets infected by the GPL license.
I've seen people use IPC to workaround the GPL, but I've also seen the FSF interpretations claiming that is still a derived work.
I don't know if this has ever been tested in court.
My interpretation of their FAQ[1] on it is that shelling out and IPC are fine, while linking is not. As you say, it's ultimately up to the courts to decide on.
[1]: https://www.gnu.org/licenses/gpl-faq.html#MereAggregation
you are correct. its about linking as in LD does it, not conceptual linking.
GPL is for boomers at this point. Floppy disks? Distribution? You can use a tool but you cant change it? A DLL call means you need to redistribute your code but forking doesn't?
Sillyness
GPL post-dates network software distribution (we got our first gcc via ftp).
Yes, but if you use open source libraries for your closed source SaaS - thats fine. People get their software _over_ the network delivered to them in a VM (your browser).
Given that the FSF considers Apache-2.0 to be compatible with GPL-3.0 [0], how could the fact that phonemizer is GPL-3.0 possibly be an issue?
[0]: https://www.gnu.org/licenses/license-list.html#apache2
Compatible means they can be linked together, BUT the result is GPL-3.
> the result is GPL-3
The result can only be distributed under the terms of the GPL-3. That's actually a crucial difference: there's nothing preventing Kitten TTS from being Apache licensed, soliciting technical contributions under that license, and parts of its code being re-used in other software under that license. Yes, for the time being, this limits what you can do with Kitten TTS if you want to use the software as a whole (e.g. by embedding it into your product), but the license itself is still Apache and that can have value.
Okay, what's stopping you from feeding the code into an LLM and re-write it and make it yours? You can even add extra steps like make it analyze the code block by block then supervise it as it is rewriting it. Bam. AI age IP freedom.
Morals may stop you but other than that? IMHO all open source code is public domain code if anyone is willing to spend some AI tokens.
That would be a derivative work, and still be subject to the license terms and conditions, at best.
There are standard ways to approach this called clean room engineering.
https://en.m.wikipedia.org/wiki/Clean-room_design
One person reads the code and produces a detailed technical specification. Someone reviews it to ensure that there is nothing in there that could be classified as copyrighted material, then a third person (who has never seen the original code) implements the spec.
You could use an LLM at both stages, but you'd have to be able to prove that the LLM that does the implementation had no prior knowledge of the code in question... Which given how LLMs have been trained seems to me to be very dubious territory for now until that legal situation gets resolved.
AI is useful in Chinese walling code, but it’s not as easy as you make it sound. To stay out of legal trouble, you probably should refactor the code into a different language, then back into the target language. In the end, it turns into a process of being forced to understand the codebase and supervising its rewriting. I’ve translated libraries into another language using LLMs, I’d say that process was 1/2 the labor of writing it myself. So in the end, going 2 ways, you may as well rewrite the code yourself… but working with the LLM will make you familiar with the subject matter so you -could- rewrite the code, so I guess you could think of it as a sort of buggy tutorial process?
I am not sure even that is enough. You would really need to do a clean room reimplementation to be safe - for exactly the same reasons that people writing code write clean room reimplementations.
Yeah, the algorithms and program flow would have to be materially distinct to be really safe. Maybe switching language paradigms would get that for you in most cases? Js->haskell->js? Sounds like a nightmare lol.
Tell me you haven't used LLMs on large, non-trivial codebases without telling me... :)
Tell me you don't know how to use LLMs properly without telling me.
You don't give the whole codebase to an LLM and expect it to have one shot output. Instead, you break it down and and write the code block by block. Then the size if the codebase doesn't matter. You use the LLM as a tool, it is not supposed to replace you. You don't try to become George from Jetsons who is just pressing a button and doesn't touch anything, instead you are on top of it as the LLM does the coding. You test the code on every step to see if the implementation behaves as expected. Do enough of this and you have proper, full "bespoke" software.
I'll help you along - this is the core function that Kitten ends up calling. Good luck!
https://github.com/espeak-ng/espeak-ng/blob/a4ca101c99de3534...
A Festival's English model, festvox-kallpc16k, is about 6 MB, and it is a large model; festvox-kallpc8k is about 3.5 MB.
eSpeak NG's data files take about 12 MB (multi-lingual).
I guess this one may generate more natural-sounding speech, but older or lower-end computers were capable of decent speech synthesis previously as well.
Custom voices could be added, but the speed was more important to some users.
$ ls -lh /usr/bin/flite
Listed as 27K last I checked.
I recall some Blind users were able to decode Gordon 8-bit dialogue at speeds most people found incomprehensible. =3
I'm not blind but spoken English it's far more difficult to grasp than written one (I'm a non-native speaker), and Flite runs on n270 netbooks at crazy speeds with really good enough voices.
> KittenTTS is Apache-2.0
What about the training data? Is everyone 100% confident that models are not a derived work of the training inputs now, even if they can reproduce input exactly?
I play around with a nvidia jetson orin nano super right now and its actually pretty usuable with gemma3:4b and quite fast - even image processing is done in like 10-20 seconds but this is with GPU support. When something is not working and ollama is not using the GPU this calls take ages because the cpu is just bad.
Iam curious how fast this is with CPU only.
It depends on espeak-ng which is GPLv3
This opens up voice interfaces for medical devices, offline language learning tools, and accessibility gadgets for the visually impaired - all markets where cloud dependency and proprietary licenses were showstoppers.
But Pi Zero has a GPU, so why not make use of it?
Because then you're stuck on that device only.
The github just has a few KB of python that looks like an install script. How is this used from C++ ?
On another machie the python version is too new, and the package/dependencies don't want to install.
I opened a couple of PRs to fix this situation:
https://github.com/KittenML/KittenTTS/pull/21 https://github.com/KittenML/KittenTTS/pull/24 https://github.com/KittenML/KittenTTS/pull/25
If you have `uv` installed, you can try my merged ref that has all of these PRs (and #22, a fix for short generation being trimmed unnecessarily) with
Thanks for the quick intro into UV, it looks like docker layers for python
I found the TTS a bit slow so I piped the output into ffplay with 1.2x speedup to make it sound a bit better
Ah, yeah, good catch – I added the model-native speed multiplier to the CLI too (`--speed=1.2` for instance).
https://github.com/KittenML/KittenTTS/pull/21/commits/0aacfc...
Nice one, thanks!
Install it with uvx that should solve the python issues.
https://docs.astral.sh/uv/guides/tools/
uv installation:
https://docs.astral.sh/uv/getting-started/installation/
Yeah some people have a problem and think "I'll use Python". Now they have like fifty problems.
I had the too new.
This package is the epitome of dependency hell.
Seriously, stick with piper-tts.
Easy to install, 50MB gives you excellent results and 100MB gives you good results with hundreds of voices.
It doesn't work on Fedora because of the lack of g++ having the right version.
Not sure if they've fixed between then and now, but I just had it working locally on Fedora.
We are working to fix that. Thanks
"Fixing python packaging" is somewhat harder than AGI.
I was commiserating with my brother over how difficult it is to set up an environment to run one LLM or diffusion model, let alone multiple or a combination. It's 5 percent CUDA/ROCm difficulties and 95% Python difficulties. We have a theory that Lanyone working with generative AI has to tolerate output that is only 90% right, and is totaly fine working with a language and environment that only 90% works.
Why is Python so bad at that? It's less kludgy than Bash scripts, but even those are easier to get working.
This is a generic problem.
JS/TS/npm is just as bad with probably more build tools/frameworks.
Rust is a mess.
Go, well.
Even perl was quite complicated.
Yeah, but it's easily solved, with directives, headers, or make files that specify which language standard it follows. Better yet, you can use different syntax with different language standards, so it's clear which to follow. If a compiler can automatically figure whether I'm compiling C or C++, why can't a Python interpreter figure out if I'm running version two or three, of the same language?
> JS/TS/npm is just as bad with probably more build tools/frameworks.
This is flat out wrong. NPM packages by default are local to a directory. And I haven't seen a package rely on a specific minor version of node in literally years. Node's back compat is also great, there was one hiccup 5 or 6 years ago where a super popular native package was deprecated ago but that's been about it.
I can take current LTS node and run just about any package from the NPM repo written within the last 4 or 5 years and it will just work. Meanwhile plenty of python packages somehow need specific point releases. What the unholy hell.
Node version manager does exist, and it can be setup to work per directory, which is super cool, but I haven't needed NVM in literal years.
This is how we'll know ASI has arrived.
Have you considered offering a uvx command to run to get people going quickly?
Though I think you would still need to have the Python build dependencies installed for that to work.
If you restrict your dependencies to only those for which wheels are available, then uv should just be able to handle them for you.
I think it can install Python itself too. Though I have had issues with that - especially with SSL certificate locations, which is one of Linux's other clusterfucks.
Just point people to uv/uvx.
A tool that was only released, what, a year or two ago? It simply won't be present in nearly all OS/distros. Only modern or rolling will have it (maybe). It's funny when the recommended python dependency manager managers are just as hard to install and use as the script themselves. Very python.
[flagged]
[flagged]
The project is like 80% there by having a pyproject file that should work with uv and poetry. The just aren't any package versions specified and the python version is incredibly lax, and no lock file is provided.
in this context uv works perfectly fine with poertry, if you publish a wheel from poetry uv can use it. You don't have to switch anything in your project to make it work.
Python man
Python is used not because it's good but because it's good enough just like Windows and plastics.
I thought we were doing https://www.gnu.org/fun/jokes/ed-msg.html.
You're getting a lot of comments along the lines of "Why don't you just ____," which only shows how Stockholmed the entire Python community is.
With no other language are you expected to maintain several entirely different versions of the language, each of which is a relatively large installation. Can you imagine if we all had five different llvms or gccs just to compile five different modern C projects?
I'm going to get downvoted to oblivion, but it doesn't change the reality that Python in 2025 is unnecessarily fragile.
That’s exactly what I have. The C++ codebases I work on build against a specific pinned version of LLVM with many warnings (as errors) enabled, and building with a different version entails a nonzero amount of effort. Ubuntu will happily install several versions of LLVM side by side or compilation can be done in a Docker container with the correct compiler. Similarly, the TypeScript codebases I work with test against specific versions of node.js in CI and the engine field in package.json is specified. The different versions are managed via nvm. Python is the same via uv and pyproject.yaml.
I don't doubt it, but I don't think that situation is accepted as the default in C/C++ development. For the most part, I expect OSS to compile with my own clang.
I agree with your point, but
> if we all had five different llvms or gccs
Oof, those are poor examples. Most compilers using LLVM other than clang do ship with their own LLVM patches, and cross-compiling with GCC does require installing a toolchain for each target.
Cross-compiling is a totally different subject… I'm trying to make an apples-to-apples comparison. If you compile a lot of OSS C projects for the host architecture, you typically do not need multiple LLVMs or GCCs. Usually, the makefile detects various things about the platform and compiler and then fails with an inscrutable error. But that is a separate issue! haha
> Can you imagine if we all had five different llvms or gccs just to compile five different modern C projects?
Yes, because all I have to do is look at the real world.
There are still people who use machine wide python installs instead of environments? Python dependency hell was already bad years ago, but today it's completely impractical to do it this way. Even on raspberries.
Yep. Python stopped being Python a decade ago. Now there are just innumberable Pythons. Perl... on the otherhand, you can still run any perl script from any time on any system perl interpreter and it works! Granted, perl is unpopular and not getting constant new features re: hardcore math/computation libs.
Anyway, I think I'll stick with Festival 1.96 for TTS. It's super fast even on my core2duo and I have exactly zero chance of getting this Python 3'ish script to run on any machine with an OS older than a handful of years.
It breaks my heart that Perl fell out of favor. Perl “6” didn’t help in the slightest.
Debian pretty much "solved" this by making pip refuse to install packages if you are not in an venv.
It needed distro buy in and implementation, but this is from the Python side: https://peps.python.org/pep-0668/
IIRC that's actually a change in upstream pip.
Well, with my python 3.13.5 not even that works!
Pretty impressive but this seems to be a staple of most AI/ML projects.
"Works on my machine" or "just use docker", although here the later doesn't even seem to be an option.
Ditto OpenSUSE, at least on Tumbleweed
Using venv won't save you from having the wrong version of the actual Python interpreter installed.
venvs can be attached to arbitrary interpreters.
Such an ignorant thing to say for something that requires 25MB RAM.
Not sure what the size has to do with anything.
I send you a 500kb Windows .exe file and claim it runs literally everywhere.
Would it be ignorant to say anything against it because of its size?
we all know runs anywhere in this context means compute wise. It's dumb to blame author for your dev setup issues.
I didn’t realize that that’s what it meant until you mentioned it.
It reminds me of the costs and benefits of RollerCoaster Tycoon being written in assembly language. Because it was so light on resources, it could run on any privately owned computer, or at least anything x86, which was pretty much everything at the time.
Now, RISC architectures are much more common, so instead of the rare 68K Apple/Amiga/etc computer that existed at the time, it's super common to want to run software on an ARM or occasionally RISC-V processor, so writing in x86 assembly language would require emulation, making for worse performance than a compiled language.
system python is for system applications that are known to work together. If you need a python install for something else, there's venv or conda and then pip install stuff.
[flagged]
You're supposed to use venv for everything but the python scripts distributed with your os
I tried it. Not bad for the size (of the model) and speed. Once you install all the massive number of libraries and things needed we are a far cry away from 25MB though. Cool project nonetheless.
That's a great point about the dependencies.
To make the setup easier and add a few features people are asking for here (like GPU support and long text handling), I built a self-hosted server for this model: https://github.com/devnen/Kitten-TTS-Server
The goal was a setup that "just works" using a standard Python virtual environment to avoid dependency conflicts.
The setup is just the standard git clone, pip install in a venv, and python server.py.
Oh wow, really impressive. How long did this take you to make?
It didn't take too long. I already have two similar projects for Dia and Chatterbox tts models so I just needed to convert a few files.
It mentions ONNX, so I imagine an ONNX model is or will be available.
ONNX runtime is a single library, with C#'s package being ~115MB compressed.
Not tiny, but usually only a few lines to actually run and only a single dependency.
The repository already runs an ONNX model. But the onnx model doesn't get English text as input, it gets tokenized phonemes. The prepocessing for that is where most of the dependencies come from.
Which is completely reasonable imho, but obviously comes with tradeoffs.
For space sensitive applications like embedded systems, could you shift the preprocessing to compile time?
You would need to constrain the vocabulary to see any benefits, but that could be reasonable. For example, you an enumeration of numbers, units and metric names could handle dynamic time, temperature and other dashboard items.
For something more complex like offline navigation, you already need to store a map. You could store street names as tokens instead of text. Add a few turn commands, and you have offline spoken directions without on device pre-processing.
We will try to get rid of dependencies.
Usually pulling in lots of libraries helps develop/iterate faster. Then can be removed later once the whole thing starts to take shape.
This case might be different, but ... usually that "later" never happens.
I don't mind so much the size in MB, the fact that it's pure CPU and the quality, what I do mind however is the latency. I hope it's fast.
Aside: Are there any models for understanding voice to text, fully offline, without training?
I will be very impressed when we will be able to have a conversation with an AI at a natural rate and not "probe, space, response"
Nvidia's parakeet https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2 appears to be state of the art for english: 10x faster than Whisper.
My mid-range AMD CPU is multiple times faster than realtime with parakeet.
>Aside: Are there any models for understanding voice to text, fully offline, without training?
OpenAI's whisper is a few years old and pretty solid.
https://github.com/openai/whisper
Whisper tends to fill silence with random garbage from its training set. [0] [1] [2]
[0]: https://github.com/openai/whisper/discussions/679 [1]: https://github.com/openai/whisper/discussions/928 [2]: https://github.com/openai/whisper/discussions/2608
Voice to text fully offline can be done with whisper. A few apps offer it for dictation or transcription.
"The brown fox jumps over the lazy dog.."
Average duration per generation: 1.28 seconds
Characters processed per second: 30.35
--
"Um"
Average duration per generation: 0.22 seconds
Characters processed per second: 9.23
--
"The brown fox jumps over the lazy dog.. The brown fox jumps over the lazy dog.."
Average duration per generation: 2.25 seconds
Characters processed per second: 35.04
--
processor : 0
vendor_id : AuthenticAMD
cpu family : 25
model : 80
model name : AMD Ryzen 7 5800H with Radeon Graphics
stepping : 0
microcode : 0xa50000c
cpu MHz : 1397.397
cache size : 512 KB
Hmm that actually seems extremely slow, Piper can crank out a sentence almost instantly on a Pi 4 which is a like a sloth compared to that Ryzen and the speech quality seems about the same at first glance.
I suppose it would make sense if you want to include it on top of an LLM that's already occupying most of a GPU and this could run in the limited VRAM that's left.
assuming most answers will be more than a sentence, 2.25 seconds is already long enough if you factor the token generation in between... and imagine with reasoning!... We're not there yet.
Any idea what factors play into latency in TTS models?
Mostly model size, and input size. Some models which use attention are O(N^2)
Cool.
While I think this is indeed impressive and has a specific use case (e.g. in the embedded sector), I'm not totally convinced that the quality is good enough to replace bigger models.
With fish-speech[1] and f5-tts[2] there are at least 2 open source models pushing the quality limits of offline text-to-speech. I tested F5-TTS with an old NVidia 1660 (6GB VRAM) and it worked ok-ish, so running it on a little more modern hardware will not cost you a fortune and produce MUCH higher quality with multi-language and zero-shot support.
For Android there is SherpaTTS[3], which plays pretty well with most TTS Applications.
1: https://github.com/fishaudio/fish-speech
2: https://github.com/SWivid/F5-TTS
3: https://github.com/woheller69/ttsengine
We have released just a preview of the model. We hope to get the model much better in the future releases.
Fish Speech says its weights are for non-commercial use.
Also, what are the two's VRAM requirents? This model has 15 million parameters which might run on low-power, sub-$100 computers with up-to-date software. Your hardware was an out-of-date 6GB GPU.
Hmm the quality is not so impressive. I'm looking for a really naturally sounding model. Not very happy with piper/kokoro, XTTS was a bit complex to set up.
For STT whisper is really amazing. But I miss a good TTS. And I don't mind throwing GPU power at it. But anyway. this isn't it either, this sounds worse than kokoro.
> Hmm the quality is not so impressive. [...] And I don't mind throwing GPU power at it.
This isn't for you, then. You should evaluate quality here based on the fact you don't need a GPU.
Back in the pre-Tacotron2 days, I was running slim TTS and vocoder models like GlowTTS and MelGAN on Digital Ocean droplets. No GPU to speak of. It cost next to nothing to run.
Since then, the trend has been to scale up. We need more models to scale down.
In the future we'll see small models living on-device. Embedded within toys and tools that don't need or want a network connection. Deployed with Raspberry Pi.
Edge AI will be huge for robotics, toys and consumer products, and gaming (ie. world models).
> This isn't for you, then. You should evaluate quality here based on the fact you don't need a GPU.
I know but it was more of a general comment. A really good TTS just isn't around yes in the OSS sphere. I looked at some of the other suggestions here but they have too many quirks. Dia sounds great but messages must have certain lengths etc and it picks a random voice every time. I'd love to have something self hosted that's as good as openai.
The best open one I've found so far is Dia - https://github.com/nari-labs/dia - it has some limitations, but i think it's really impressive and I can run it on my laptop.
Thanks I'll try! I like how it sounds, the quality is really good. But the limitations are really severe (shorter than 5 seconds is not ok, > 30 seconds is not ok, it will play a random voice every time, those make it pretty much unusable for an assistant to be honest).
But it might be worth setting it up and seeing if it improves over time.
You can get consistent voice by providing a sample - and yea the timing stuff is what you have to work around - have to basically chunk your inputs.
You should give try to https://pinokio.co/
Thanks I'll try!
Imho chatterbox is the current open weight SOTA model in terms of quality: https://huggingface.co/ResembleAI/chatterbox
Thank you, I hadn't heard of it. Will have a look! The samples sound excellent indeed.
Chatterbox is also worth a try.
Try https://github.com/Picovoice/orca
Thanks!
Microsoft's and some of Google's TTS models make the simplest mistakes. For instance, they sometimes read "i.e." as "for example." This is a problem if you have low vision and use TTS for, say, proofreading your emails.
Why does it happen? I'm genuinely curious.
Well, speech synthesizers are pretty much famous for speaking all sorts of things wrong. But what I find very concerning about LLM based TTS is that some of them cant really speak numbers greater then 100. They try, but fail a lot. At least tts-1-hd was pretty much doing this for almost every 3 or 4 digit number. Especially noticeable when it is supposed to read a year number.
From the web demo this model is really good at numbers. It rushes through them, slurs them a bit together, but they are all correct, even 7 digit numbers (didn't test further).
Looks like they are sidestepping these kinds of issues by generating the phonemes with the preprocessing stage of traditional speech synthesizers, and using the LLM only to turn those phonemes into natural-ish sounding speech. That limits how natural the model can become, but it should be able to correctly pronounce anything the preprocessing can pronounce
Not entirely related but humans have the same problem.
For scriptwriting when doing voice overs we always explicitly write out everything. So instead of 1 000 000 we would write one million or a million. This is a trivial example but if the number was 1 548 736 you will almost never be able to just read that off. However one million, five hundred and forty eight thousand, seven hundred and thirty six can just be read without parsing.
Same with urls, W W W dot Google dot com.
Regarding humans, yes and no. If a human had constantly problems with 3 and 4 digit numbers like tts-1-hd does, I'd ask myself if they were neurodivergent in some way.
And yes, I added instructions along the lines of what you describe to my prompt. Its just sad that we have to. After all, LLM TTS has solved a bunch of real problems, like switching languages in a text, or foreign words. The pronounciation is better then anything we ever had. But it fails to read short numbers. I feel like that small issue could probably have been solved by doing some fine tuning. But I actually dont really understand the tech for it, so...
They're often trained from video subtitles, and humans writing subtitles make that kind of mistake too.
You probably mean "e.g." as "for example", not "i.e."?
This might be on purpose and part of the training data because "for example" just sounds much better than "e.g.". Presumably for most purposes, linguistic naturalness is more important than fidelity.
Sometimes I use “for example” and “e.g.” in consecutive sentences to not sound repetitive, or possibly even within the same sentence (e.g. in parentheses). In that case, speaking both as “for example” would degrade it linguistically.
In any case, I’d like TTS to not take that kind of artistic freedom.
I did mean "i.e." That's why it is a problem : - \
https://huggingface.co/KittenML/kitten-tts-nano-0.1
https://github.com/KittenML/KittenTTS
This is the model and Github page, this blog post looks very much AI generated.
Wow, amazing and good work, I hope to see more amazing models running on CPUs!
thanks, we're going to release many more models in the future, that can run on just CPUs.
amazing! can't wait to integrate it into https://desktop.with.audio I'm already using KokorosTTS without a GPU. It works fairly well on Apple Silicon.
Foundational tools like this open up the possiblity of one-time payment or even free tools.
would love to see how that turns out. the full model release next week will be more expressive and higher quality than this one so we're excited to see you try that out.
Where does the training data come for the models? Is there an openly available dataset the people use?
This great for english, but is there something similar for other languages? Could this be trained somehow to support other languages?
Okay, lots of details information and example code, great. But skimming through I didn’t see any audio samples to judge the quality?
They posted a demo on reddit[0]. It sounds amazing given the tiny size.
[0] https://old.reddit.com/r/LocalLLaMA/comments/1mhyzp7/kitten_...
Thanks! Yeah. It definitely isn’t the absolute best in quality but it trounces the default TTS options on macOS (as third party developers are locked out of the Siri voices). And for less than the size of many modern web pages…
Most of these comments were originally posted to a different thread (https://news.ycombinator.com/item?id=44806543). I've moved them hither because on HN we always prefer to give the project creators credit for their work.
(it does however explain how many of these comments are older than the thread they are now children of)
The samples featured elsewhere seem to be from a larger model?
After testing this locally, it still sounds quite mechanical, and fails catastrophically for simple phrases with numbers ("easy as 1-2-3"). If the 80M model can improve on this and keep the expressiveness seen in the reddit post, that looks promising.
Might be useful to split it up to use the same speech features as some off the shelf vocoder, such as FARGAN used by RADE (https://freedv.org/radio-autoencoder/) and DRED (https://jmvalin.ca/papers/valin_dred_journal.pdf).
Hi. Will the training and fine-tuning code also be released?
It would be great if the training data were released too!
I'm curious why smallish TTS models have metallic voice quality.
The pronunciation sounds about right - i thought it's the hard part. And the model does it well. But voice timbre should be simpler to fix? Like, a simple FIR might improve it?
We change our tone based on personal style, emotion, context, and other factors. An accurate generator might need to encode all that information in the model. It will be larger than a model that doesn't do all of that.
Probably "metallicity" is due to lack of details and cannot be fixed that easy.
Question for the experts here; What would be a SOTA TTS that can run on an average laptop (32GB RAM, 4GB VRAM). I just want to attach a TTS to my SLM output, and get the highest possible voice quality/ human resembleness.
Try Unmute by Kyutai - https://unmute.sh/
Someone please port this to ONNX so we don't need to do all this ass tooling
What's a good one in reverse; speech to text?
Whisper and the many variants. Here's a good implementation.
https://github.com/ggml-org/whisper.cpp
This one is a whisper-based Python package
https://github.com/primaprashant/hns
It is not the best TTS but it is freaking amazing it can be done by such a small model and it is good enough for so many use cases.
thanks, but keep in mind that this model is just a preview checkpoint that is only 10% trained. the full release next week will be of much higher quality and it will include a 15M model and an 80M model.
How does this compare to say piper-tts?
I ask because their models are pretty small. Some sound awesome and there is no depdendency hell like I'm seeing here.
Example: https://rhasspy.github.io/piper-samples/#en_US-ryan-high
I am blind and use NVDA with a sinth. How is this news? I don't get it! My sinth is called eloquence and is 4089KB
Does your Eloquence installation include multiple languages? The one I have is only 1876 KB for US English only. And classic DECtalk is even smaller; I have here a version that's only 638 KB (again, US English only).
I like the direction we are heading. Build models that can run on CPUs & AI can become even more mainstream.
This feels different. This feels like a genuinely monumental release. Holy cow.
Very well done. The quality is excellent and the technical parameters are, simply, unbelievable. Makes me want to try to embed this on a board just to see if it's possible.
This is a fun model for circuit-bending, because the voice style vectors are pretty small.
For instance, try adding `np.random.shuffle(ref_s[0])` after the line `ref_s = self.voices[voice]`...
EDIT: be careful with your system volume settings if you do this.
I'm so confused on how the model is actually made. It doesn't seem to be in the code or this stuff is way simpler than i thought. It seems to use a fancy library from japan, not sure how much it's just that
How does one build similar model, but for different languages? I was under impression that being open source, there would be some instructions how to build everything on your own.
Have you considered adding some 'rendered' examples of what the model sounds like?
I'm curious, but right now I don't want to install the package and run some code.
Chrome does TTS too.
https://codepen.io/logicalmadboy/pen/RwpqMRV
Not bad for the size (with my very limited knowledge of this field) !
In a couple tests, the "Male 2" voice sounds reasonable, but I've found it has problem with some groups of words, specially when played with little context. I think it's small sentences.
For example, if you try to do just "Hey gang!", it will sound something like "Chay yang". But if you add an additional sentence after that, it will sound a bit different (but still weird).
Is this english only?
If you're looking for other languages, Piper has been around in this scene for much longer and they have open-source training code and a lot of models (they're ~60MB instead of 25MB but whatever...) https://huggingface.co/rhasspy/piper-voices/tree/main
[flagged]
you need api key and internet access to run locally? lol. Classic .NET project.
Yeah, they're just spamming the threads at the moment with their commercial product.
Actually I found it irritating that the readme does not mention the language at all. I think it is not good practice to deduce it from the language of the readme itself. I would not like to have German language tts models with only a German readme...
I tried on some Japanese for the kicks of it, it reads... "Chinese letter chinese letter japanese letter chinese letter..." :D
But yeah, if it's like any of the others we'll likely see a different "model" per language down the line based on the same techniques
TTS is generally not multilingual. One might think a well-annotated phonetic descriptions of voices would suffice, but that's not quite how languages work nor how TTS work.
(but somehow LLMs handle multilingual input perfectly fine! that's a bit strange, if you think about that)
Yes. The FAQ says that multilingual capabilities are in the works.
I’m new to TTS models but is this something I can plug into my own engine like with LLMs, or does it require the Python stack it ships with?
Atom n270 running flite with a good voice -slt- vs this... would it be fast enough to play a MUD? Flite it's almost realtime fast...
Would love to se something like this trained for multilingual purposes. It seems kinda like the same tier as piper, but a bit faster.
Awesome work! Often times in the TTS space, human-similarity is given way too much emphasis at the expense of hurting user access. Frankly as long as a voice is clear and you listen to it for a while, the brain filters out most quirks you would perceive on the first pass. Hence why many blind folks still are perfectly fine using espeak-ng. The other properties like speed of generation and size make it worth it.
I've been using a custom AI audiobook generation program [0] with piper for quite a while now and am very excited to look at integrating kitten. Historically piper has been the only good option for a free CPU-only local model so I am super happy to see more competition in the space. Easy installation is a big deal, since piper historically has had issues with that. (Hence why I had to add auto installation support in [0])
[0] https://github.com/C-Loftus/QuickPiperAudiobook
What I am still looking for is a way to clone voice locally. I have OK hardware. For example I can use Mistral Small 3.1 or what it is called locally. Premade voices can be interesting too, but I am looking for custom voice. Perhaps by providing audio and the corresponding transcript to the model, training it, and then give it a new text and let it speak that.
I have tested its speed on CPU and compared it with Piper, kokoro, and matcha. See https://github.com/KittenML/KittenTTS/issues/40
BEAT THIS! Commodore C64 has the same feature called SAM - speaker synthesizer, speaks English and Polish. 48 kB of RAM
BEAT THIS!
It's not the same feature, but at least that's not several orders of magnitude away from "run anywhere"
TL;DR: If you are interested in TTS, you should explore alternatives
I tried to use it...
Its python venv has grown to 6 GBytes in size. The demo sentence
> "This high quality TTS model works without a GPU"
works, it takes 3s to render the audio. Audio sounds like a voice in a tin can.
I tried to have a news article read aloud and failed with
> [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running Expand node. Name:'/bert/Expand' > Status Message: invalid expand shape
If you are interested in TTS, you should explore alternatives
say is only 193K on MacOS
Usage:That’s not a far comparison. Say just calls the speech synthesis APIs that have been around since at least Mac OS 8.
That being said, the ‘classical’ (pre-AI) speech synthesisers are much smaller than kitten, so you’re not wrong per se, just for the wrong reason.
The linked repository at the top-level here has several gigabytes of dependencies, too.
SAM on Commodore 64 was only 6K:
https://project64.c64.org/Software/SAM10.TXT
Obviously it's not fair to compare these with ML models.
And what dynamic libraries s it linked to? And what other data are they pulling in?
`say` sounds terrible compared to modern neural network based text to speech engines.
Sounds about the same as Kitten TTS.
To me it sounds worse, especially on the construction of certain more complex sentences or words.
Tried that on 26 beta, and the default voice sounds a lot smoother than it used it.
Running `man say` reveals that "this tool uses the Speech Synthesis manager", so I'm guessing the Apple Intelligence stuff is kicking in.
Nothing to do with Apple Intelligence. The speech synthesiser manager (the term manager was used for OS components in Classic Mac OS) has been around since the mid 90s or so. The change you’re hearing is probably a new/modified default voice.
> Nothing to do with Apple Intelligence Sorry, the clarify - I mean in macOS beta 26, the `say` command's output matches the settings set in the System Settings -> Apple Intelligence & Siri -> Voice > Select... menu.
> speech synthesiser manager (the term manager was used for OS components in Classic Mac OS)
Especially fun to play with on the rainbow iMacs back then, too.
If you make a shell script that calls say, that script will be even smaller!
Not bad. Something I would not want to listen for long without more clarity. Could work very well for non-english speakers in various tools an such.
Can coqui run in cpu only?
Yes, XTTS2 has been reasonably performant for me and the cloning is acceptable.
I wonder what would it take to extend it with a custom voice?
One thing any GitHub project never has. A few-second demo.
A localized version of this, and I could finally build my tiny Amazon Echo replacement. I would love to see all speech synthesis performed on a local device.
I'm doing this now with Home Assistant voice. All the TTS, STT, and LLMs involved run locally on my network. It's absurdly superior to every other voice assistant product. (Would be nice if it was just a pure multi-modal model though)
Good TTS feels like it is something that should be natively built into every consumer device. So the user can decide if they want to read or listen to the text at hand.
I'm surprised that phone manufacturers do not include good TTS models in their browser APIs for example. So that websites can build good audio interfaces.
I for one would love to build a text editor that the user can use completely via audio. Text input might already be feasible via the "speak to type" feature, both Android and iOS offer.
But there seems to be no good way to output spoken text without doing round-trips to a server and generate the audio there.
The interface I would like would offer a way to talk to write and then commands like "Ok editor, read the last paragraph" or "Ok editor, delete the last sentence".
It could be cool to do writing this way while walking. Just with a headset connected to a phone that sits in one's pocket.
On Mac OS you can "speak" a text in almost every app, using built in voice (like the Siri voice or some older voices). All offline, and even from the terminal with "say".
I tried it a few months ago to narrate an epub in Apple Books and it was very broken in a weird way. It starts out decent but after a few pages, it starts slurring, skipping words, trailing off not finishing sentences and then goes silent.
(I've just tried it again without seeing that issue within a few pages)
> Siri voice or some older voices
You can choose "Enhanced" and "Premium" versions of voices which are larger and sound nice and modern to me. The "Serena Premium" voice I was using is over 200Mb and far better that this Show HN. It's very natural but kind of ruined by diabolical pronunciation of anything slightly non-standard which sadly seems to cover everything I read e.g. people/place names, technical/scientific terms or any neologisms in scifi/fantasy.
It's so wildly incomprehensible for e.g. Tibetan names in a mountaineering book, that you have to check the text. If the word being butchered is frequently repeated e.g. main character’s name, then it's just too painful to use.
Can't most people read faster than they can hear? Isn't this why phone menus are so awful?
> But there seems to be no good way to output spoken text without doing round-trips to a server and generate the audio there
As people have been pointing out, we've had mediocre TTS since the 80s. If it was a real benefit people would be using even the inadequate version.
The sample rate does more than change the quality.
Impressive, might use this for https://hnup.date
Love the idea, but the text it produces is way too flowery for my taste
"A new tool is stirring up excitement and debate in the programming community"
Just give me the facts without American style embellishments. You're not trying to sell me anything =)
Sorry, just saw this.
I absolutely agree, but it's really stubborn with the flowery language. I tried adding things like "DO NOT USE EMPTY PHRASES LIKE 'EVER-EVOLVING TECH LANDSCAPE'!!!!!" to the prompt, but it just can't resist.
I want to give the whole system an overhaul, maybe newer models are better at this. Or maybe a second LLM pass to de-flowerize (lol) the language.
Elixir folks. How would I use this with Elixir? I'm new to Elixir and could use this in about 15 days.
It looks like it's Python, so it might be possible to use via https://github.com/livebook-dev/pythonx ? But the parallel huggingface/bumblebee idea was also good, hadn't seen or thought of, that definitely works for a lot of other models, curious if you get working! Some chance I'll play with this myself in a few months, so feel free to report back here or DM me!
I just decided to try this quickly and hit some issues on my Mac FYI, it might work better on Linux but I hit a compilation issue with `curated-tokenizers`, possibly from a typo in setup.py or pyproject.toml in curated-tokenizers, spotted by AI: -Wno-sign-compare-Wno-strict-prototypes should be -Wno-sign-compare -Wno-strict-prototypes so could perhaps fix with a PR to curated-tokenizers or by forking it...
Might well be other issues behind that, and unclear if need any other dependencies that kitten doesn't rely on directly like torch or torchaudio? but... not 5 mins easy, but looks like issues might be able to be worked through...
For reference this is all I was trying basically:
to get the above error.It's not possible so far via Bumblebee, unfortunately[1].
[1] https://github.com/elixir-nx/bumblebee/issues/209
Cool, it looks like this model is pretty similar to StyleTTS 2? Would it be possible to confirm?
This look pretty awesome. I will definitely give it a try and let you know the results
Are there any speech to text (opposite direction) that I can load on mobile app?
Is the name a joke on "If the emperor had a tts device"? It's funny
We can integrate this into the browser directly!
-- browserOS.com
Is there a paper describing the architecture of the model?
Can you run it in reverse for speech recognition?
We will release an STT model as well.
no, but whisper has a 39M model: https://github.com/openai/whisper
Is there any way to get a .gguf version?
Very good model, thanks for the open source
thanks a lot, this model is just a preview checkpoint. the full release next week will be of much higher quality.
It's just so annoying and idiotic that there aren't a few samples on the home page. It didn't occur to you that it's the very first thing people would want to hear?
Can this work on intel npu unit?
It looks great
This is really great.
it would be great if there is typescript support in the future
Yup it runs on the web browser. https://clowerweb.github.io/kitten-tts-web-demo/
Just amazing
Kudos guys!
Thanks
♥
Now if only we could get LLMs to this sort of size! I don't know much about how TTS works under the hood, why is it so much easier?
I think one of the female voices belongs to Elizabeth Warren.
25M ? lol . the venv is 6.9G
"please join our DISCORD!"...
[dead]
[dupe]
[dead]
[flagged]
[dead]
[dead]
[dead]
[dead]