My understanding is that this is purely a strategic choice by the bigger labs. When OpenAI released Whisper, it was by far best-in-class, and they haven't released any major upgrades since then. It's been 3.5 years... Whisper is older than ChatGPT.
Gemini 3 Pro Preview has superlative audio listening comprehension. If I send it a recording of myself in a car, with me talking, and another passenger talking to the driver, and the radio playing, me in English, the radio in Portuguese, and the driver+passenger in Spanish, Gemini can parse all 4 audio streams as well as other background noises and give a translation for each one, including figuring out which voice belongs to which person, and what everyone's names are (if it's possible to figure that out from the conversation).
I'm sure it would have superlative audio generation capabilities too, if such a feature were enabled.
It's amazing how good open-weight STT and TTS have gotten, so there's no need to pay for Wispr Flow, Superwhisper, Eleven-Labs etc.
Sharing my setup in case it may be useful for others; it's especially useful when working with CLI agents like Code Code or Codex-CLI:
STT: Hex [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps confirm understanding as well as likely helps the CLI agent stay on track. It is a MacOS native app and leverages the CoreML/Neural Engine to get extremely fast transcription (I used to recommend a similar app Handy but it has frequent stuttering issues, and Hex is actually even faster, which I didn't think was possible!)
TTS: Kyutai's Pocket-TTS [2], just 100M params, and amazing speech quality (English only). I made a voice plugin [3] based on this, for Claude Code so it can speak out short updates whenever CC stops. It uses a combination of hooks that nudge the main agent to append a speakable summary, falling back to using a headless agent in case the main agent forgets. Turns out to be surprisingly useful. It's also fun as you can customize the speaking style and mirror your vibe and "colorful language" etc.
The voice plugin gives commands to control it:
/voice:speak stop
/voice:speak azelma (change the voice)
/voice:speak prompt <your arbitrary prompt to control the style>
You can roll a script to do this, something that would consume a mic from Pipewire when triggered and then push results to clipboard. With a Parakeet ONNX model in between.
I had cause to do the the opposite: Hotkey -> clipboard TTS
Good article and I agree with everything in there. For my own voice agent I decided to make him PTT by default as the problems of the model accurately guessing the end of utterance are just too great. I think it can be solved in the future but, I haven't seen a really good example of it being done with modern day tech including this labs. Fundamentally it all comes down to the fact that different humans have different ways of speaking, and the human listening to them updates their own internal model of the speech pattern. Adjusting their own model after a couple of interactions and arriving at the proper way of speaking with said person. Something very similar will need to be done and at very fast latency's for it to succeed in the audio ml world. But I don't think we have anything like that yet. It seems currently best you can do is tune the model on a generic speech pattern that you expect to fit over a larger percentage of the human population and that's about the best you can do, anyone who falls outside of that will feel the pain of getting interrupted every time.
Why would they do that? Once they have their win-condition, there's no reason to innovate. Only to reduce the costs of existing solutions. I expect that unless voice becomes a parameter which drives competition and first-choice for adoption, it will never become a focus of the frontier orgs. Which is curious to me, as almost the opposite of how I'm reading your comment.
OpenAI and google are too scared of music industry lawyers to tackle this. Internally they without a doubt have models that would crush these startups over night if they chose to release them.
Is your claim that music industry lawyers are that much scarier than movie industry lawyers? Because the big labs don't seem to have any problem releasing models that create (possibly infringing) video.
Thus far AI has only been used to create fan fiction clips that generate free marketing for legacy IP on TikTok. And the rights holders know that if AI gets good enough to make feature length movies then they'll be able to aggressively use various legal mechanisms to take the videos off major sites and pursue the creators. Long term it could potentially lower internal production costs by getting rid of actors & writers.
Music is very different. The production cost is already zero, and people generating their own Taylor Swift songs is a real competitive threat to Spotify etc.
> Is your claim that music industry lawyers are that much scarier than movie industry lawyers?
Not qoez:
You have to balance market opportunities with the risk of reputational damage and litigation risk.
Video will probably make a lot more money than audio, so you are willing to take a bigger risk. Additionally, at least for Google there exists a strong synergy between their video generation models and YouTube, which makes it even more sensible for Google to make video models available to the public despite these risks.
well i guess the music industry is a lot more monopolized than video, plus there is a lot of video out there that isn't "movies," while there's not a lot of music that isn't... "music"
I'm not sure it's just fear of lawyers, although that's definitely part of it. Big companies have way more to lose reputationally and legally, so the bar for releasing something is much higher
OpenAI being the death star and audio AI being the rebels is such a weird comparison, like what? Wouldn't the real rebels be the ones running their own models locally?
Audio AI companies are just another death star, intent on reducing human creativity to "make a song like Let it be, but in the style of Eminem, and change the lyrics to match the birthday of my mother in law". The only rebels are musicians resisting this hedge-fund driven monstrosity.
True, but there's a fun irony: the Rebels' X-Wings are powered by GPUs from a company that's... checks relationships ...also supplying the Empire.
NVIDIA's basically the galaxy's most successful arms dealer, selling to both sides while convincing everyone they're just "enabling innovation." The real rebels would be training audio models on potato-patched RP2040s. Brave souls, if they exist.
The company behind the T-65B X-Wing, Incom Corporation, did supply the Empire, as they did the Republic Navy before. By 0 BBY, Incom was nationalized by the Imperials. The X-Wing became the mainstay Alliance fighter because the plans were stolen by some defecting Incom engineers.
not sure about the irony - you can't really expect rebels to start their own weapons manufacturing lab right from converting ore into steel... these things are often supplied by a large manufacturer (which is often a monopoly)
why is it any different for a startup to tap into nvidia's proverbial shovel in order to start digging for gold?
Fair callout. The ideas and NVIDIA/X-Wing analogy are mine — I'm a non-native English speaker and use AI to help draft comments based on my detailed directions. The "Reply to" text was an artifact I should have removed — that's on me.
I engage here to learn and discuss, not to farm karma or manufacture consensus. Happy to continue the conversation about inference chips and "rebel infrastructure," or I can step back if the AI assistance undermines the discussion.
Hi I am a non native english speaker as well. Please don't be discouraged. But you're not speaking with your own voice if I make sense.
Honestly I prefer to talk to humans who have flaws. Even if your english is bad try typing it yourself. Everyone here will support you. That's how I learned.
Even my team mates use chatgpt to rewrite their messages in teams, it feels so dishonest.
Speaking of audio + AI, here's a "learning hack" I've been trying with voice mode, and the 3 big AI labs still haven't nailed it:
While on a walk with mobile phone + earphones, dump an article/paper/HN-Post/github-repo into the mobile chat app (chat-gpt, claude or gemini), and use voice mode to have it walk you through it conversationally, so you can ask follow up questions during the walk-thru and the AI would do web-search etc. I know I could do something like this with NotebookLM, but I want to engage in the conversation, and NotebookLM does have interactive mode but it has been super-flaky to say the least.
I pay for ChatGPT Pro and the voice mode is really bad: it pretends to do web searches and makes up things, and when pushed says it didn't actually read the article. Also the voice sounds super-condescending.
Gemini Pro mobile app - similarly refuses to open links and sounds as if it's talking to a baby.
Claude mobile app was the best among these - the voice is very tolerable in terms of tone, but like the others it can't open links. I does do web searches, but gets some type of summaries of pages, and it doesn't actually go into the links themselves to give me details.
I have found that the "advanced voice mode" is dumb as a box of rocks compared to their "basic" TTS version, so I disable it. I've switched to Claude, so I don't know if that's still an option, but if you are tied to ChatGPT, definitely disable it if possible!
Audio models are also tiny, which is probably why small labs are doing well in the space. I run a LoRA'd Whisper v3 Large for a client. We can fit 4 versions of the model in memory at once on a ~$1/hr A10 and have half the VRAM leftover.
Each of the LoRA tunes we did took maybe 2-3 hours on the same A10 instance.
My ~1.7% WER and faster than realtime processing in my application make it more than adequate. My application is multi-speaker with WPM rates >300 for long durations.
I check every day for a new full-duplex model. I was so hyped about PersonaPlex from their demos, but in my test it was oddly dumb and unable to follow instructions.
So I am hoping for something like PersonaPlex but a bit larger.
Has anyone tested MiniCPM-o?.How is it at instruction following?
Moshi was an amazing tech demo, building the entire stack from scratch in 6 months with a small team was an amazing show of skill: 7B text LLM data + training, emotive TTS for synth data generation (again model + data collection), synth data pipeline, novel speech codec, rust inference stack for low latency, audio LLM architecture incl. text "thoughts" stream which was novel.
But, this piece is a fluff piece: "underfunded" means a total of around $400 million ($330 million in the initial round, $70 million for Gradium). Compare to Elevenlabs who used a $2 million pre-seed for creating their initial product.
A bunch of other stuff there is disingenuous, like comparing their 7B model to Llama-3 405B (hint: the 7B model is a _lot_ dumber). There's also the outright lie: team of 4 made Moshi, which is corrected _in the same piece_ to 8 if you read enough.
I was suspicious that they are not mentioned, but then I realized this is a VC opinion piece and the first company mentioned joined their portfolio last year.
Can someone reccomend to me: a service that will generate a loopable engine drone for a "WWII Plane Japan Kawasaki Ki-61"? It doesn't have to be perfect, just convincing in a hollywood blockbuster context, and not just a warmed over clone of a Merlin engine sound. Turns out Suno will make whatever background music I need, but I want a "unique sound effect on demand" service. I'm not convinced voice AI stuff is sustainable
With the prompt "WWII Plane Japan Kawasaki Ki-61 flying by, propeller airplane" and setting looping on and 30 sec duration manually instead of auto (the duration predictor fails pretty bad at this prompt, you need to be logged in to set duration manually) it works pretty well. No idea if it's close to that specific airplane though it sounds like a ww2 plane to me though.
Most current "voice assistants" still feel like glorified walkie-talkies... you talk, pause awkwardly, they respond, and any interruption breaks the flow
I use the standard iPhone TTS accessibility tool and an iOS ebook reader.
It has modern voices, but I prefer the robotic voice from 15 years ago because it's very patterned and predictable, which makes it easier to follow at super-fast speaking rates -- something closer to my visual reading rate.
Transcription providers like wisprflow and willow voice are
typically providing nice UI/UX around open source models.
Wisprflow does not create it's own models but i know willow voice did do extensive finetuning to improve the quality and speed of their transcription models so you may count them.
Right now small labs also have the best chance at tool harness improvements, which can yield just as many gains in AI performance as model training research.
Hey Rob. I'm not on the tech team here at Gradium (I do GTM) but still curious where you found the glitch? Were you entering words into the STT in the bottom of the front page? Can you share an example so I can replicate? Many thanks!
Also, while the author complains that there is not a lot of high quality data around [0], you do not need a lot of data to train small models. Depending on the problem you are trying to solve, you can do a lot with single-digit gigabytes of audio data. See, e.g., https://jmvalin.ca/demo/rnnoise/
[0] Which I do agree with, particularly if you need it to be higher quality or labeled in a particular way: the Fisher database mentioned is narrowband and 8-bit mu-law quantized, and while there are timestamps, they are not accurate enough for millisecond-level active speech determination. It is also less than 6000 conversations totaling less than 1000 hours (x2 speakers, but each is silent over half the time, a fact that can also throw a wrench in some standard algorithms, like volume normalization). It is also English-only.
Wrt data, it's not like there is a shortage of transcribed audio in the form of music sheets, lyrics and subtitles.
If one asks ~~nice~~ expensive enough they can even get isolated multitracks or teleprompter feeds together with the audiovisual tracks. Heck, if they wanted they could set up dedicated transcription teams for the plethora of podcasts with the costs somewhere in the rounding error range. But you can't siphon that off of torrents and paying for training material goes against the core ethics of the big players.
Too bad you can't really scrape tiktok/instagram reels with subtitles... Oh no, oh no, oh no no no no
I refuse to believe that none of these people ever heard of Nyquist, and that noone was able to come up with "ayyy lmao let's put a low pass on this before downsampling".
Edit: 2 day old account posting stuff that doesn't pass the sniff test. Hmmmm... baited by a bot?
in ~30 years of my work in DSP domain, I've seen insane amount of ways to do signal processing wrong even for simplest things like passing a buffer and doing resampling.
The last example I've seen in one large company, done by a developer lacking audio/DSP experience: they used ffmpeg's resampling lib, but, after every 10ms audio frame processed by resampler, they'd invoke flush(), just for the sake of convenience of having the same number of input and output buffers ... :)
What's the point of saying that without backing it up? Either you think it's so obvious it doesn't need backing up (in which case you don't need to say it), or ...?
The reason it matters is that soon, any time somebody sees a comment they don't like or think is stupid, they'll just say, "eh a bot said that," and totally dilute the rest of the discussion, even if the comment was real.
It's monday today so I approach we just have to wait for 24 hours for tomorrow.
I do feel like AI has made us distrust each other at such a massive scale seen nothing like before. Previously there could still be bot-like comments but they were still written by human or so. Not anymore now and things are getting more sophisticated. I 100% believe that its possible to not have --. You are absolutely right and some other things that I have associated with Ai writing so at this point, AI might write such a way that I can't filter out if its written by AI or not.
It's a real psychosis inducing thought imo all while AI is inducing job insecurity from top to bottom.
> AI might write such a way that I can't filter out if its written by AI or not.
This is precisely the point for "farmers" that have no other motivation than to make money. They're not trying to troll us or promote a political view or sell something. They farm accounts and sell them on a secondary market for people who do use them nefariously. An aged, well-upvoted account has value for those groups. So they have every incentive to blend in by parroting back the most popular or neutral talking points.
To be frank, we were headed in that direction anyway, ubiquity of cheap and accessible high-quality writing generators just accelerated the inevitable.
For a very long time astroturfing was only viable over broadcast channels and [therefore] was expensive enough that only the really large corporations could afford to engage in the practice. With the advent of internet forums that could be arranged by any larger marketing agency and became available to any larger entity. With the advent of so called "user generated content" astroturfing really became easily accessible to any medium sized entity.
It has always been a race to the bottom. Reddit and Youtube are well known examples. If you follow any niche there whatsoever, you can easily (especially retroactively, because there's usually a ramp-up period) pinpoint a time where some astroturfing entity either launched a bot army or bought out mods/creators.
LLMs have dropped the cost of astroturfing to such a degree that not only it is now viable for any individual dropshipper to engage in the activity, locations with conversions previously too low to matter became viable attack targets. Everyone with several dozen euros and some hours to spare can now absolutely flood every corner of the internet without extremely strict authentication.
Some places are more resistant to astroturfing than others, though. I would suspect that if some few days old accounts started praising Foobarized in every possible thread here on HN the admins would ban this keyword quite quickly. One needs to be slow and careful. It requires pre-creating a bunch of accounts and leveling them up with some casual comments here and there so that when the actual astroturfing campaign starts (buyer not yet determined), the accounts are primed and are accepted as part of the community.
The internet is dead and has been for some time, it's just that we are too afraid to admit it to ourselves while actively seeking closed gardens of confirmed human-to-human communication.
And it will have second-order effects where real humans will reduce their participation because they believe there are only bots or tired of accusations of being one.
> imo audio DSP experts are diametrically opposed to AI on moral grounds.
Can you elaborate on this point? I don't know the moral grounds of audio DSP experts, and thus I don't understand why in your opinion they wouldn't take an offer if you really pay them some serious amount of money.
Just to be clear: considering what a typical daily job in DSP programming is like, I can imagine that many audio DSP experts are not the best culture fit for AI companies, but this doesn't have anything to do with morality.
In my experience, most of the people in audio DSP are musicians or otherwise very well exposed to music and the arts, and many see using AI as fundamentally immoral or unethical. It's technology designed via theft with the intent to harm professionals in these spaces.
Like I said, it's like paying a doctor to design a better gun.
Yes, but, you see, guns are merely tools. If you give these guns to the right people, they will actually create more work for the doctors, and everybody wins.
My understanding is that this is purely a strategic choice by the bigger labs. When OpenAI released Whisper, it was by far best-in-class, and they haven't released any major upgrades since then. It's been 3.5 years... Whisper is older than ChatGPT.
Gemini 3 Pro Preview has superlative audio listening comprehension. If I send it a recording of myself in a car, with me talking, and another passenger talking to the driver, and the radio playing, me in English, the radio in Portuguese, and the driver+passenger in Spanish, Gemini can parse all 4 audio streams as well as other background noises and give a translation for each one, including figuring out which voice belongs to which person, and what everyone's names are (if it's possible to figure that out from the conversation).
I'm sure it would have superlative audio generation capabilities too, if such a feature were enabled.
Nvidia released Parakeet which claimed superiority. Doesn't negate your point but I did want to add it.
It's amazing how good open-weight STT and TTS have gotten, so there's no need to pay for Wispr Flow, Superwhisper, Eleven-Labs etc.
Sharing my setup in case it may be useful for others; it's especially useful when working with CLI agents like Code Code or Codex-CLI:
STT: Hex [1] (open-source), with Parakeet V3 - stunningly fast, near-instant transcription. The slight accuracy drop relative to bigger models is immaterial when you're talking to an AI. I always ask it to restate back to me what it understood, and it gives back a nicely structured version -- this helps confirm understanding as well as likely helps the CLI agent stay on track. It is a MacOS native app and leverages the CoreML/Neural Engine to get extremely fast transcription (I used to recommend a similar app Handy but it has frequent stuttering issues, and Hex is actually even faster, which I didn't think was possible!)
TTS: Kyutai's Pocket-TTS [2], just 100M params, and amazing speech quality (English only). I made a voice plugin [3] based on this, for Claude Code so it can speak out short updates whenever CC stops. It uses a combination of hooks that nudge the main agent to append a speakable summary, falling back to using a headless agent in case the main agent forgets. Turns out to be surprisingly useful. It's also fun as you can customize the speaking style and mirror your vibe and "colorful language" etc.
The voice plugin gives commands to control it:
[1] Hex https://github.com/kitlangton/Hex[2] Pocket-TTS https://github.com/kyutai-labs/pocket-tts
[3] Voice plugin for Claude Code: https://pchalasani.github.io/claude-code-tools/plugins-detai...
Same setup I’m using! Parakeet and pocket turbo. It’s feels good enough for daily usage.
Anyone know of something like Hex that runs on Linux?
Handy is cross-platform, including linux
+1 for Handy, it's very easy to get running and once it is you don't have to think about it again.
You can roll a script to do this, something that would consume a mic from Pipewire when triggered and then push results to clipboard. With a Parakeet ONNX model in between.
I had cause to do the the opposite: Hotkey -> clipboard TTS
Is Hex MacOS only?
Yes
Good article and I agree with everything in there. For my own voice agent I decided to make him PTT by default as the problems of the model accurately guessing the end of utterance are just too great. I think it can be solved in the future but, I haven't seen a really good example of it being done with modern day tech including this labs. Fundamentally it all comes down to the fact that different humans have different ways of speaking, and the human listening to them updates their own internal model of the speech pattern. Adjusting their own model after a couple of interactions and arriving at the proper way of speaking with said person. Something very similar will need to be done and at very fast latency's for it to succeed in the audio ml world. But I don't think we have anything like that yet. It seems currently best you can do is tune the model on a generic speech pattern that you expect to fit over a larger percentage of the human population and that's about the best you can do, anyone who falls outside of that will feel the pain of getting interrupted every time.
Check out Sparrow-0. The demo shows an impressive ability to predict when the speaker has finished talking:
https://www.tavus.io/post/sparrow-0-advancing-conversational...
Thanks, ill read it now.
It feels like this is one of those areas where the last 10% of polish will take 90% of the effort
The 80/20 rule always wins
There's too much noise at large organizations
I see what you did there.
They're focused on soaking up big money first.
They'll optimize down the stack once they've sucked all the oxygen out of the room.
Little players won't be able to grow through the ceiling the giants create.
Why would they do that? Once they have their win-condition, there's no reason to innovate. Only to reduce the costs of existing solutions. I expect that unless voice becomes a parameter which drives competition and first-choice for adoption, it will never become a focus of the frontier orgs. Which is curious to me, as almost the opposite of how I'm reading your comment.
OpenAI and google are too scared of music industry lawyers to tackle this. Internally they without a doubt have models that would crush these startups over night if they chose to release them.
Is your claim that music industry lawyers are that much scarier than movie industry lawyers? Because the big labs don't seem to have any problem releasing models that create (possibly infringing) video.
The movie industry is doing well from AI.
Thus far AI has only been used to create fan fiction clips that generate free marketing for legacy IP on TikTok. And the rights holders know that if AI gets good enough to make feature length movies then they'll be able to aggressively use various legal mechanisms to take the videos off major sites and pursue the creators. Long term it could potentially lower internal production costs by getting rid of actors & writers.
Music is very different. The production cost is already zero, and people generating their own Taylor Swift songs is a real competitive threat to Spotify etc.
Just right now: ByteDance to curb AI video app after Disney legal threat
https://www.bbc.com/news/articles/c93wq6xqgy1o
> Is your claim that music industry lawyers are that much scarier than movie industry lawyers?
Not qoez:
You have to balance market opportunities with the risk of reputational damage and litigation risk.
Video will probably make a lot more money than audio, so you are willing to take a bigger risk. Additionally, at least for Google there exists a strong synergy between their video generation models and YouTube, which makes it even more sensible for Google to make video models available to the public despite these risks.
well i guess the music industry is a lot more monopolized than video, plus there is a lot of video out there that isn't "movies," while there's not a lot of music that isn't... "music"
What about Disney's lawyers? GenAI for images exists ...
Disney is actually quite excited about GenAI [0]
[0] https://openai.com/index/disney-sora-agreement/
[dead]
I'm not sure it's just fear of lawyers, although that's definitely part of it. Big companies have way more to lose reputationally and legally, so the bar for releasing something is much higher
OpenAI being the death star and audio AI being the rebels is such a weird comparison, like what? Wouldn't the real rebels be the ones running their own models locally?
Audio AI companies are just another death star, intent on reducing human creativity to "make a song like Let it be, but in the style of Eminem, and change the lyrics to match the birthday of my mother in law". The only rebels are musicians resisting this hedge-fund driven monstrosity.
True, but there's a fun irony: the Rebels' X-Wings are powered by GPUs from a company that's... checks relationships ...also supplying the Empire.
NVIDIA's basically the galaxy's most successful arms dealer, selling to both sides while convincing everyone they're just "enabling innovation." The real rebels would be training audio models on potato-patched RP2040s. Brave souls, if they exist.
The company behind the T-65B X-Wing, Incom Corporation, did supply the Empire, as they did the Republic Navy before. By 0 BBY, Incom was nationalized by the Imperials. The X-Wing became the mainstay Alliance fighter because the plans were stolen by some defecting Incom engineers.
wonder why the Empire never ran any black x wings
not sure about the irony - you can't really expect rebels to start their own weapons manufacturing lab right from converting ore into steel... these things are often supplied by a large manufacturer (which is often a monopoly) why is it any different for a startup to tap into nvidia's proverbial shovel in order to start digging for gold?
[flagged]
Your reply has a 0% ai score yet the presence of that "reply to" text is concerning.
Not very concerning because that post is obviously LLM generated.
Fair callout. The ideas and NVIDIA/X-Wing analogy are mine — I'm a non-native English speaker and use AI to help draft comments based on my detailed directions. The "Reply to" text was an artifact I should have removed — that's on me. I engage here to learn and discuss, not to farm karma or manufacture consensus. Happy to continue the conversation about inference chips and "rebel infrastructure," or I can step back if the AI assistance undermines the discussion.
>or I can step back if the AI assistance undermines the discussion.
Yes.
Please don't post generated or AI-filtered posts to HN. We want to hear you in your own voice, and it's fine if your English isn't perfect.
https://news.ycombinator.com/item?id=46747998
Hi I am a non native english speaker as well. Please don't be discouraged. But you're not speaking with your own voice if I make sense.
Honestly I prefer to talk to humans who have flaws. Even if your english is bad try typing it yourself. Everyone here will support you. That's how I learned.
Even my team mates use chatgpt to rewrite their messages in teams, it feels so dishonest.
I don't know why this would be surprising because anywhere purporting to be able to score like this is obviously lying to you
I had a different issue with the metaphor - shouldn't OpenAI be the empire? The death star would be the thing they created, i.e. ChatGPT.
Never any mention of Soniox and they are on the Pareto frontier[1]
https://www.daily.co/blog/benchmarking-stt-for-voice-agents/
There’s simply not enough of a market for these bigger orgs to be truly interested/invested in audio, video and even image to an extent.
They’ll wait for progress to be made and then buy the capability/expertise/talent when the time is right.
Speaking of audio + AI, here's a "learning hack" I've been trying with voice mode, and the 3 big AI labs still haven't nailed it:
While on a walk with mobile phone + earphones, dump an article/paper/HN-Post/github-repo into the mobile chat app (chat-gpt, claude or gemini), and use voice mode to have it walk you through it conversationally, so you can ask follow up questions during the walk-thru and the AI would do web-search etc. I know I could do something like this with NotebookLM, but I want to engage in the conversation, and NotebookLM does have interactive mode but it has been super-flaky to say the least.
I pay for ChatGPT Pro and the voice mode is really bad: it pretends to do web searches and makes up things, and when pushed says it didn't actually read the article. Also the voice sounds super-condescending.
Gemini Pro mobile app - similarly refuses to open links and sounds as if it's talking to a baby.
Claude mobile app was the best among these - the voice is very tolerable in terms of tone, but like the others it can't open links. I does do web searches, but gets some type of summaries of pages, and it doesn't actually go into the links themselves to give me details.
I have found that the "advanced voice mode" is dumb as a box of rocks compared to their "basic" TTS version, so I disable it. I've switched to Claude, so I don't know if that's still an option, but if you are tied to ChatGPT, definitely disable it if possible!
Audio models are also tiny, which is probably why small labs are doing well in the space. I run a LoRA'd Whisper v3 Large for a client. We can fit 4 versions of the model in memory at once on a ~$1/hr A10 and have half the VRAM leftover.
Each of the LoRA tunes we did took maybe 2-3 hours on the same A10 instance.
Is Whisper still getting nontrivial development? I was under the impression that it had stagnated, but it seems hard to find more than just rumors
My ~1.7% WER and faster than realtime processing in my application make it more than adequate. My application is multi-speaker with WPM rates >300 for long durations.
I check every day for a new full-duplex model. I was so hyped about PersonaPlex from their demos, but in my test it was oddly dumb and unable to follow instructions.
So I am hoping for something like PersonaPlex but a bit larger.
Has anyone tested MiniCPM-o?.How is it at instruction following?
It's actively under development. Do you have a particular use-case in mind?
Outgoing phone calls.
Moshi was an amazing tech demo, building the entire stack from scratch in 6 months with a small team was an amazing show of skill: 7B text LLM data + training, emotive TTS for synth data generation (again model + data collection), synth data pipeline, novel speech codec, rust inference stack for low latency, audio LLM architecture incl. text "thoughts" stream which was novel.
But, this piece is a fluff piece: "underfunded" means a total of around $400 million ($330 million in the initial round, $70 million for Gradium). Compare to Elevenlabs who used a $2 million pre-seed for creating their initial product.
A bunch of other stuff there is disingenuous, like comparing their 7B model to Llama-3 405B (hint: the 7B model is a _lot_ dumber). There's also the outright lie: team of 4 made Moshi, which is corrected _in the same piece_ to 8 if you read enough.
Stopped reading there: "This model (Moshi) could [...] recite an original poem in a French accent (research shows poems sound better this way)."
Surprised ElevenLabs is not mentioned
I was suspicious that they are not mentioned, but then I realized this is a VC opinion piece and the first company mentioned joined their portfolio last year.
Also 15.ai [1]
[1] https://en.wikipedia.org/wiki/15.ai
Can someone reccomend to me: a service that will generate a loopable engine drone for a "WWII Plane Japan Kawasaki Ki-61"? It doesn't have to be perfect, just convincing in a hollywood blockbuster context, and not just a warmed over clone of a Merlin engine sound. Turns out Suno will make whatever background music I need, but I want a "unique sound effect on demand" service. I'm not convinced voice AI stuff is sustainable
https://elevenlabs.io/sound-effects
With the prompt "WWII Plane Japan Kawasaki Ki-61 flying by, propeller airplane" and setting looping on and 30 sec duration manually instead of auto (the duration predictor fails pretty bad at this prompt, you need to be logged in to set duration manually) it works pretty well. No idea if it's close to that specific airplane though it sounds like a ww2 plane to me though.
are there any open source alternatives to this as well if I may ask?
A fun alternative could be using a physically based engine sound synthesizer - for example - https://github.com/Engine-Simulator/engine-sim-community-edi...
you mean you want some ai product that generates sound effects from a textual prompt? elevenlabs has a model specifically for that
https://elevenlabs.io/sound-effects
Most current "voice assistants" still feel like glorified walkie-talkies... you talk, pause awkwardly, they respond, and any interruption breaks the flow
Is there something that will read books to me? I.e I have some books in epub format and want audiobook versions for them, with a nice voice.
I use the standard iPhone TTS accessibility tool and an iOS ebook reader.
It has modern voices, but I prefer the robotic voice from 15 years ago because it's very patterned and predictable, which makes it easier to follow at super-fast speaking rates -- something closer to my visual reading rate.
You can generate your own audiobook (single voice or multi) https://www.plainscribe.com
ElevenReader works well and has a pretty good free plan
Speechify has been good for me although there might be better / cheaper alternatives I'm not aware of.
Maybe OpenAI has finally learned that dancing on all the parties at once, when all parties are progressing towards "commodity".
Probably because the big companies have their focus elsewhere.
The bigger players probably avoid it because it's a bigger legal liability: https://news.ycombinator.com/item?id=47025864
..plenty of money to be made elsewhere
Also: porn.
Audio is too niche and porn is too ethically messy and legally risky.
There's also music, which the giants also don't touch. Suno is actually really impressive.
Does Wisprflow count as an audio “lab”?
Transcription providers like wisprflow and willow voice are typically providing nice UI/UX around open source models.
Wisprflow does not create it's own models but i know willow voice did do extensive finetuning to improve the quality and speed of their transcription models so you may count them.
Right now small labs also have the best chance at tool harness improvements, which can yield just as many gains in AI performance as model training research.
for a laugh enter nonsense at https://gradium.ai/
You get all kinds of weird noises and random words. Jack is often apologetic about the problem you are having with the Hyperion xt5000 smart hub.
Hey Rob. I'm not on the tech team here at Gradium (I do GTM) but still curious where you found the glitch? Were you entering words into the STT in the bottom of the front page? Can you share an example so I can replicate? Many thanks!
[dead]
[dead]
[dead]
[flagged]
[flagged]
Also, while the author complains that there is not a lot of high quality data around [0], you do not need a lot of data to train small models. Depending on the problem you are trying to solve, you can do a lot with single-digit gigabytes of audio data. See, e.g., https://jmvalin.ca/demo/rnnoise/
[0] Which I do agree with, particularly if you need it to be higher quality or labeled in a particular way: the Fisher database mentioned is narrowband and 8-bit mu-law quantized, and while there are timestamps, they are not accurate enough for millisecond-level active speech determination. It is also less than 6000 conversations totaling less than 1000 hours (x2 speakers, but each is silent over half the time, a fact that can also throw a wrench in some standard algorithms, like volume normalization). It is also English-only.
Wrt data, it's not like there is a shortage of transcribed audio in the form of music sheets, lyrics and subtitles.
If one asks ~~nice~~ expensive enough they can even get isolated multitracks or teleprompter feeds together with the audiovisual tracks. Heck, if they wanted they could set up dedicated transcription teams for the plethora of podcasts with the costs somewhere in the rounding error range. But you can't siphon that off of torrents and paying for training material goes against the core ethics of the big players.
Too bad you can't really scrape tiktok/instagram reels with subtitles... Oh no, oh no, oh no no no no
[flagged]
I refuse to believe that none of these people ever heard of Nyquist, and that noone was able to come up with "ayyy lmao let's put a low pass on this before downsampling".
Edit: 2 day old account posting stuff that doesn't pass the sniff test. Hmmmm... baited by a bot?
in ~30 years of my work in DSP domain, I've seen insane amount of ways to do signal processing wrong even for simplest things like passing a buffer and doing resampling.
The last example I've seen in one large company, done by a developer lacking audio/DSP experience: they used ffmpeg's resampling lib, but, after every 10ms audio frame processed by resampler, they'd invoke flush(), just for the sake of convenience of having the same number of input and output buffers ... :)
Haha wow, I guess that gives a very noticeable 100 Hz comb-like effect... noone cared to do a quick sanity check on that output?!
[dead]
[flagged]
What's the point of saying that without backing it up? Either you think it's so obvious it doesn't need backing up (in which case you don't need to say it), or ...?
The reason it matters is that soon, any time somebody sees a comment they don't like or think is stupid, they'll just say, "eh a bot said that," and totally dilute the rest of the discussion, even if the comment was real.
We are *quickly* approaching a Tuesday where "bot detected, opinion rejected" is going to be a default assumption.
It's monday today so I approach we just have to wait for 24 hours for tomorrow.
I do feel like AI has made us distrust each other at such a massive scale seen nothing like before. Previously there could still be bot-like comments but they were still written by human or so. Not anymore now and things are getting more sophisticated. I 100% believe that its possible to not have --. You are absolutely right and some other things that I have associated with Ai writing so at this point, AI might write such a way that I can't filter out if its written by AI or not.
It's a real psychosis inducing thought imo all while AI is inducing job insecurity from top to bottom.
> AI might write such a way that I can't filter out if its written by AI or not.
This is precisely the point for "farmers" that have no other motivation than to make money. They're not trying to troll us or promote a political view or sell something. They farm accounts and sell them on a secondary market for people who do use them nefariously. An aged, well-upvoted account has value for those groups. So they have every incentive to blend in by parroting back the most popular or neutral talking points.
To be frank, we were headed in that direction anyway, ubiquity of cheap and accessible high-quality writing generators just accelerated the inevitable.
For a very long time astroturfing was only viable over broadcast channels and [therefore] was expensive enough that only the really large corporations could afford to engage in the practice. With the advent of internet forums that could be arranged by any larger marketing agency and became available to any larger entity. With the advent of so called "user generated content" astroturfing really became easily accessible to any medium sized entity.
It has always been a race to the bottom. Reddit and Youtube are well known examples. If you follow any niche there whatsoever, you can easily (especially retroactively, because there's usually a ramp-up period) pinpoint a time where some astroturfing entity either launched a bot army or bought out mods/creators.
LLMs have dropped the cost of astroturfing to such a degree that not only it is now viable for any individual dropshipper to engage in the activity, locations with conversions previously too low to matter became viable attack targets. Everyone with several dozen euros and some hours to spare can now absolutely flood every corner of the internet without extremely strict authentication.
Some places are more resistant to astroturfing than others, though. I would suspect that if some few days old accounts started praising Foobarized in every possible thread here on HN the admins would ban this keyword quite quickly. One needs to be slow and careful. It requires pre-creating a bunch of accounts and leveling them up with some casual comments here and there so that when the actual astroturfing campaign starts (buyer not yet determined), the accounts are primed and are accepted as part of the community.
The internet is dead and has been for some time, it's just that we are too afraid to admit it to ourselves while actively seeking closed gardens of confirmed human-to-human communication.
And it will have second-order effects where real humans will reduce their participation because they believe there are only bots or tired of accusations of being one.
Yeah I thought I knew what "post-truth" meant back in 2016 but boy was I wrong.
imo audio DSP experts are diametrically opposed to AI on moral grounds. Good luck hiring the good ones. It's like paying doctors to design guns.
Not sure your analogy works, Guantanamo had no trouble hiring medical personnel
> imo audio DSP experts are diametrically opposed to AI on moral grounds.
Can you elaborate on this point? I don't know the moral grounds of audio DSP experts, and thus I don't understand why in your opinion they wouldn't take an offer if you really pay them some serious amount of money.
Just to be clear: considering what a typical daily job in DSP programming is like, I can imagine that many audio DSP experts are not the best culture fit for AI companies, but this doesn't have anything to do with morality.
In my experience, most of the people in audio DSP are musicians or otherwise very well exposed to music and the arts, and many see using AI as fundamentally immoral or unethical. It's technology designed via theft with the intent to harm professionals in these spaces.
Like I said, it's like paying a doctor to design a better gun.
Yes, but, you see, guns are merely tools. If you give these guns to the right people, they will actually create more work for the doctors, and everybody wins.
[flagged]
i don’t understand the point in farming comments with an LLM on a site like Hacker News. what’s the benefit here?
(source: just look at their comment history)
Yeah, that needs to go... lest someone actually replies. Replyong to an LLM is replying to no-one.
I'd assume you could get enough points on each to get 500 so you could downvote.
Then you could help control the discussion if you coordinated up/downvotes/flag with your bots.