I think one of the hardest things about voice AI is being able to gracefully modulate between styles of input/output delay. In actual technical conversations sometimes it is appropriate to patiently wait for someone to ask a very difficult question with delays in speech, and sometimes it is appropriate to interrupt regularly and ask brief pointed questions that require one-word answers, and everywhere in between. I'm really looking forward to having that kind of interchange with an LLM.
Maybe a variable-pressure PTT button for the user would be effective. Push harder or less hard depending on whether or not a word is on the tip of your tongue. The AI could appropriately interrupt or not.
you are 100% correct! Here we just check if there is a pause of 200ms (you can change that in the code). I haven't seen any models that can detect if the user has finish speaking base on the question or social cues. You also don't want to make the delay so long since then it sound unnatural
Interesting demo. I'd argue that a good tutor will ask more questions than answer. The tutor has to gauge the learner's understanding and adapt instruction. Properly formulating a question can be tricky if you're confused.
Thats a interesting thought. We could get it to periodically ask a question(s) once a concept has been explained in the demo. There have actually been a few studies that students learn best via discussions
Ending most responses with something like, "Did that make sense?" or "Should I clarify anything I just talked about?"
Also, I asked some math questions, and the AI started talking about and giving equations. For me, at least, this was impossible to understand. I know part of the point of this is to make it conversational, but I would think having a transcript displayed somewhere would help a lot.
For bonus points you could do this with the voice input. Only show the transcription when it's relevant or if the user asks.
1. The first instruction starts "When the bot has connected...", and
2. After waiting a couple of minutes, 'BOT STATUS' still says 'Connecting'.
I am running this in a 'Guest' profile in Chrome, that has no extensions installed.
These are the errors/warnings I see in the dev console:
Unrecognized feature: 'web-share'.
ta$2 @ index-CElq2hz-.js:33
index-CElq2hz-.js:2067 Loading VAD
index-CElq2hz-.js:2043 env.wasm.numThreads is set to 4, but this will not work unless you enable crossOriginIsolated mode. See https://web.dev/cross-origin-isolation-guide/ for more info.
ug @ index-CElq2hz-.js:2043
10Third-party cookie will be blocked in future Chrome versions as part of Privacy Sandbox.
If I wanted to make something half as good that I can run on a server somewhere, where would you suggest I start with STT. I've been looking for some online (i.e. realtime services) but found nothing great. Can you point me to a low effort/cost solution I can use myself? Are there good tutorials or example projects that have this working but not quite as well as you?
Can you please elaborate on what you would like help with?
I linked all the code in the post so you can experiment with it yourself and even extend it. Also we are making use of the Pipecat framework (https://github.com/pipecat-ai/pipecat) which means you can swap in/out any STT, LLM or TTS model you would like to use:)
Yeah I got lazy with the "continue with lecture part" by doing fuzzy matching on the text. If you have an accent or its noisy its not that great. The more robust way to do it would be function calling
We partnered with them and so its locally running their STT model on the container. You will see in the code we have a image reference which includes the Deepgram model
That is only for the data processing step that I run locally on my Mac to embed all his Youtube videos and upload to the VectorDB. Im not running Deepgram locally there
I think one of the hardest things about voice AI is being able to gracefully modulate between styles of input/output delay. In actual technical conversations sometimes it is appropriate to patiently wait for someone to ask a very difficult question with delays in speech, and sometimes it is appropriate to interrupt regularly and ask brief pointed questions that require one-word answers, and everywhere in between. I'm really looking forward to having that kind of interchange with an LLM.
Maybe a variable-pressure PTT button for the user would be effective. Push harder or less hard depending on whether or not a word is on the tip of your tongue. The AI could appropriately interrupt or not.
I wonder if you need to account for body language to get that figured out. A technical conversation on the phone is usually not as fluid as in-person.
you are 100% correct! Here we just check if there is a pause of 200ms (you can change that in the code). I haven't seen any models that can detect if the user has finish speaking base on the question or social cues. You also don't want to make the delay so long since then it sound unnatural
Interesting demo. I'd argue that a good tutor will ask more questions than answer. The tutor has to gauge the learner's understanding and adapt instruction. Properly formulating a question can be tricky if you're confused.
Thats a interesting thought. We could get it to periodically ask a question(s) once a concept has been explained in the demo. There have actually been a few studies that students learn best via discussions
Ending most responses with something like, "Did that make sense?" or "Should I clarify anything I just talked about?"
Also, I asked some math questions, and the AI started talking about and giving equations. For me, at least, this was impossible to understand. I know part of the point of this is to make it conversational, but I would think having a transcript displayed somewhere would help a lot.
For bonus points you could do this with the voice input. Only show the transcription when it's relevant or if the user asks.
This is a good suggestion! I will at it to the list of things that the community can do to extend
I tried to use it, but:
1. The first instruction starts "When the bot has connected...", and
2. After waiting a couple of minutes, 'BOT STATUS' still says 'Connecting'.
I am running this in a 'Guest' profile in Chrome, that has no extensions installed.
These are the errors/warnings I see in the dev console:
Unrecognized feature: 'web-share'. ta$2 @ index-CElq2hz-.js:33 index-CElq2hz-.js:2067 Loading VAD index-CElq2hz-.js:2043 env.wasm.numThreads is set to 4, but this will not work unless you enable crossOriginIsolated mode. See https://web.dev/cross-origin-isolation-guide/ for more info. ug @ index-CElq2hz-.js:2043 10Third-party cookie will be blocked in future Chrome versions as part of Privacy Sandbox.
Sorry its getting smashed right now - increasing capacity
If I wanted to make something half as good that I can run on a server somewhere, where would you suggest I start with STT. I've been looking for some online (i.e. realtime services) but found nothing great. Can you point me to a low effort/cost solution I can use myself? Are there good tutorials or example projects that have this working but not quite as well as you?
Can you please elaborate on what you would like help with?
I linked all the code in the post so you can experiment with it yourself and even extend it. Also we are making use of the Pipecat framework (https://github.com/pipecat-ai/pipecat) which means you can swap in/out any STT, LLM or TTS model you would like to use:)
Cool idea! Been experimenting with similar stuff, still trying to solve for detecting when a question ends.
BTW at one point it got confused and said that it doesn't have the ability to continue a lecture. I even have a recording in case you need it.
Awesome! Let me know if you figure it out!
Yeah I got lazy with the "continue with lecture part" by doing fuzzy matching on the text. If you have an accent or its noisy its not that great. The more robust way to do it would be function calling
Super cool idea. I think it might be getting the hug of death right now because even once I got in I never seem to get a connected bot.
Sorry it was indeed receiving the hug of death - but it should be alleviated now
This is awesome! Nice work to you and your team!
We also want to implement similar functions. You did a great job and inspired us.
Im glad! Keep pushing!
How is using a Deepgram model "local"?
We partnered with them and so its locally running their STT model on the container. You will see in the code we have a image reference which includes the Deepgram model
Not sure why it requires a Deepgram API key if it is running locally?
> DeepgramClient(os.getenv("DEEPGRAM_API_KEY"))
That is only for the data processing step that I run locally on my Mac to embed all his Youtube videos and upload to the VectorDB. Im not running Deepgram locally there
That is awesome!