Super cool. Love seeing these writeups of hobbyists getting their hands dirty, breaking things, and then coming out on the other side of it with something interesting.
Yea, it has been a little shocking to me that the rising narratives around "AI agents everywhere" and "enable the web for AI agents" requires what we've all been wanting for awhile on the web (openness and interoperability) but that the same big players in tech have been clearly against for a long time. Like the fact that Google recently released that Google Workspace CLI (https://github.com/googleworkspace/cli) is a perfect example.
They could've released something like that years ago (the discovery service it's built on has existed for over a decade) but creating a simple, accessible, unified CLI for general integration apparently wasn't worth it until agents became the hot thing.
I wonder when / if there will be a rug pull on all of this. Because I really don't see what the long-term incentives are for incumbent tech platforms to make it easy for automated systems to essentially pull users away from the actual platform. I guess they're focused on the short term incentives. And once they decide the party's over, promising upstarts and competition can get absorbed and it'll be business as usual. Idk, we'll see.
I've been working solely on voice agents for the past couple years (and have worked at one of the frontier voice AI companies).
The cascading model (STT -> LLM -> TTS), is unlikely to go away anytime soon for a whole lot of reasons. A big one is observability. The people paying for voice agents are enterprises. Enterprises care about reliability and liability. The cascading model approach is much more amenable to specialization (rather than raw flexibility / generality) and auditability.
Organizations in regulated industries (e.g. healthcare, finance, education) need to be able to see what a voice agent "heard" before it tries to "act" on transcribed text, and same goes for seeing what LLM output text is going to be "said" before it's actually synthesized and played back.
Speech-to-Speech (end-to-end) models definitely have a place for more "narrative" use cases (think interviewing, conducting surveys / polls, etc.).
But from my experience from working with clients, they are clamoring for systems and orchestration that actually use some good ol' fashioned engineering and that don't solely rely on the latest-and-greatest SoTA ML models.
OpenAI's Semantic mode is looking at the semantic meaning of the transcribed text to make an educated guess about where the user's end of utterance is.
According to Deepgram, Flux's end-of-turn detection is not just a semantic VAD (which inherently is a separate model from the STT model that's doing the transcribing). Deepgram describes Flux as:
> the same model that produces transcripts is also responsible for modeling conversational flow and turn detection.
[...]
> With complete semantic, acoustic, and full-turn context in a fused model, Flux is able to very accurately detect turn ends and avoid the premature interruptions common with traditional approaches.
So according to them, end-of-turn detection isn't just based on semantic content of the transcript (which makes sense given the latency), but rather the the characteristics of the actual audio waveform itself as well.
Which Pipecat (open source voice AI orchestration platform) actually does as well seemingly with their smart-turn native turn detection model as well: https://github.com/pipecat-ai/smart-turn (minus the built-in transcription)
"Good" is subjective. But yes, all wealth creation requires working with other people. No one is an island. And most people are increasingly disturbed by the types of decisions required to amass more wealth than sovereign nations.
Yea, it's puzzling to me that this isn't asked of folks like Altman and Amodei in every interview. Maybe it's because Altman would just start shilling his eye scanning orb and start repeating "WORLD COIN" ad nauseum. Either way, they should be getting pressed on this by all media.
It's not puzzling. Journalism was murdered because it asked Nixon too many questions. So now unless you softball interviews, you just don't get to interview anyone, so the only news orgs with content to monetize are the ones just printing Press Releases and being a backboard for "interviews".
It sure is fun how the party who screams about "personal responsibility" seems to get very upset if you ask a responsible person to explain themselves and their actions.
reply