Voice Agents Can Just Do Things

This is an updated version of a talk I gave at the AI Engineer World’s Fair.

Part of my job on the developer experience team at OpenAI is talking to developers to understand what and how they’re building with our models, whether that’s text, image, or audio.

And lately, I’ve been thinking about a misconception I’ve seen, or maybe it’s just a misunderstanding: the idea that voice agents have to talk back.

At some level, that might sound absurd.

“It’s a voice agent. What do you mean it’s not supposed to talk?”

But I’ve increasingly come to believe that speech is not the only way a voice model can respond. Models are becoming intelligent and capable enough that they’re starting to open up new modes of design. And there are three modes in particular that I see emerging: speech-to-speech, speech-to-action, and event-to-speech1.

They aren’t meant to be mutually exclusive - the best products combine all of these modes.

Subscribe

Everybody knows it and (hopefully) loves it. The user talks, and then the model talks back.

There are a few examples I can give for this use case, but hopefully it’s fairly straightforward:

• Live practice and coaching: especially when it comes to language learning, the ability to hear emphasis or emotion and give people that feedback is really powerful.
• Concierge experiences: which is another way to say “actually good customer support.” As we add richness and depth to voice models, as has been happening in recent months and recent years, we can build something much more enjoyable to use than talking your way through a phone tree.
• Live translation: models have gotten good enough and fast enough that we can dynamically translate content on the fly with little to no latency.

These are useful experiences, and there’s still a lot of room to make them better. But they all start from the same assumption of a two-participant back and forth. What happens when we loosen that assumption?

The second category is speech-to-action: the user talks, and the model uses tools. I truly think this is one of the most underexplored areas we have.



There’s a broad spectrum of examples, but below are three categories I find particularly interesting.

The first is form filling. So much of the internet is just filling out forms. Today, there’s no reason why you shouldn’t be able to just talk. I would love it if, instead of spending an hour filling out a government document, I could talk for five minutes and have it ready to submit.

That would be a vastly superior experience to having to type in every single name and address I’ve had in the last five years, and all of my previous identities. And form filling, though it may seem boring, already underpins a significant share of the internet’s GDP.

The next category is creative tools. I’m privileged enough that I can speak the language of software, so I can tell Codex, “Here’s exactly what I want you to build.” I can articulate it in a way that gets me much more leverage than clumsily trying to iterate one thing at a time. But I can’t do that when it comes to making music, or painting, or creating animations2.

If I don’t have the ability to articulate the exact aesthetic I’m looking for, and I don’t know how to use Photoshop or Ableton, I’m left in this state where my taste exceeds my capability. I’m reminded of the classic quote on taste from Ira Glass, the host of This American Life:

Nobody tells this to people who are beginners, I wish someone told me. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase, they quit.

As voice models (and models more broadly) get better, they lower the friction of continuing to iterate and experiment. My hope is that fewer people quit before they can make something they’re happy with. Because I desperately want voice-driven tools to reach the point where I can clumsily vibe create, vibe compose, and vibe paint, and make something that’s beautiful to me.

If you squint, you can sort of see the generalized third category that subsumes both of the others: computer use.

Computer use has come a massive way since I first started thinking about this (way back in June). And Astra is now at the point where it can do truly almost anything I can do on my MacBook; often faster and more accurately!

Yet it raises an interesting question: if models are getting good enough to rival (or surpass) humans at using a computer, why am I talking to an app? Why am I talking to a terminal? Why am I not just talking to the entire computer?

The last category is event-to-speech: the model receives an event and speaks to the user. As a counterpoint to speech-to-action, I think this one is still very, very exploratory. To me, we haven’t yet seen what AI-native really looks like in this vein. But of the things I’ve seen, there are a couple of through lines I tend to notice.



The first is hands-free or screen-free experiences. There might be times when I need to interact with software or objects, and I can’t use my hands, or, more importantly, my attention is diverted elsewhere. That might be something like a recipe app: I’m chopping veggies and the app tells me it’s time to turn down the heat or start the next step. I don’t have to stop what I’m doing or deal with messy hands on my phone.

The other category is proactive outreach, where the model needs to tell you something or get your attention when you might not be looking. Every developer has an endless number of notifications and events happening in their software. No developer in their right mind would say, “I should show all of these logs” (nor would they say, “I should speak all of these logs.”).

We can start to conceive of voice as an upper level in that escalation path. Maybe you animate something, and then maybe you pop something up. If that doesn’t work, maybe you can talk to the user to get their attention.

We’ve had all these things for a while. People have been prototyping them for a while. Why focus on them now? Why think about building with them now?

Traditionally, voice agents have been built in a chained architecture. You talk, the system transcribes, and it sends that to a language model. It calls tools, and hopefully it doesn’t take too long to respond. It then generates text output, makes that into audio, and plays it back to the user.

But recent models, like GPT-Realtime, have taken a different approach: native audio. With native audio models, you send audio in and get audio back out, without first turning speech into text for the model to understand it.

It turns out there’s a lot of stuff you lose when you transcribe speech:

• Tone
• Emotion
• Cadence
• Backchannels
• Background noise

All of these are important context for humans to understand when communicating. Albert Mehrabian famously said that when speaking we convey 55% of the information via body language, 38% via tone of voice, and a mere 7% via the actual words we’re saying3.

Through that lens, it’s crazy that we’re expecting models to do a good job on only 7% of the context that humans are sharing!



GPT-Live has ended up blurring both of these architectures together. It pairs a frontend voice model, which can continuously ingest and emit audio (so it can backchannel with “mhm” while you’re talking, much like a person), but can also delegate work to a smarter backend model to call tools or just think harder.

With this approach, it means the audio model (GPT-Live) can focus on getting better at specific audio elements (naturalness, realism, etc) and worry less about having to be both natural and very intelligent.



If you’re a developer today, what does that mean for building your own software?

Fortunately, it’s much easier than you think to start adding audio as an intelligence layer to the apps you already have.

If you’re building a modern web application, you already expose so much of it as nouns and verbs. Think about the verbs: API endpoints and React hooks. Each of those can relatively easily be converted into a tool that you expose to a model, and then you can give the user the ability to drive your existing software with their voice.

Yes, you still need guardrails and safety checks. There’s work to do in securing and productizing this. But I want you to think about what it would mean to take your existing software and just talk to it.

To go back to that misconception from the beginning: if you’re talking to software, maybe it can talk back. We’ve also been developing other ways of communicating with the user for decades.

You can go back to the early days of personal computing and find people working through exactly these questions. Apple’s 1987 Human Interface Guidelines had a section called “Feedback and dialog,” which opened with “Keep the user informed. Provide immediate feedback.”

It discussed telling people when an operation was underway, when it had finished, and why they might have to wait. Elsewhere, it described animation as a way to show that an action was being carried out.

We take a lot of that work for granted now. Think about the different ways an app can communicate with you:

• What you can do: hover states, changes to the cursor, and disabled buttons help you see what’s interactive and what’s available.
• What changed: selected text, a checked box, or a highlighted row shows the result of an action.
• What’s still happening: progress bars, loading indicators, and animations let you know the software is working.
• What needs your attention: notifications, pop-ups, and modal dialogs can flag an update or ask you to make a decision.



Source: Apple’s Human Interface Guidelines

A voice agent baked into existing software can leverage all of these - sometimes the right thing to do is not to talk, but to create a pop-up or disable buttons.

And being able to drive software with your voice matters well beyond convenience. The same Apple guidelines had a section on designing for people with disabilities, including the observation that changes made for accessibility can make software easier for everyone to use.

On a personal note, I know multiple developers who, over the course of their careers, lost mobility in their hands or dexterity in their fingers. Many of them thought their careers as programmers were more or less over.

Then came large language models, coding agents, and voice agents. Now these developers generate orders of magnitude more code than they previously did on a given day or month.

The final thing I want to leave you with is this: when building voice agents, don’t start with the question of what kind of voice agent you’re trying to build. Start with the question of what the role of voice and audio is in this interaction, and then how to move forward from there.

Often when I ask that question, it leads to a bunch more questions.

• What can the model perceive? What context does it have?
• What tools are available to it, and which of those tools should it be executing safely and correctly?
• Should it communicate now? Should it wait?
• How should it communicate? Through which interfaces?

Taken together, I hope we can start to build much richer experiences with voice. A colleague of mine likes to say that Iron Man doesn’t text JARVIS - he talks. Because ultimately, AGI will be spoken, not typed.

Thanks for reading! If you found this interesting or insightful, please consider subscribing.

Subscribe

1

There are a couple of things worth pointing out about these categories. The first is that they aren’t new. There’s a long history of building these types of systems in and around voice. If you squint, you could make the argument that the Moviefone hotline, where you called in to get showtimes, was an example of a speech-to-speech system. GPS navigation in your car, which has existed since I was a kid, is an example of an event-to-speech system.

We can do much more interesting things with them now, and they’re remixable.

2

Increasingly, I don’t need to - Astra has been good enough out of the box to do some incredible painting and modeling work; but it’s still deeply useful to have the vocabulary to articulate what I want, lest I end up going back and forth with Codex to try and nail down a specific style or aesthetic.

3

The full context here is that this applies to when we’re trying to communicate feelings and emotions, not facts and figures. But AGI should be able to understand us when we’re feeling emotional!

Previous