Skip to content

Commit f78149a

Browse files
wuliang229copybara-github
authored andcommitted
feat(live): add Gemini speech-to-text and text-to-speech for CascadeLive
Adds `GeminiSTT` and `GeminiTTS`, the `LiveIngress` and `LiveEgress` transforms backed by Gemini, so a `CascadeLive` agent can run recognition, reasoning and synthesis on Gemini with one set of credentials and no extra dependency. Both are built for low latency. - `GeminiSTT` streams the user's audio to `gemini-3.5-transcribe-live` over the Live API, so speech is transcribed while the user talks. The server's voice activity detection reports `UserSpeechStarted`, interim transcripts become `PartialTranscript` captions, and the final transcript becomes `UserTurnFinished` after 500 ms of silence by default. `UserSpeechStarted` lets `CascadeLive` interrupt the agent as soon as the user talks over it. If the client signals the end of the user's turn itself, for example by sending `ActivityEnd` when a push-to-talk button is released, the final transcript arrives right away instead of after the silence. Sessions are replaced before the Live API's 10-minute limit, between utterances or right away on GoAway, and a session that drops is reopened; either way the new session is sent the speech the old one had not finished transcribing, up to `max_replay_ms`, and the start of that speech is not reported twice. - `GeminiTTS` splits the reasoner's text into segments and synthesizes each with a streaming `generate_content` call to `gemini-3.8-flash-lite-tts`. The next segment is synthesized while the current one is delivered, so there are no gaps between segments, and the first reply warms up the connection while the reasoner thinks. A request that fails with a transient error before returning any audio is retried. The first segment ends at the first sentence end; later ones group sentences until they reach `min_segment_chars` (50), because a spelled-out letter such as `Y.` synthesized on its own is often not read as its name. - Both are experimental, behind the `CASCADE_LIVE` feature. - The `live/cascade` sample picks its speech transforms with the `CASCADE_SPEECH_PROVIDER` environment variable, `gemini` by default or `elevenlabs`, with a companion README for each provider. - Adds unit guides for `GeminiSTT` and `GeminiTTS`, referenced from the CascadeLive guide. Co-authored-by: Liang Wu <wuliang@google.com> PiperOrigin-RevId: 1000461641
1 parent 59fe93b commit f78149a

13 files changed

Lines changed: 3462 additions & 198 deletions

File tree

Lines changed: 94 additions & 190 deletions
Original file line numberDiff line numberDiff line change
@@ -1,233 +1,137 @@
1-
# Cascade Live Agent (ElevenLabs STT + TTS)
1+
# Cascade Live Agent
22

3-
## What cascade live mode is
3+
## Overview
44

5-
In cascade live mode, the agent's `model` is a `CascadeLive`, which wraps three
6-
things: a plain text reasoner, an STT transform on the audio-in edge, and a TTS
7-
transform on the audio-out edge. Speech becomes text, the reasoner answers in
8-
text, and the answer becomes speech again. Tools and instruction are the same
9-
as in text mode, while callbacks, tool confirmation, and long-running tools
10-
follow live-mode semantics, as with any other live model. The same agent
11-
definition still runs under `run_async` with no live session at all.
5+
A voice agent built with `CascadeLive`: an STT transform turns the user's
6+
speech into text, a Gemini text model answers and calls tools, and a TTS
7+
transform speaks the answer. The agent spells random words out and converts
8+
temperatures, and stops speaking when the user interrupts it. Tools and
9+
instruction are the same as in text mode, while
10+
callbacks, tool confirmation, and long-running tools follow live-mode
11+
semantics, as with any other live model. The same agent definition still runs
12+
under `run_async` with no live session at all.
1213

13-
```python
14-
from google.adk.agents.llm_agent import Agent
15-
from google.adk.integrations.eleven_labs import ElevenLabsSTT
16-
from google.adk.integrations.eleven_labs import ElevenLabsTTS
17-
from google.adk.live import CascadeLive
14+
The `CASCADE_SPEECH_PROVIDER` environment variable picks where the speech
15+
transforms come from:
1816

19-
root_agent = Agent(
20-
model=CascadeLive(
21-
model='gemini-3.5-flash',
22-
stt=ElevenLabsSTT(),
23-
tts=ElevenLabsTTS(),
24-
),
25-
instruction='...',
26-
tools=[convert_temperature],
27-
)
28-
```
29-
30-
This sample implements both transforms against ElevenLabs. An ElevenLabs
31-
free-tier account includes enough credits and concurrency to run and test this
32-
sample locally.
33-
34-
## Get an ElevenLabs API key
17+
| `CASCADE_SPEECH_PROVIDER` | Speech to text | Text to speech | Setup |
18+
| ------------------------- | ------------------------------------- | -------------------------------------- | -------------------------------------------- |
19+
| `gemini` (default) | `GeminiSTT`, Gemini 3.5 Transcribe | `GeminiTTS`, Gemini 3.8 Flash-Lite TTS | [README_gemini.md](README_gemini.md) |
20+
| `elevenlabs` | `ElevenLabsSTT`, ElevenLabs Scribe v2 | `ElevenLabsTTS`, ElevenLabs Flash v2.5 | [README_elevenlabs.md](README_elevenlabs.md) |
3521

36-
1. **Create the key.** Sign up or sign in at [elevenlabs.io](https://elevenlabs.io) (the free plan is sufficient) and
37-
create an API key at
38-
**[elevenlabs.io/app/settings/api-keys](https://elevenlabs.io/app/settings/api-keys)**
39-
(Dashboard → Settings → API Keys).
22+
## Setup
4023

41-
1. **Set the environment variable.** The sample reads `ELEVENLABS_API_KEY` from
42-
the environment:
24+
1. Set up the credentials of the provider you pick, as its README describes.
25+
The reasoner is a Gemini text model, so it needs Google credentials with
26+
either provider.
4327

44-
Export it in your shell:
28+
1. Pick the provider in a `.env` file in this directory, which `adk web` loads,
29+
or in your shell. Leave it unset for Gemini:
4530

46-
```bash
47-
export ELEVENLABS_API_KEY=sk_...
31+
```sh
32+
CASCADE_SPEECH_PROVIDER=elevenlabs
4833
```
4934

50-
Alternatively, add it to a local `.env` file in this directory (which is
51-
git-ignored) — `adk web` loads it automatically:
35+
## Run With Dev UI
5236

53-
```bash
54-
ELEVENLABS_API_KEY=sk_...
55-
```
56-
57-
1. **Install the SDK.**
37+
From the directory that *contains* the `cascade` folder (`samples/live/`),
38+
start the ADK web server:
5839

59-
```bash
60-
pip install "google-adk[elevenlabs]"
61-
```
62-
63-
This installs a supported `elevenlabs` release, which also brings in
64-
`websockets` for the realtime recognizer.
65-
66-
## Credentials for the reasoner
67-
68-
The reasoner is an ordinary Gemini text model and authenticates the same way
69-
every other ADK sample does. Put a `.env` in this directory with either:
70-
71-
```sh
72-
# Gemini API
73-
GOOGLE_GENAI_USE_ENTERPRISE=FALSE
74-
GOOGLE_API_KEY=...
75-
```
76-
77-
or:
78-
79-
```sh
80-
# Vertex AI
81-
GOOGLE_GENAI_USE_ENTERPRISE=TRUE
82-
GOOGLE_CLOUD_PROJECT=your-project
83-
GOOGLE_CLOUD_LOCATION=us-central1
40+
```bash
41+
adk web
8442
```
8543

86-
Unlike the native-audio samples, the project does **not** need Live API access:
87-
cascade only ever calls `generate_content` on a text model.
88-
89-
## Run it
90-
91-
1. **Start the ADK web server.** From the directory that *contains* the
92-
`cascade` folder (`samples/live/`):
93-
94-
```bash
95-
adk web
96-
```
97-
98-
1. **Open the ADK web UI** at the URL printed in the terminal, usually
99-
`http://localhost:8000`.
100-
101-
1. **Select the agent.** Pick `cascade` from the dropdown in the top-left.
102-
103-
1. **Start streaming.** Click the **Audio** icon next to the chat input.
104-
105-
1. **Talk to it.** Try "Spell five random words." or "What is twenty-three
106-
degrees Celsius in Fahrenheit?". You will see the live transcript appear as
107-
captions while you speak, and hear the answer back.
108-
109-
1. **Interrupt it.** While it is spelling the words, start talking, for
110-
example "Stop. Which words did you spell?". The agent stops speaking and
111-
answers with only the words you heard before you interrupted.
112-
44+
Open the URL printed in the terminal, usually `http://localhost:8000`, pick
45+
`cascade` from the dropdown in the top-left, then click the **Audio** icon next
46+
to the chat input and speak. Captions of what you say appear while you speak.
11347
Click **Audio** again to stop streaming. You can stop and restart the stream
11448
any number of times within the same session.
11549

116-
## Which ElevenLabs models and voices are used
117-
118-
| Role | Value | Where to change it |
119-
| -------------- | --------------------------------- | -------------------------------------- |
120-
| Speech to text | `scribe_v2_realtime` | `ElevenLabsSTT(model_id=...)` |
121-
| Text to speech | `eleven_flash_v2_5` | `ElevenLabsTTS(model_id=...)` |
122-
| Voice | `JBFqnCBsd6RMkjVDRZzb` ("George") | `ElevenLabsTTS(voice_id=...)` |
123-
| Output audio | `24000` (format: `pcm_24000`) | `ElevenLabsTTS(sample_rate=...)` (int) |
50+
## Sample Inputs
12451

125-
All four can be customized via keyword arguments to `ElevenLabsSTT` and
126-
`ElevenLabsTTS` in `agent.py` (which currently uses their default values). Browse
127-
voices at [elevenlabs.io/app/voice-library](https://elevenlabs.io/app/voice-library)
128-
and paste the Voice ID. `eleven_flash_v2_5` is the low-latency synthesis model
129-
(~75 ms excluding network); swap in `eleven_multilingual_v2` for better quality
130-
at higher latency, or `eleven_v3_conversational` for a more expressive delivery.
131-
See the [models overview](https://elevenlabs.io/docs/overview/models).
52+
Say these out loud:
13253

133-
## How it works
54+
- `Spell five random words.`
13455

135-
### Speech to text — `ElevenLabsSTT` (`google.adk.integrations.eleven_labs`)
56+
*The model picks five words and spells each one, letter by letter, in a
57+
single reply.*
13658

137-
This uses **Scribe v2 Realtime**, ElevenLabs' streaming WebSocket recognizer.
138-
The connection is opened with `commit_strategy=CommitStrategy.VAD`, so the
139-
server performs voice activity detection and determines utterance boundaries.
59+
- `Stop. Which words did you spell?`, said while the agent is still spelling.
14060

141-
The mapping onto ADK's ingress events is then direct:
61+
*The agent stops speaking at once and answers with only the words you heard
62+
before you interrupted.*
14263

143-
| Scribe message | ADK event |
144-
| ---------------------- | ----------------------------------------------------------------------- |
145-
| `partial_transcript` | `PartialTranscript` (and `UserSpeechStarted` on the first of a segment) |
146-
| `committed_transcript` | `UserTurnFinished` |
147-
| any error message | Raises `RuntimeError` (which closes the live connection) |
64+
- `What is twenty-three degrees Celsius in Fahrenheit?`
14865

149-
`UserTurnFinished` is emitted when a commit occurs, which triggers the reasoning
150-
pass. Empty commits containing only silence are ignored.
66+
*The model calls `convert_temperature` and reads the result back.*
15167

152-
Implementation notes:
68+
## Graph
15369

154-
- The `ElevenLabsSTT` adapter batches incoming audio frames into ~100 ms chunks
155-
to match ElevenLabs' streaming recommendations.
156-
- When the audio stream ends (the microphone is closed), the adapter flushes its
157-
buffer, sends an explicit `commit()`, and waits up to `flush_timeout_secs`
158-
for the final transcript.
159-
160-
**ElevenLabs STT Reference:**
161-
162-
- [Server-side streaming](https://elevenlabs.io/docs/eleven-api/guides/how-to/speech-to-text/realtime/server-side-streaming)
163-
- [Transcripts and commit strategies](https://elevenlabs.io/docs/eleven-api/guides/how-to/speech-to-text/realtime/transcripts-and-commit-strategies)
164-
- [Event reference](https://elevenlabs.io/docs/eleven-api/guides/how-to/speech-to-text/realtime/event-reference)
165-
166-
### Text to speech — `ElevenLabsTTS` (`google.adk.integrations.eleven_labs`)
167-
168-
Text deltas from the reasoner are buffered and segmented into sentences using a
169-
regex heuristic (terminal punctuation `[.!?…]` followed by optional
170-
quotes/brackets and whitespace or newlines, falling back to a word boundary
171-
after 240 characters). It streams each complete sentence through
172-
`text_to_speech.stream()` with `output_format='pcm_24000'` — raw 16-bit
173-
little-endian PCM, so the bytes are forwarded to the client with no transcode.
174-
Every `AudioChunk` carries `mime_type='audio/pcm;rate=24000'` so the client plays
175-
it at the right rate.
176-
177-
`AgentSpokenOutput` is emitted after a sentence has been synthesized in full to
178-
record it in conversation history. Synthesis stops when the `cancel` event is
179-
set, which `CascadeLive` does when the user interrupts the agent (barge-in).
70+
```mermaid
71+
graph TD
72+
Agent[cascade_agent] -->|model| Cascade[CascadeLive]
73+
Cascade -->|stt| STT(GeminiSTT or ElevenLabsSTT)
74+
Cascade -->|reasoner| LLM(gemini-3.8-flash)
75+
Cascade -->|tts| TTS(GeminiTTS or ElevenLabsTTS)
76+
Agent -->|calls| Convert(convert_temperature)
77+
```
18078

181-
Docs: [stream speech](https://elevenlabs.io/docs/api-reference/text-to-speech/stream).
79+
## How To
18280

183-
## Limitations
81+
Set a `CascadeLive` as the agent's `model`, with an STT and a TTS transform,
82+
and keep the reasoner's thinking low:
18483

185-
- **Turn detection relies only on silence.** The recognizer ends the user's
186-
turn after a fixed period of silence (`vad_silence_threshold_secs`, 0.8 s by
187-
default). Each time it does, it emits a `UserTurnFinished` and the model
188-
answers. This causes two problems:
84+
```python
85+
root_agent = Agent(
86+
model=CascadeLive(
87+
model='gemini-3.8-flash',
88+
stt=GeminiSTT(), # Or ElevenLabsSTT().
89+
tts=GeminiTTS(speech_config='Kore'), # Or ElevenLabsTTS().
90+
),
91+
instruction='...',
92+
tools=[convert_temperature],
93+
generate_content_config=types.GenerateContentConfig(
94+
thinking_config=types.ThinkingConfig(
95+
thinking_level=types.ThinkingLevel.LOW
96+
),
97+
),
98+
)
99+
```
189100

190-
- If the user pauses mid-sentence (for example, "Convert twenty-three
191-
degrees… um… Celsius"), the sentence is split into two turns, and the model
192-
answers each one separately.
193-
- In a noisy room, the recognizer may never detect silence, so the turn never
194-
ends.
101+
- **Thinking level**: thinking delays the first word of every reply, so the
102+
reasoner thinks at `LOW`, the lowest level Gemini 3.8 Flash supports.
103+
- **Instruction**: both TTS transforms cut the reply at sentence ends, so the
104+
instruction asks for short sentences that end with punctuation; a sentence
105+
that never ends waits before it is spoken. It also asks for numbers, symbols,
106+
and abbreviations written as words, because the synthesizer reads the text
107+
as it is, and an abbreviation such as `Dr.` would end a sentence.
195108

196-
There is no semantic turn detection. To adjust the behavior, tune
197-
`vad_silence_threshold_secs`, `vad_threshold`, `min_speech_duration_ms`, and
198-
`min_silence_duration_ms`.
109+
### Limitations
199110

111+
- **Turn detection relies on silence.** Both recognizers end the user's turn
112+
after a fixed period of silence, so a pause mid-sentence splits it into two
113+
turns, and the model answers each one. Each provider's README says how to
114+
tune the silence.
200115
- **Don't speak while a tool is running.** If you finish speaking after the
201116
model calls a tool but before the tool returns, your words are added to the
202117
conversation before the tool's result. The model then gets the tool call and
203118
its result out of order, and this turn and all later turns in the session
204119
fail. Wait for the agent to answer before you speak again.
205-
206-
- **Sentence splitting uses a simple regex.** `ElevenLabsTTS` splits text after
207-
`.`, `!`, `?`, or `…` (followed by optional quotes or brackets and then
208-
whitespace), and at line breaks. If no sentence end appears within 240
209-
characters, it splits at the last space before that limit.
210-
211-
- *Abbreviations:* Text such as `Dr. Smith` or `Google Inc.` is split at the
212-
period. To avoid this, `agent.py` tells the model to write abbreviations as
213-
words (for example, "Doctor"). It also tells the model to end every
214-
sentence with punctuation, so each sentence is spoken as soon as it is
215-
complete instead of waiting for the 240-character limit.
216-
- *Other languages:* The regex works for languages that separate words with
217-
spaces and use Latin punctuation. It does not recognize punctuation such as
218-
the CJK `。` or the Hindi `।`. Hindi text is still split at the last space
219-
before the 240-character limit. Chinese and Japanese text has no spaces, so
220-
the whole reply is buffered and spoken only after the model finishes.
221-
- *Production use:* Replace the regex with a sentence-boundary library such
222-
as `pysbd`, `blingfire`, or `PyICU` (Unicode UAX #29).
223-
120+
- **The agent can hear itself.** The recognizer hears whatever the microphone
121+
picks up, so without echo cancellation the agent's own voice from a speaker
122+
can count as the user speaking and interrupt the reply. Use headphones if
123+
that happens.
224124
- **Responses are slower than with native audio models.** Each turn runs three
225125
steps in sequence (recognize, reason, synthesize), whereas a native
226126
speech-to-speech model runs one. The model also sees only the transcribed
227127
words, so it cannot perceive the user's tone, and the synthesized speech does
228128
not carry the model's intent beyond the words.
229129

230-
- **Each audio segment opens a new recognizer session.** `ActivityEnd` and
231-
`audio_stream_end` close the current STT session, and the next audio frame
232-
opens a new one. Stopping and restarting the microphone works, but each
233-
restart adds the time it takes to open a new STT connection.
130+
## Related Guides
131+
132+
- [CascadeLive](../../../../docs/guides/live/cascade_live/index.md) - How
133+
`CascadeLive` composes a text model with STT and TTS transforms.
134+
- [GeminiSTT](../../../../docs/guides/live/gemini_stt/index.md) - Speech
135+
recognition with Gemini streaming transcription.
136+
- [GeminiTTS](../../../../docs/guides/live/gemini_tts/index.md) - Speech
137+
synthesis with Gemini text-to-speech models.

0 commit comments

Comments
 (0)