|
1 | | -# Cascade Live Agent (ElevenLabs STT + TTS) |
| 1 | +# Cascade Live Agent |
2 | 2 |
|
3 | | -## What cascade live mode is |
| 3 | +## Overview |
4 | 4 |
|
5 | | -In cascade live mode, the agent's `model` is a `CascadeLive`, which wraps three |
6 | | -things: a plain text reasoner, an STT transform on the audio-in edge, and a TTS |
7 | | -transform on the audio-out edge. Speech becomes text, the reasoner answers in |
8 | | -text, and the answer becomes speech again. Tools and instruction are the same |
9 | | -as in text mode, while callbacks, tool confirmation, and long-running tools |
10 | | -follow live-mode semantics, as with any other live model. The same agent |
11 | | -definition still runs under `run_async` with no live session at all. |
| 5 | +A voice agent built with `CascadeLive`: an STT transform turns the user's |
| 6 | +speech into text, a Gemini text model answers and calls tools, and a TTS |
| 7 | +transform speaks the answer. The agent spells random words out and converts |
| 8 | +temperatures, and stops speaking when the user interrupts it. Tools and |
| 9 | +instruction are the same as in text mode, while |
| 10 | +callbacks, tool confirmation, and long-running tools follow live-mode |
| 11 | +semantics, as with any other live model. The same agent definition still runs |
| 12 | +under `run_async` with no live session at all. |
12 | 13 |
|
13 | | -```python |
14 | | -from google.adk.agents.llm_agent import Agent |
15 | | -from google.adk.integrations.eleven_labs import ElevenLabsSTT |
16 | | -from google.adk.integrations.eleven_labs import ElevenLabsTTS |
17 | | -from google.adk.live import CascadeLive |
| 14 | +The `CASCADE_SPEECH_PROVIDER` environment variable picks where the speech |
| 15 | +transforms come from: |
18 | 16 |
|
19 | | -root_agent = Agent( |
20 | | - model=CascadeLive( |
21 | | - model='gemini-3.5-flash', |
22 | | - stt=ElevenLabsSTT(), |
23 | | - tts=ElevenLabsTTS(), |
24 | | - ), |
25 | | - instruction='...', |
26 | | - tools=[convert_temperature], |
27 | | -) |
28 | | -``` |
29 | | - |
30 | | -This sample implements both transforms against ElevenLabs. An ElevenLabs |
31 | | -free-tier account includes enough credits and concurrency to run and test this |
32 | | -sample locally. |
33 | | - |
34 | | -## Get an ElevenLabs API key |
| 17 | +| `CASCADE_SPEECH_PROVIDER` | Speech to text | Text to speech | Setup | |
| 18 | +| ------------------------- | ------------------------------------- | -------------------------------------- | -------------------------------------------- | |
| 19 | +| `gemini` (default) | `GeminiSTT`, Gemini 3.5 Transcribe | `GeminiTTS`, Gemini 3.8 Flash-Lite TTS | [README_gemini.md](README_gemini.md) | |
| 20 | +| `elevenlabs` | `ElevenLabsSTT`, ElevenLabs Scribe v2 | `ElevenLabsTTS`, ElevenLabs Flash v2.5 | [README_elevenlabs.md](README_elevenlabs.md) | |
35 | 21 |
|
36 | | -1. **Create the key.** Sign up or sign in at [elevenlabs.io](https://elevenlabs.io) (the free plan is sufficient) and |
37 | | - create an API key at |
38 | | - **[elevenlabs.io/app/settings/api-keys](https://elevenlabs.io/app/settings/api-keys)** |
39 | | - (Dashboard → Settings → API Keys). |
| 22 | +## Setup |
40 | 23 |
|
41 | | -1. **Set the environment variable.** The sample reads `ELEVENLABS_API_KEY` from |
42 | | - the environment: |
| 24 | +1. Set up the credentials of the provider you pick, as its README describes. |
| 25 | + The reasoner is a Gemini text model, so it needs Google credentials with |
| 26 | + either provider. |
43 | 27 |
|
44 | | - Export it in your shell: |
| 28 | +1. Pick the provider in a `.env` file in this directory, which `adk web` loads, |
| 29 | + or in your shell. Leave it unset for Gemini: |
45 | 30 |
|
46 | | - ```bash |
47 | | - export ELEVENLABS_API_KEY=sk_... |
| 31 | + ```sh |
| 32 | + CASCADE_SPEECH_PROVIDER=elevenlabs |
48 | 33 | ``` |
49 | 34 |
|
50 | | - Alternatively, add it to a local `.env` file in this directory (which is |
51 | | - git-ignored) — `adk web` loads it automatically: |
| 35 | +## Run With Dev UI |
52 | 36 |
|
53 | | - ```bash |
54 | | - ELEVENLABS_API_KEY=sk_... |
55 | | - ``` |
56 | | - |
57 | | -1. **Install the SDK.** |
| 37 | +From the directory that *contains* the `cascade` folder (`samples/live/`), |
| 38 | +start the ADK web server: |
58 | 39 |
|
59 | | - ```bash |
60 | | - pip install "google-adk[elevenlabs]" |
61 | | - ``` |
62 | | - |
63 | | - This installs a supported `elevenlabs` release, which also brings in |
64 | | - `websockets` for the realtime recognizer. |
65 | | - |
66 | | -## Credentials for the reasoner |
67 | | - |
68 | | -The reasoner is an ordinary Gemini text model and authenticates the same way |
69 | | -every other ADK sample does. Put a `.env` in this directory with either: |
70 | | - |
71 | | -```sh |
72 | | -# Gemini API |
73 | | -GOOGLE_GENAI_USE_ENTERPRISE=FALSE |
74 | | -GOOGLE_API_KEY=... |
75 | | -``` |
76 | | - |
77 | | -or: |
78 | | - |
79 | | -```sh |
80 | | -# Vertex AI |
81 | | -GOOGLE_GENAI_USE_ENTERPRISE=TRUE |
82 | | -GOOGLE_CLOUD_PROJECT=your-project |
83 | | -GOOGLE_CLOUD_LOCATION=us-central1 |
| 40 | +```bash |
| 41 | +adk web |
84 | 42 | ``` |
85 | 43 |
|
86 | | -Unlike the native-audio samples, the project does **not** need Live API access: |
87 | | -cascade only ever calls `generate_content` on a text model. |
88 | | - |
89 | | -## Run it |
90 | | - |
91 | | -1. **Start the ADK web server.** From the directory that *contains* the |
92 | | - `cascade` folder (`samples/live/`): |
93 | | - |
94 | | - ```bash |
95 | | - adk web |
96 | | - ``` |
97 | | - |
98 | | -1. **Open the ADK web UI** at the URL printed in the terminal, usually |
99 | | - `http://localhost:8000`. |
100 | | - |
101 | | -1. **Select the agent.** Pick `cascade` from the dropdown in the top-left. |
102 | | - |
103 | | -1. **Start streaming.** Click the **Audio** icon next to the chat input. |
104 | | - |
105 | | -1. **Talk to it.** Try "Spell five random words." or "What is twenty-three |
106 | | - degrees Celsius in Fahrenheit?". You will see the live transcript appear as |
107 | | - captions while you speak, and hear the answer back. |
108 | | - |
109 | | -1. **Interrupt it.** While it is spelling the words, start talking, for |
110 | | - example "Stop. Which words did you spell?". The agent stops speaking and |
111 | | - answers with only the words you heard before you interrupted. |
112 | | - |
| 44 | +Open the URL printed in the terminal, usually `http://localhost:8000`, pick |
| 45 | +`cascade` from the dropdown in the top-left, then click the **Audio** icon next |
| 46 | +to the chat input and speak. Captions of what you say appear while you speak. |
113 | 47 | Click **Audio** again to stop streaming. You can stop and restart the stream |
114 | 48 | any number of times within the same session. |
115 | 49 |
|
116 | | -## Which ElevenLabs models and voices are used |
117 | | - |
118 | | -| Role | Value | Where to change it | |
119 | | -| -------------- | --------------------------------- | -------------------------------------- | |
120 | | -| Speech to text | `scribe_v2_realtime` | `ElevenLabsSTT(model_id=...)` | |
121 | | -| Text to speech | `eleven_flash_v2_5` | `ElevenLabsTTS(model_id=...)` | |
122 | | -| Voice | `JBFqnCBsd6RMkjVDRZzb` ("George") | `ElevenLabsTTS(voice_id=...)` | |
123 | | -| Output audio | `24000` (format: `pcm_24000`) | `ElevenLabsTTS(sample_rate=...)` (int) | |
| 50 | +## Sample Inputs |
124 | 51 |
|
125 | | -All four can be customized via keyword arguments to `ElevenLabsSTT` and |
126 | | -`ElevenLabsTTS` in `agent.py` (which currently uses their default values). Browse |
127 | | -voices at [elevenlabs.io/app/voice-library](https://elevenlabs.io/app/voice-library) |
128 | | -and paste the Voice ID. `eleven_flash_v2_5` is the low-latency synthesis model |
129 | | -(~75 ms excluding network); swap in `eleven_multilingual_v2` for better quality |
130 | | -at higher latency, or `eleven_v3_conversational` for a more expressive delivery. |
131 | | -See the [models overview](https://elevenlabs.io/docs/overview/models). |
| 52 | +Say these out loud: |
132 | 53 |
|
133 | | -## How it works |
| 54 | +- `Spell five random words.` |
134 | 55 |
|
135 | | -### Speech to text — `ElevenLabsSTT` (`google.adk.integrations.eleven_labs`) |
| 56 | + *The model picks five words and spells each one, letter by letter, in a |
| 57 | + single reply.* |
136 | 58 |
|
137 | | -This uses **Scribe v2 Realtime**, ElevenLabs' streaming WebSocket recognizer. |
138 | | -The connection is opened with `commit_strategy=CommitStrategy.VAD`, so the |
139 | | -server performs voice activity detection and determines utterance boundaries. |
| 59 | +- `Stop. Which words did you spell?`, said while the agent is still spelling. |
140 | 60 |
|
141 | | -The mapping onto ADK's ingress events is then direct: |
| 61 | + *The agent stops speaking at once and answers with only the words you heard |
| 62 | + before you interrupted.* |
142 | 63 |
|
143 | | -| Scribe message | ADK event | |
144 | | -| ---------------------- | ----------------------------------------------------------------------- | |
145 | | -| `partial_transcript` | `PartialTranscript` (and `UserSpeechStarted` on the first of a segment) | |
146 | | -| `committed_transcript` | `UserTurnFinished` | |
147 | | -| any error message | Raises `RuntimeError` (which closes the live connection) | |
| 64 | +- `What is twenty-three degrees Celsius in Fahrenheit?` |
148 | 65 |
|
149 | | -`UserTurnFinished` is emitted when a commit occurs, which triggers the reasoning |
150 | | -pass. Empty commits containing only silence are ignored. |
| 66 | + *The model calls `convert_temperature` and reads the result back.* |
151 | 67 |
|
152 | | -Implementation notes: |
| 68 | +## Graph |
153 | 69 |
|
154 | | -- The `ElevenLabsSTT` adapter batches incoming audio frames into ~100 ms chunks |
155 | | - to match ElevenLabs' streaming recommendations. |
156 | | -- When the audio stream ends (the microphone is closed), the adapter flushes its |
157 | | - buffer, sends an explicit `commit()`, and waits up to `flush_timeout_secs` |
158 | | - for the final transcript. |
159 | | - |
160 | | -**ElevenLabs STT Reference:** |
161 | | - |
162 | | -- [Server-side streaming](https://elevenlabs.io/docs/eleven-api/guides/how-to/speech-to-text/realtime/server-side-streaming) |
163 | | -- [Transcripts and commit strategies](https://elevenlabs.io/docs/eleven-api/guides/how-to/speech-to-text/realtime/transcripts-and-commit-strategies) |
164 | | -- [Event reference](https://elevenlabs.io/docs/eleven-api/guides/how-to/speech-to-text/realtime/event-reference) |
165 | | - |
166 | | -### Text to speech — `ElevenLabsTTS` (`google.adk.integrations.eleven_labs`) |
167 | | - |
168 | | -Text deltas from the reasoner are buffered and segmented into sentences using a |
169 | | -regex heuristic (terminal punctuation `[.!?…]` followed by optional |
170 | | -quotes/brackets and whitespace or newlines, falling back to a word boundary |
171 | | -after 240 characters). It streams each complete sentence through |
172 | | -`text_to_speech.stream()` with `output_format='pcm_24000'` — raw 16-bit |
173 | | -little-endian PCM, so the bytes are forwarded to the client with no transcode. |
174 | | -Every `AudioChunk` carries `mime_type='audio/pcm;rate=24000'` so the client plays |
175 | | -it at the right rate. |
176 | | - |
177 | | -`AgentSpokenOutput` is emitted after a sentence has been synthesized in full to |
178 | | -record it in conversation history. Synthesis stops when the `cancel` event is |
179 | | -set, which `CascadeLive` does when the user interrupts the agent (barge-in). |
| 70 | +```mermaid |
| 71 | +graph TD |
| 72 | + Agent[cascade_agent] -->|model| Cascade[CascadeLive] |
| 73 | + Cascade -->|stt| STT(GeminiSTT or ElevenLabsSTT) |
| 74 | + Cascade -->|reasoner| LLM(gemini-3.8-flash) |
| 75 | + Cascade -->|tts| TTS(GeminiTTS or ElevenLabsTTS) |
| 76 | + Agent -->|calls| Convert(convert_temperature) |
| 77 | +``` |
180 | 78 |
|
181 | | -Docs: [stream speech](https://elevenlabs.io/docs/api-reference/text-to-speech/stream). |
| 79 | +## How To |
182 | 80 |
|
183 | | -## Limitations |
| 81 | +Set a `CascadeLive` as the agent's `model`, with an STT and a TTS transform, |
| 82 | +and keep the reasoner's thinking low: |
184 | 83 |
|
185 | | -- **Turn detection relies only on silence.** The recognizer ends the user's |
186 | | - turn after a fixed period of silence (`vad_silence_threshold_secs`, 0.8 s by |
187 | | - default). Each time it does, it emits a `UserTurnFinished` and the model |
188 | | - answers. This causes two problems: |
| 84 | +```python |
| 85 | +root_agent = Agent( |
| 86 | + model=CascadeLive( |
| 87 | + model='gemini-3.8-flash', |
| 88 | + stt=GeminiSTT(), # Or ElevenLabsSTT(). |
| 89 | + tts=GeminiTTS(speech_config='Kore'), # Or ElevenLabsTTS(). |
| 90 | + ), |
| 91 | + instruction='...', |
| 92 | + tools=[convert_temperature], |
| 93 | + generate_content_config=types.GenerateContentConfig( |
| 94 | + thinking_config=types.ThinkingConfig( |
| 95 | + thinking_level=types.ThinkingLevel.LOW |
| 96 | + ), |
| 97 | + ), |
| 98 | +) |
| 99 | +``` |
189 | 100 |
|
190 | | - - If the user pauses mid-sentence (for example, "Convert twenty-three |
191 | | - degrees… um… Celsius"), the sentence is split into two turns, and the model |
192 | | - answers each one separately. |
193 | | - - In a noisy room, the recognizer may never detect silence, so the turn never |
194 | | - ends. |
| 101 | +- **Thinking level**: thinking delays the first word of every reply, so the |
| 102 | + reasoner thinks at `LOW`, the lowest level Gemini 3.8 Flash supports. |
| 103 | +- **Instruction**: both TTS transforms cut the reply at sentence ends, so the |
| 104 | + instruction asks for short sentences that end with punctuation; a sentence |
| 105 | + that never ends waits before it is spoken. It also asks for numbers, symbols, |
| 106 | + and abbreviations written as words, because the synthesizer reads the text |
| 107 | + as it is, and an abbreviation such as `Dr.` would end a sentence. |
195 | 108 |
|
196 | | - There is no semantic turn detection. To adjust the behavior, tune |
197 | | - `vad_silence_threshold_secs`, `vad_threshold`, `min_speech_duration_ms`, and |
198 | | - `min_silence_duration_ms`. |
| 109 | +### Limitations |
199 | 110 |
|
| 111 | +- **Turn detection relies on silence.** Both recognizers end the user's turn |
| 112 | + after a fixed period of silence, so a pause mid-sentence splits it into two |
| 113 | + turns, and the model answers each one. Each provider's README says how to |
| 114 | + tune the silence. |
200 | 115 | - **Don't speak while a tool is running.** If you finish speaking after the |
201 | 116 | model calls a tool but before the tool returns, your words are added to the |
202 | 117 | conversation before the tool's result. The model then gets the tool call and |
203 | 118 | its result out of order, and this turn and all later turns in the session |
204 | 119 | fail. Wait for the agent to answer before you speak again. |
205 | | - |
206 | | -- **Sentence splitting uses a simple regex.** `ElevenLabsTTS` splits text after |
207 | | - `.`, `!`, `?`, or `…` (followed by optional quotes or brackets and then |
208 | | - whitespace), and at line breaks. If no sentence end appears within 240 |
209 | | - characters, it splits at the last space before that limit. |
210 | | - |
211 | | - - *Abbreviations:* Text such as `Dr. Smith` or `Google Inc.` is split at the |
212 | | - period. To avoid this, `agent.py` tells the model to write abbreviations as |
213 | | - words (for example, "Doctor"). It also tells the model to end every |
214 | | - sentence with punctuation, so each sentence is spoken as soon as it is |
215 | | - complete instead of waiting for the 240-character limit. |
216 | | - - *Other languages:* The regex works for languages that separate words with |
217 | | - spaces and use Latin punctuation. It does not recognize punctuation such as |
218 | | - the CJK `。` or the Hindi `।`. Hindi text is still split at the last space |
219 | | - before the 240-character limit. Chinese and Japanese text has no spaces, so |
220 | | - the whole reply is buffered and spoken only after the model finishes. |
221 | | - - *Production use:* Replace the regex with a sentence-boundary library such |
222 | | - as `pysbd`, `blingfire`, or `PyICU` (Unicode UAX #29). |
223 | | - |
| 120 | +- **The agent can hear itself.** The recognizer hears whatever the microphone |
| 121 | + picks up, so without echo cancellation the agent's own voice from a speaker |
| 122 | + can count as the user speaking and interrupt the reply. Use headphones if |
| 123 | + that happens. |
224 | 124 | - **Responses are slower than with native audio models.** Each turn runs three |
225 | 125 | steps in sequence (recognize, reason, synthesize), whereas a native |
226 | 126 | speech-to-speech model runs one. The model also sees only the transcribed |
227 | 127 | words, so it cannot perceive the user's tone, and the synthesized speech does |
228 | 128 | not carry the model's intent beyond the words. |
229 | 129 |
|
230 | | -- **Each audio segment opens a new recognizer session.** `ActivityEnd` and |
231 | | - `audio_stream_end` close the current STT session, and the next audio frame |
232 | | - opens a new one. Stopping and restarting the microphone works, but each |
233 | | - restart adds the time it takes to open a new STT connection. |
| 130 | +## Related Guides |
| 131 | + |
| 132 | +- [CascadeLive](../../../../docs/guides/live/cascade_live/index.md) - How |
| 133 | + `CascadeLive` composes a text model with STT and TTS transforms. |
| 134 | +- [GeminiSTT](../../../../docs/guides/live/gemini_stt/index.md) - Speech |
| 135 | + recognition with Gemini streaming transcription. |
| 136 | +- [GeminiTTS](../../../../docs/guides/live/gemini_tts/index.md) - Speech |
| 137 | + synthesis with Gemini text-to-speech models. |
0 commit comments