feat(voice): add Phase 1 voice interaction for Lucia - #421
Draft
Ticed wants to merge 24 commits into
Draft
Conversation
Move the local Lucia speech bridge work onto infernode-os/infernode so future PRs target the maintained canonical repository instead of the archived NERVsystems fork. Constraint: NERVsystems/infernode is archived/read-only; Ticed/infernode-os remains the PR fork for infernode-os/infernode. Rejected: Continue developing against NERVsystems/infernode | archived source can no longer be an operational target. Confidence: high Scope-risk: moderate Directive: Treat the archived NERVsystems branch as historical input only; keep future work on infernode-os/infernode and push PR branches to origin. Tested: git diff --check; targeted Limbo compiles for lucibridge/luciuisrv/voicemode/speech9p/luciuisrv_test/speech9p_voice_test; ./build-macos-headless.sh; tests/host/audio_macos_test.sh ctl; luciuisrv_test 36 passed; speech9p_voice_test 6 passed. Not-tested: full ./run-tests.sh; full appl/cmd mk install is blocked by unrelated appl/cmd/git/push.b interface mismatch.
Move Phase 1 voice mode onto a single speech-provider contract so wake, listen, say, cancel, and voices can be served by speechshim9p, a Parakeet export, or a remote 9P provider without changing the Lucia-facing /n/speech surface. speech9p delegates streaming speech I/O to the provider mount and parks blocking wake/listen/hear/say reads asynchronously so ctl and cancel remain live for barge-in. voicemode becomes a resident input-mode-driven state machine, lucibridge reads keyboard and voice inputs concurrently, Lucifer maps Esc to keyboard mode, and boot starts the speech stack before lucibridge. Constraint: helper binaries remain external and optional; the in-tree shim adapts them through namespace files and soft-fails when unavailable. Rejected: Run wake/listen/TTS helpers directly inside speech9p | blocking helper reads freeze the 9P serveloop and break cancel/barge-in. Confidence: high Scope-risk: moderate Directive: Keep future speech engines behind the provider 9P contract; do not add engine-specific paths to lucibridge or voicemode. Tested: speechshim_test 5/5; speech_wake/speech_listen/speech_kokoro 12/12; voicemode 6/6; speech9p_voice 6/6; luciuisrv 36/36; lucibridge 49/49; audio_macos smoke; dis-path verifier; targeted regressions for touched modules. Not-tested: full exhaustive suite was stopped at user request after targeted coverage; live dogfood with real Kokoro/whisper/openWakeWord helpers, mic permission, wake word, and latency targets; remote-audio Phase 2 path.
Route speechshim9p playback and capture through configured namespace paths so local helpers, remote processing hosts, and remote capture devices can use the same provider contract without adding engine-specific remoting paths. speechshim9p now accepts audiodev, capturedev, micmode, and capturerate ctl settings; micmode device pumps s16le PCM from the capture device into listen/wake helper stdin, while speech9p forwards the routing knobs to the mounted provider. The remote-audio documentation now describes the local, remote-engine, and remote-microphone topologies as the same 9P composition. Constraint: remote voice I/O must remain ordinary namespace composition through the speech provider mount, with helper binaries still external. Rejected: Require bind-based /dev/audio replacement or special speech9p remoting logic | audio routing can be expressed as provider ctl state and namespace paths. Confidence: high Scope-risk: moderate Directive: Keep future remote speech topologies behind provider ctl keys and 9P mounts; do not add per-device or per-host branches to lucibridge, voicemode, or speech9p. Tested: speechshim_test 7/7 including AudioRouting and DeviceCapture; speech_wake 4/4; speech_listen 4/4; speech_kokoro 4/4; speech9p_voice 6/6; verify-dis-paths; git diff --cached --check. Not-tested: live remote processing host, Android/phone microphone export, real helper stdin-stream configuration, and full seamless voice UX dogfood.
tools/install-speech-helpers.sh sets up the Phase 1 helper stack on the
host: Homebrew whisper.cpp, a pinned venv with kokoro-onnx and
openwakeword, model downloads, and provider-contract wrappers
(kokoro-cli, whisper-stream-cli, openwakeword-cli). It prints the
/n/speech/ctl block to paste after mount, including the micmode device
stdin-PCM variant. Host smoke coverage in
tests/host/speech_helpers_test.sh exercises the no-mic paths.
The speech contract gains a write-only chime file (wake/done/on/off
earcons) and a duplex ctl option. In half duplex — now the boot default
— speechshim9p keeps capture drained but suppresses wake/STT delivery
while playback or chimes are active, so TTS output can't re-trigger
voice mode. Esc barge-in still works; headset setups can restore spoken
barge-in with duplex full.
voicemode grows a listen timeout (-t) and wake cooldown (-w), parses
partial/final/error listen records, filters whisper silence
hallucinations ("thank you", ...), surfaces live partial transcripts
and errors through the activity context, and defers wake handling while
an approval is pending. lucictx renders a Resources section with
status indicators, and clicking the /n/speech row (or pressing v in
lucifer) toggles voice input mode.
…sh exercises the whole microphone -> STT -> TTS loop with no LLM, no login, and no API key, for validating helper installs and audio topologies without per-turn cost. Headless (default): new appl/cmd/speechtest.b bootstraps speechshim9p + speech9p in its own namespace, prints live partials to the terminal, and answers every non-junk final by speaking a hard-coded phrase (-p) or the transcript itself (-e). -c/-M compose the remote topologies from docs/SPEECH-REMOTE-AUDIO.md. GUI (--gui): boots the full lucifer desktop via the new lib/lucifer/boot-speechtest.sh with voicemode in a new test mode (-p/-e): finals bypass conversation/voiceinput, are shown as a "Heard" dialogue line, and are answered with the canned phrase — wake, live partials, chimes, barge-in and control intents unchanged. boot.sh also gains a $speechhelperbin hook that applies the helper-installer ctl block automatically instead of requiring a manual paste. Tests: tests/speechtest_test.b (5) and three TestMode* cases in tests/voicemode_test.b (17/17).
The shim's streaming helpers (whisper-stream, openwakeword) start lazily on the first listen/wake read — nothing touches the mic at boot — but were never stopped, so after the first voice session the microphone stayed open forever. New `mic on|off` ctl key in speechshim9p: `off` kills the mic-side helpers and the capture pump's device fd, and pending listen/wake reads return "error: mic off" instead of restarting them; the next read re-arms the mic. speech9p forwards the key with the other audio routing knobs, and voicemode writes `mic off` on voice-mode exit. The Voice chip / Esc-V toggle is thus the mic's on/off switch: open only during a voice session, with wake-word turn gating unchanged within it. Tests: speechshim_test MicOffReleasesHelpers (11/11), voicemode_test exit-path mic-off assertion (17/17).
wm/logon skipped the password screen whenever factotum held any key, but lib/sh/profile provisions API-key fallbacks into factotum before logon runs. A configured API key could therefore look like an authenticated desktop session. Set secstoreautounlock only on the genuine SECSTORE_PASSWORD auto-unlock path, and require that marker together with factotum keys before skipping logon. Tested: tests/host/secstore_logon_test.sh (12 checks); appl/wm/logon.b rebuilt with mk.
Remove the buffering filter from the installed whisper wrapper so final records reach InferNode immediately. Preserve carriage-return cleanup in the shell loop, keep helper stderr, and add capture/length controls plus a host relay regression that catches buffered output. Frame helper stdout as newline-delimited records in speechshim9p, retain bounded stderr diagnostics, and add listen off so inactive turns cannot queue ambient or self-speech. Forward the new controls through speech9p, clean up suppressed wake helpers, and preserve multiword wake phrases. Make failures visible through voicemode logging and Voice-chip status, acknowledge chip toggles immediately, report LLM configuration at startup, and auto-detect an installed helper stack during desktop boot. Add tools/speech-regress.sh as the one-command voice regression entry point and include speechtest_test in the test mkfile. Tested: tools/speech-regress.sh (9 suites); microphone-dependent wake/STT smoke skipped outside an interactive TCC-approved session; git diff --cached --check.
Make Lucia voice turns completion-aware and interruptible, with queued streaming TTS, live drafts, voice controls, cancellable approvals, confidence confirmation, and SDL Option-V entry. Add stdin PCM Whisper streaming, namespace-composable remote audio launch scripts, and a loadable SpeechEngine module ABI with provider-backed implementation. Expand the speech regression suite and update the architecture and acceptance documentation. Tested: tools/speech-regress.sh (12 suites); tools/verify-dis-paths.sh; bash and Python syntax checks; macOS SDL build. Microphone-dependent smoke coverage remains an interactive TCC acceptance gate.
Build the tracked stdin-PCM adapter against parakeet.cpp, select the generated helper configuration at boot, and keep Whisper as the soft fallback when the EOU model is unavailable.
Show a short send countdown, append additional finals into the pending utterance, deduplicate repeated finals, and let spoken cancel discard the turn before submission.
Route the compose button and Ctrl+Space through the same input-mode switch as the Voice chip and keyboard shortcuts, removing the duplicate one-shot dictation path.
Default the shim to 22050 Hz so Inferno does not silently fall back to 8000 Hz and stretch Kokoro playback to roughly three times its intended duration.
Retry the canonical model URL and its explicit download form, reject short responses before atomic installation, preserve complete existing models, and cover the behavior with a host-side fake-curl test.
Allow one visible refinement while an activity is busy, reject additional spoken finals until the activity returns idle, and cover the queue cap and reset behavior in the daemon tests.
Make single-host human acceptance the Phase 1 gate and recast Phase 2 around validating and productizing the already-implemented remote provider, audio-routing, deployment, model-distribution, and queue foundations.
Compose Lucia, voicemode, the speech provider stack, llmsrv, and lucibridge against deterministic loopback fixtures. Run the scenario in the blocking speech CI suite and leave only physical audio and GUI input checks for human acceptance.
Merge the current canonical development base into the Phase 1 voice-mode candidate while preserving the resident voice-mode control path and incorporating upstream speech control-name hardening. Regenerate the conflicted Lucia, speech9p, and logon bytecode from the resolved sources.\n\nTested: tools/speech-regress.sh (15 suites); tools/verify-dis-paths.sh; speech9p_security.sh; targeted macOS SDL3 and headless emulator builds; full test bytecode build.\n\nNot-tested: real microphone, audible playback, GUI controls, cross-host audio, and live external Git integration. The repository-wide macOS runner remains blocked by canonical baseline lifecycle and namespace-isolation failures outside the speech subset.
Document that the current anonymous broad-device exports do not preserve InferNode capability attenuation. Make the post-Phase 2 security audit an explicit release gate rather than treating network reachability controls as an internal security boundary.\n\nTested: git diff --check.\n\nNot-tested: remote two-host transport or adversarial-client behaviour; both remain later human and audit gates.
Replay the installer-selected speech control file in both dogfood launchers so Parakeet streaming and Kokoro remain the configured providers. Render live hypotheses and the grace countdown as a bordered unsent conversation turn, keep the Voice resource compact, and preserve a visibly locked keyboard compose buffer until voice mode exits. Add the draft-status UI surface and regression coverage for launcher configuration, daemon state, and pending-turn presentation. Tested: tools/speech-regress.sh (15 suites) Tested: tools/verify-dis-paths.sh Tested: bash -n tools/speech-test.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the Phase 1 release candidate for single-host macOS voice interaction in Lucia.
This introduces provider-backed streaming speech, hands-free voice-mode controls, local speech-helper installation, and supporting tests and documentation.
This is opened as a draft for maintainer review. Automated speech regression is now included in CI; only the normal PR checks and a final interactive macOS smoke pass remain
before merge.
Changes
speech9pandspeechshim9p.voicemodestate machine and unified voice-mode entry through:/voice mode on|offTesting
Completed during implementation:
tools/speech-regress.sh— 14 suites passed on macOStools/verify-dis-paths.shBefore final merge:
Phase boundary
Phase 1 covers usable single-host macOS voice interaction.
Two-host/Jetson deployment, remote-audio productization, model distribution, richer queue management, and other cross-host work remain Phase 2 or later.
Checklist
.disbuild artifacts were committed insideappl/ortests/