Braille Mary is a multimodal communication pipeline for deafblind accessibility. It listens for speech and ASL fingerspelling simultaneously, resolves conflicts with a reasoning model (Claude/OpenRouter), and tactually delivers the exact text through a six-dot braille cell driven by an Arduino/Maker UNO. The goal is to let a deafblind user receive spoken or signed messages without any sighted intermediary.
This is the link for the project demo -> https://vimeo.com/1182343496?share=copy&fl=sv&fe=ci
Key ideas:
• Dual recognizers – Voice input uses Faster-Whisper (English-only) while sign input uses MediaPipe Hands plus a rules-based classifier for common ASL letters. • LLM orchestration – Recognizer outputs and confidence scores are passed to Claude/OpenRouter. The model decides which modality to trust, whether to ask for a repeat, or if helper UI is needed. A deterministic fallback mirrors this logic whenever the LLM is unavailable. • Exact delivery – The orchestrator never paraphrases. Whichever text is chosen is encoded to Grade-1 braille and streamed to the hardware controller. A CLI simulator mimics the tactile output for development.
• Python 3.11+ (virtualenv recommended) • Maker UNO / Arduino-compatible board wired as:
- Dot 1 → D3, Dot 2 → D4, Dot 3 → D5, Dot 4 → D6, Dot 5 → D7, Dot 6 → D9
- Each dot pin drives its actuator through a transistor + flyback diode; grounds must be common. • Optional: microphone and webcam for voice/sign capture. • API key for either Anthropic Claude ( ANTHROPIC_API_KEY ) or OpenRouter ( OPENROUTER_API_KEY ).
bash python -m venv .venv source .venv/bin/activate pip install -r requirements.txt
bash arduino-cli compile --fqbn arduino:avr:uno firmware/braille_controller arduino-cli upload --fqbn arduino:avr:uno -p /dev/ttyUSB0 firmware/braille_controller
Replace /dev/ttyUSB0 with your actual port. The firmware listens for newline-separated SPEED:* and 6-bit braille patterns and toggles the pins above.
Run the diagnostic to pulse every dot for five seconds:
bash BRAILLE_DEBUG=1 python hardware_test.py --port /dev/ttyUSB0 --cycles 1 --hold 5
You should see D3→D4→D5→D6→D7→D9 light in order. Fix wiring issues before continuing.
Set whichever provider you are using:
bash export BRAILLE_PROVIDER=openrouter # or anthropic export OPENROUTER_API_KEY=sk-... # or set ANTHROPIC_API_KEY export BRAILLE_SERIAL_PORT=/dev/ttyUSB0 # optional override
Optional flags:
• BRAILLE_DEBUG=1 – print every serial command sent to the Arduino. • BRAILLE_LETTER_HOLD=5 – override per-letter hold time (seconds) if you want a slower tactile output.
python main.py --mode voice --port /dev/ttyUSB0
python main.py --mode sign --port /dev/ttyUSB0
python main.py --mode both --port /dev/ttyUSB0
python main.py --mode voice --simulate
The CLI will: 1. Capture voice and/or sign for the configured durations. 2. Show recognizer hypotheses and confidences. 3. Send the data to the orchestration layer. High-confidence agreements go straight to delivery; conflicts trigger confirmation/redo logic. 4. Deliver the exact text via the braille hardware (or the simulator). With BRAILLE_DEBUG=1 you can watch the 6-bit patterns and speed commands in real time.
If the LLM fails or refuses to call any tool, the fallback policy automatically selects the highest-confidence modality so the user still receives tactile feedback.
Troubleshooting :
- Check Claude API key
- This will only work via real connection on arduino