Voice Answers for Quizzes (Deepgram)¶

Let seniors answer quiz questions by speaking, with no need to tap the screen. The senior presses one large button, it turns red and shows a pulsing ear icon while it listens, it stops on its own once they finish speaking, and their words are matched to one of the on-screen choices.

This spec covers both quiz surfaces — the Accessible Video Quiz (src/features/video-quiz) and Stories From the Past (src/features/stories) — through a single shared component and hook.

Status: proposed. This is the design the follow-up in docs/video-quiz.md refers to as "Voice commands and voice answers (speech recognition)", and the reason both research schemas already carry input_method: 'tap' | 'voice'.

1. Principles¶

Voice is additive, never required. Tap always remains available. Some seniors have speech differences, dysarthria, a strong accent, or simply sit in a noisy room; voice must never become a barrier. This preserves the project's Born Accessible stance and WCAG 2.1.1 (Keyboard / non-speech operability of all functionality).
No tap to answer. In the happy path the senior taps nothing to submit — they press the listen button once, speak, and the answer is chosen for them. (Pressing the listen button is itself an affordance, not the answer tap.)
The senior is never trapped. Every listening state auto-ends (silence, timeout, or a second press cancels), and any failure falls back silently to the tap buttons that are already on screen.
The API key never ships in the app. Audio is proxied through the existing FastAPI backend, matching the env-driven pattern in backend/app/config.py.
Simple, deterministic matching. Whole-word keyword matching against a small, closed answer set — easy to reason about, easy to unit-test, no fuzzy-model surprises.

2. End-to-end flow¶

Senior presses "Answer by voice"
        │
        ▼
[permission granted?] ── no ──▶ silently keep tap buttons, show a gentle hint
        │ yes
        ▼
Button turns RED, ear icon pulses, "Listening…" announced
        │
   record on-device (expo-audio, metering on)
        │
   senior speaks ── then pauses
        │
[trailing silence ≥ pauseMs]  OR  [maxListenMs reached]  OR  [second press]
        │
        ▼
Stop recording → POST clip to  /transcribe  (FastAPI)
        │
   FastAPI → Deepgram prerecorded API (key server-side) → transcript + words
        │
        ▼
matchChoice(transcript, choices)
        │
   ┌────┴─────────────────────────────┐
   │ single confident match           │  ambiguous / no match
   ▼                                   ▼
auto-submit answer                submit nothing; show "I heard: …",
(input_method:'voice')            re-prompt once, then rest on tap

Recording, silence detection, and matching happen on-device; only the audio bytes and the returned transcript cross the network.

3. The listening button (UX)¶

A new shared component: src/components/voice-answer-button.tsx.

Visual states¶

State	Appearance	Motion
Idle	Primary fill (`colors.primary`), white ear-outline icon + label "Answer by voice"	none
Listening	`colors.danger` (red) fill, white `ear` icon pulsing, label "Listening…"	ear icon loops scale `1.0 → 1.15 → 1.0` and opacity `1.0 → 0.6 → 1.0`, ~900 ms per cycle
Processing	Red fill retained, small activity indicator, label "Thinking…"	indicator only
Error / no match	Returns to idle styling, label "Didn't catch that — try again or tap an answer"	none

Icon: Ionicons ear / ear-outline from @expo/vector-icons (already a dependency). White on danger red clears the 3:1 UI-contrast bar in both themes (danger is #B3261E light / #F2B8B5 dark — verify the icon/label color per theme against docs/accessibility.md; on the dark theme's light-red fill, use a dark icon/label, not white).
Animation: react-native-reanimated (v4.5, already a dependency, with react-native-worklets). Drive the pulse with a withRepeat + withTiming shared value on the ear icon only.
Reduced motion: wrap the pulse duration with useMotionSafeDuration() (src/hooks/use-reduced-motion.ts). When reduced motion is on, the icon does not pulse; the state is conveyed by the red fill, the "Listening…" label, and the screen-reader announcement instead.
Size: full-width, generous height (target ≥ 64 pt tall). The shared Pressable already enforces minHitTarget (44 pt); this control should be visibly larger, in line with the 48 pt playback control precedent.

Accessibility¶

accessibilityRole="button".
accessibilityLabel tracks state: "Answer by voice" → "Listening. Speak your answer now." → "Thinking."
accessibilityHint: "Double tap to start listening. It stops on its own when you finish speaking."
On each transition, call AccessibilityInfo.announceForAccessibility(...) (same pattern the video-quiz screen uses for feedback), so VoiceOver / TalkBack users hear the state change.
Optional light haptic on start/stop (expo-haptics) — nice-to-have, gated on a setting; not required.

4. Recording + auto-stop (mobile)¶

New dependency: expo-audio (the supported recorder for Expo SDK 57; expo-av is deprecated). Add its config plugin in apps/mobile/app.config.ts with a microphone-permission string, which provisions NSMicrophoneUsageDescription (iOS) and RECORD_AUDIO (Android).

New hook: src/features/voice/use-voice-answer.ts. It owns a small state machine and returns { state, start, cancel, result } where state is 'idle' | 'listening' | 'processing' | 'done' | 'error'.

Responsibilities:

Permission — request the mic permission lazily on first start(). On denial, resolve to 'idle' and surface a one-line hint; never block.
Record — start expo-audio recording with metering enabled.
Silence-based auto-stop — poll the recorder's metering level (dBFS) on a short interval and decide when to stop with a pure, injectable function so it is unit-testable:

// src/features/voice/silence.ts
export interface SilenceOptions {
  speechThresholdDb: number; // level above which counts as speech, e.g. -35
  pauseMs: number; // trailing silence that ends the turn
  minSpeechMs: number; // ignore stray taps/coughs before this much speech
  maxListenMs: number; // hard safety cap
}
// Given the running series of (timestampMs, levelDb) samples, return
// whether recording should stop now, and why.
export function shouldStop(
  samples: { tMs: number; levelDb: number }[],
  opts: SilenceOptions,
): { stop: boolean; reason: 'silence' | 'max' | null };

Stop when either (a) the senior has produced ≥ minSpeechMs of speech and then ≥ pauseMs of continuous trailing silence, or (b) maxListenMs elapses.

Upload + match — on stop, read the recorded file, POST it to /transcribe, run matchChoice, resolve result.

The `pauseMs` value (your "200 ms")¶

Your original request was "after 200 ms it should stop listening and send the audio." Taken literally as a total recording length, 200 ms captures no usable answer, so this maps instead to Deepgram's endpointing idea: 200 ms of trailing silence after the senior stops speaking. We agreed to use a gentler default so seniors are not cut off mid-thought:

Constant	Default	Notes
`pauseMs`	1200 ms	Trailing silence that ends the turn. Tunable; 200 ms is the practical floor and is likely too eager for this audience.
`minSpeechMs`	300 ms	Guards against a cough or a stray press ending the turn instantly.
`maxListenMs`	10000 ms	Safety cap so the mic never runs open indefinitely.
`speechThresholdDb`	-35 dBFS	Above this = speech; below = silence. Calibrate on device.

Expose pauseMs (and ideally the whole SilenceOptions) as constants in one place so it can be tuned from a single edit, and later surfaced in Settings if seniors want a longer pause.

5. Matching (very simple, deterministic)¶

New module: src/features/voice/matching.ts.

export interface VoiceMatchOptions {
  /** Extra spoken forms per choice id, e.g. { gifts: ['presents','christmas'] }. */
  aliases?: Record<string, string[]>;
  /** Allow "the first one", "number two", "B" → choice by position. */
  allowPositional?: boolean;
}

export interface VoiceMatch {
  choiceId: string | null;
  reason: 'exact-phrase' | 'keyword' | 'positional' | 'ambiguous' | 'none';
}

export function matchChoice(
  transcript: string,
  choices: { id: string; label: string }[],
  options?: VoiceMatchOptions,
): VoiceMatch;

Algorithm (in order; first decisive rule wins):

Normalize the transcript: lowercase, strip punctuation, collapse whitespace, split into word tokens. Apply the same normalization to each choice label.
Exact phrase — if a choice's full normalized label appears as a substring of the transcript, match it.
Keyword — build a keyword set per choice from its label words minus a small stop-word list (a, an, the, of, and, to, is, are, playing, …), plus any aliases[id]. If exactly one choice has a keyword present as a whole word, match it. Light normalization only: lowercase and simple singular/plural folding (presents ≈ present); no fuzzy edit-distance in v1.
Positional (if allowPositional) — map "first / one / A", "second / two / B", etc. to choice order. Helpful for very short-answer confidence.
Ambiguity guard — if more than one choice matches, return { choiceId: null, reason: 'ambiguous' }. Do not guess.
No match — return { choiceId: null, reason: 'none' }.

Only choiceId !== null auto-submits. ambiguous and none re-prompt once ("I heard '…'. You can say it again, or tap your answer.") and then rely on the visible tap buttons.

Because the answer set is small and closed, we bias Deepgram toward it: the client sends the current choices' key words as recognition hints (see keyterm in §6), which sharply improves transcription of the exact words we need to match.

6. Backend: `/transcribe` endpoint (FastAPI)¶

Follows the existing router/service split and reuses the upload guards.

New router — `backend/app/routers/transcribe.py`¶

router = APIRouter(
    prefix="/transcribe", tags=["transcribe"],
    dependencies=[Depends(reject_oversized_request)],
)

@router.post("")
async def transcribe(
    file: UploadFile,
    hints: Annotated[list[str], Query()] = [],   # choice keywords to bias STT
    language: Annotated[str, Query()] = "en-US",
) -> TranscriptResult:
    audio = await read_upload(file)              # reuses services/uploads.py
    try:
        return await transcribe_audio(audio, file.content_type, hints, language)
    except TranscriptionError as exc:
        raise HTTPException(status_code=502, detail=str(exc)) from exc

Register it in backend/app/main.py alongside health, images, videos. Empty-body (422) and oversize (413) handling come free from read_upload / reject_oversized_request.

New service — `backend/app/services/transcription_service.py`¶

Call Deepgram's prerecorded REST endpoint directly with httpx (already present as a dev dependency; promote it to a runtime dependency in pyproject.toml). Alternatively use the official deepgram-sdk; the REST call keeps the dependency surface minimal.

POST https://api.deepgram.com/v1/listen
  ?model=nova-3&smart_format=true&punctuate=true&language=<language>
  &keyterm=<hint1>&keyterm=<hint2>…
Headers:
  Authorization: Token <DEEPGRAM_API_KEY>
  Content-Type: <audio content-type, e.g. audio/m4a>
Body: raw audio bytes

Return a compact result the client can match on:

class TranscriptResult(BaseModel):
    transcript: str          # channels[0].alternatives[0].transcript
    confidence: float        # channels[0].alternatives[0].confidence
    words: list[str]         # normalized word list, convenience for matching

Model: nova-3 (English). Configurable.
keyterm biases recognition toward the expected answer words — the main lever that makes the closed-set matching reliable.
Timeout: short (e.g. 8 s) via httpx; on timeout or non-2xx, raise TranscriptionError → surfaces as 502 so the app falls back to tap.
No persistence: audio is held in memory only and discarded after the call. Do not write it to disk or logs.

Config — add to `backend/app/config.py`¶

Env var	Default	Purpose
`DEEPGRAM_API_KEY`	(required)	Server-side key; never sent to the app.
`DEEPGRAM_MODEL`	`nova-3`	Deepgram model.
`DEEPGRAM_LANGUAGE`	`en-US`	Default language.
`DEEPGRAM_TIMEOUT_S`	`8`	Upstream request timeout.

Add the key to the backend's environment / compose.yaml and to the Dockerfile runtime env documentation. Fail fast with a clear error at request time if DEEPGRAM_API_KEY is unset.

Client wiring — `apps/mobile/src/lib/api.ts`¶

Extend the api object with a transcribe call that POSTs multipart/form-data (audio file + hints[] + language) to resolveApiBaseUrl() + '/transcribe', reusing the existing ApiError handling. Keep it a thin method next to api.health.

7. Integrating into the two quizzes¶

Both screens keep their existing tap buttons; the voice button sits directly above the choices.

Video quiz — `src/features/video-quiz/video-quiz-screen.tsx`¶

In the question phase, render <VoiceAnswerButton choices={item.choices} onMatched={(choiceId) => answer(choiceId, 'voice')} /> above the mapped choice Buttons.
Thread the input method: answer() currently hard-codes inputMethod: 'tap' in the buildVideoQuizResponse(...) call. Change the signature to answer(choiceId: string, method: InputMethod = 'tap') and pass method through. VideoQuizResponse.input_method already exists.

Stories — `src/features/stories/stories-screen.tsx`¶

Render the same component for choice-format questions.
buildAnswerRecord in scoring.ts already accepts inputMethod; pass 'voice' when a match auto-submits. Reminiscence/open questions have no choices to match, so voice answering is out of scope for them in v1 (they are ungraded anyway).

Optional research fields (needs sign-off)¶

For research value (Lisa Part), consider adding optional, non-breaking fields to the response records: voice_transcript (raw text) and voice_match_confidence (Deepgram confidence). Because the schemas "mirror the approved research schema," treat any schema change as requiring research approval before shipping, and update export.ts (toCsv) columns to match.

8. Accessibility checklist¶

Tap remains a first-class path on every quiz; voice never gates progress.
Reduced-motion honored — no pulse when the OS setting is on.
State changes announced to VoiceOver / TalkBack.
Red listening fill uses the danger token; icon/label contrast verified per theme (§3) against docs/accessibility.md targets (≥ 3:1 for UI/icon, ≥ 4.5:1 for text).
Microphone-permission denial degrades silently to tap.
Text never disables font scaling; the button label scales with Dynamic Type.
Include the voice flow in the VoiceOver (iPhone/iPad) and TalkBack (Android) release passes listed in docs/accessibility.md.

This is a health-adjacent app used by seniors, and recorded voice is sensitive.

Audio is transient: recorded to a temp file on-device, uploaded, and deleted; never stored server-side and never written to logs.
Configure Deepgram for no data retention / no model training on this traffic, and confirm this in the vendor settings, not just in code.
Obtain explicit consent for voice capture and, separately, for any use of transcripts in research. Surface this in onboarding/Settings and gate voice features on it.
Document the vendor (Deepgram) and this data flow in the project's privacy materials; confirm alignment with the study's IRB / consent scope before any voice transcript is used as research data.

10. Testing¶

Mirror the repo's existing test conventions (apps/mobile/tests, backend/tests).

Pure logic (unit):

matching.test.ts — exact phrase; single keyword; alias; plural folding; positional ("the second one" / "B"); ambiguous → null; no match → null; punctuation and casing robustness.
silence.test.ts — shouldStop across sample series: stops after pauseMs of trailing silence only once minSpeechMs is met; stops at maxListenMs; never stops on a lone cough before minSpeechMs.

Backend:

test_transcribe.py — mock the Deepgram HTTP call (e.g. respx or a monkeypatched client): success → TranscriptResult; upstream 4xx/5xx or timeout → 502; empty body → 422; oversize → 413; missing DEEPGRAM_API_KEY → clear error.

Component (React Native Testing Library):

Button renders idle → listening → processing labels; sets the red fill and accessibilityLabel in the listening state; no pulse under reduced motion; emits the accessibility announcements.
Video-quiz and stories screens: a matched transcript auto-submits with input_method: 'voice'; an ambiguous/no-match result submits nothing and leaves the tap buttons operable.

Manual:

Real senior testing in quiet and noisy rooms; varied accents and speech rates; VoiceOver + TalkBack; permission-denied path.

11. Suggested rollout¶

Backend — /transcribe router + service + config + tests. Ship behind the key; verify against a real Deepgram key with sample clips.
Matching — matching.ts + silence.ts with full unit tests (no UI).
Recording hook — use-voice-answer.ts, expo-audio, permissions, app.config.ts plugin.
Button — voice-answer-button.tsx with reanimated pulse + reduced motion + a11y.
Integrate — wire into video-quiz and stories; thread input_method through answer().
Research (gated) — optional transcript/confidence fields + toCsv update, pending research approval.

12. Open decisions¶

Confirm-before-submit? Auto-submitting on a confident single match best serves the "no tap" goal, but showing "I heard: X" for a beat improves research-data trust. Recommendation: auto-submit on a single confident match; show the heard text only on low confidence / ambiguity.
pauseMs final value — start at 1200 ms, tune with real seniors; decide whether to expose it in Settings.
Model/language — nova-3 English by default; confirm whether any seniors need another language before launch.
Positional answers — enable "first/second/A/B"? Simple and forgiving, but only meaningful if choice order is stable on screen.

7. The on-screen voice log¶

Every screen with voice input (Stories and the video quiz) carries a collapsed "Voice log" row near the Speak control. Open it to watch the pipeline live while testing why audio is not being transcribed:

when recording starts, and when the clip is sent to the backend's /transcribe (the Deepgram proxy), including the resolved URL and the hint words for the current choices
what Deepgram heard: the transcript, its confidence, and the recognized words one by one
which answer choice the transcript matched, or that none did
every problem with its actual cause, prefixed "Problem:" in words (a denied microphone permission, a recording that produced no file, the backend's HTTP status, a network failure)

The log lives in memory (src/features/voice/voice-log.ts), keeps the latest 50 entries, and can be cleared from inside the panel. Nothing is persisted or sent anywhere.