fix: mobile tab timeout — idle session reaper + wake lock + visibility reconnect

Backend:
- Session.last_subscriber_left_at: monotonic timestamp set when last SSE subscriber
  leaves, cleared when a new one arrives
- Session.subscriber_count(): replaces len(_subscribers) access from outside the model
- session_store._reaper_loop(): kills sessions with no subscribers for >SESSION_IDLE_TTL_S
  (default 90s); runs every TTL/2 seconds via asyncio.create_task at startup
- session_store._reaper_loop_once(): single-cycle variant for deterministic tests
- app/main.py lifespan: starts reaper on startup, cancels it cleanly on shutdown
- config.py: SESSION_IDLE_TTL_S setting (90s default, overridable per-env)

Frontend:
- useWakeLock.ts: Screen Wake Lock API wrapper; acquires on connect, releases on
  disconnect; degrades silently when unsupported (battery saver, iOS Safari)
- useToneStream.ts: visibilitychange handler — on hidden: closes EventSource without
  ending backend session (grace window stays open); on visible: GET /session/{id}
  liveness check, reconnects SSE + re-acquires wake lock if alive, sets expired=true
  and calls store.reset() if reaped
- ComposeBar.vue: surfaces expired state with calm 'Session timed out' notice
  (not an error — expected behaviour on long screen-off)

Tests:
- test_reaper.py: 7 tests covering subscriber idle tracking, reaper eligibility
  (kills idle, spares active subscriber, spares within-TTL)
This commit is contained in:
pyr0ball 2026-04-11 09:42:42 -07:00
parent 321abe0646
commit 1bc47b8e0f
30 changed files with 4660 additions and 67 deletions

View file

@ -4,11 +4,21 @@ WORKDIR /app
# System deps for audio inference (cf-voice real mode only) # System deps for audio inference (cf-voice real mode only)
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
libsndfile1 \ libsndfile1 git \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# GIT_TOKEN: Forgejo token for private circuitforge-core install.
# Passed at build time via compose.cloud.yml build.args — never baked into a layer.
ARG GIT_TOKEN
# Install public sibling + private circuitforge-core (token consumed here, not cached)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && \
pip install --no-cache-dir \
"git+https://${GIT_TOKEN}@git.opensourcesolarpunk.com/Circuit-Forge/circuitforge-core.git@main"
COPY pyproject.toml . COPY pyproject.toml .
RUN pip install --no-cache-dir -e . RUN pip install --no-cache-dir -e . --no-deps
COPY app/ app/ COPY app/ app/

View file

@ -1,3 +1,3 @@
# linnet # linnet
Real-time tone annotation — Elcor-style emotional/tonal subtext for ND and autistic users Local-first voice transcription with speaker diarization, meeting notes, dictation, and real-time tone annotation — tonal subtext labels for calls, appointments, and conversations

View file

@ -2,15 +2,19 @@
# #
# Receives raw PCM Int16 audio chunks from the browser's AudioWorkletProcessor. # Receives raw PCM Int16 audio chunks from the browser's AudioWorkletProcessor.
# Each message is a binary frame: 16kHz mono Int16 PCM. # Each message is a binary frame: 16kHz mono Int16 PCM.
# The backend accumulates chunks until cf-voice processes them.
# #
# Notation v0.1.x: audio is accepted and acknowledged but inference runs # When CF_VOICE_URL is set (cf-voice sidecar allocated by cf-orch), each chunk
# through the background ContextClassifier (started at session creation), # is base64-encoded and forwarded to cf-voice /classify. The resulting tone
# not inline here. This endpoint is wired for the real audio path # events are broadcast to SSE subscribers via the session store.
# (Navigation v0.2.x) where chunks feed the STT + diarizer directly. #
# When CF_VOICE_URL is unset (local dev / mock mode), chunks are acknowledged
# but not forwarded — the in-process ContextClassifier.stream() generates
# synthetic frames independently.
from __future__ import annotations from __future__ import annotations
import base64
import logging import logging
import time
from fastapi import APIRouter, WebSocket, WebSocketDisconnect from fastapi import APIRouter, WebSocket, WebSocketDisconnect
@ -19,18 +23,19 @@ from app.services import session_store
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
router = APIRouter(prefix="/session", tags=["audio"]) router = APIRouter(prefix="/session", tags=["audio"])
_SESSION_START: dict[str, float] = {}
@router.websocket("/{session_id}/audio") @router.websocket("/{session_id}/audio")
async def audio_ws(websocket: WebSocket, session_id: str) -> None: async def audio_ws(websocket: WebSocket, session_id: str) -> None:
""" """
WebSocket endpoint for binary PCM audio upload. WebSocket endpoint for binary PCM audio upload.
Clients (browser AudioWorkletProcessor) send binary frames. Clients (browser AudioWorkletProcessor) send binary Int16 frames.
Server acknowledges each frame with {"ok": true}. Server acknowledges each frame with {"ok": true, "bytes": N}.
In mock mode (CF_VOICE_MOCK=1) the session's ContextClassifier generates When CF_VOICE_URL is configured, each chunk is forwarded to the cf-voice
synthetic frames independently -- audio sent here is accepted but not sidecar and the resulting tone events are broadcast to SSE subscribers.
processed. Real inference wiring happens in Navigation v0.2.x.
""" """
session = session_store.get_session(session_id) session = session_store.get_session(session_id)
if session is None: if session is None:
@ -38,12 +43,17 @@ async def audio_ws(websocket: WebSocket, session_id: str) -> None:
return return
await websocket.accept() await websocket.accept()
_SESSION_START[session_id] = time.monotonic()
logger.info("Audio WS connected for session %s", session_id) logger.info("Audio WS connected for session %s", session_id)
try: try:
while True: while True:
data = await websocket.receive_bytes() data = await websocket.receive_bytes()
# Notation v0.1.x: acknowledge receipt; real inference in v0.2.x timestamp = time.monotonic() - _SESSION_START.get(session_id, 0.0)
await websocket.send_json({"ok": True, "bytes": len(data)}) await websocket.send_json({"ok": True, "bytes": len(data)})
# Forward to cf-voice sidecar (no-op if CF_VOICE_URL is unset)
audio_b64 = base64.b64encode(data).decode()
await session_store.forward_audio_chunk(session, audio_b64, timestamp)
except WebSocketDisconnect: except WebSocketDisconnect:
logger.info("Audio WS disconnected for session %s", session_id) logger.info("Audio WS disconnected for session %s", session_id)
_SESSION_START.pop(session_id, None)

5
app/api/corrections.py Normal file
View file

@ -0,0 +1,5 @@
# app/api/corrections.py — tone annotation correction submissions
from circuitforge_core.api import make_corrections_router
from app.db import get_db
router = make_corrections_router(get_db=get_db, product="linnet")

71
app/api/samples.py Normal file
View file

@ -0,0 +1,71 @@
# app/api/samples.py — Demo text samples for model testing via the imitate tab
#
# Returns pre-curated sentences with varied emotional subtext so that
# external tools (e.g. avocet imitate tab) can feed them to local LLMs
# and evaluate tone annotation quality without a live audio session.
from __future__ import annotations
from fastapi import APIRouter
router = APIRouter(prefix="/samples", tags=["samples"])
# Each sample has a text field (the raw utterance) and a context field
# describing the situation — both fed into the imitate prompt template.
_SAMPLES: list[dict] = [
{
"text": "Sure, I can take care of that.",
"context": "Employee responding to manager's request during a tense project meeting.",
"category": "compliance",
},
{
"text": "That's a really interesting approach.",
"context": "Colleague reacting to a proposal they visibly disagree with.",
"category": "polite_disagreement",
},
{
"text": "I'll have it done by end of day.",
"context": "Developer already working at capacity, second deadline added without discussion.",
"category": "overcommitment",
},
{
"text": "No, it's fine. Don't worry about it.",
"context": "Person whose suggestion was dismissed without acknowledgment.",
"category": "suppressed_hurt",
},
{
"text": "I guess we could try it your way.",
"context": "Team lead conceding after strong pushback, despite having a valid concern.",
"category": "reluctant_agreement",
},
{
"text": "That's one way to look at it.",
"context": "Researcher responding to a peer who has misread their data.",
"category": "implicit_correction",
},
{
"text": "I appreciate you letting me know.",
"context": "Employee informed their project is being cancelled with two hours notice.",
"category": "formal_distress",
},
{
"text": "We should probably circle back on this.",
"context": "Manager avoiding a decision they don't want to make.",
"category": "deferral",
},
{
"text": "I just want to make sure I'm understanding correctly.",
"context": "Autistic person who has been given contradictory instructions twice already.",
"category": "clarification_exhaustion",
},
{
"text": "Happy to help however I can!",
"context": "Volunteer who has already put in 60 hours this month responding to another ask.",
"category": "masking_burnout",
},
]
@router.get("")
def list_samples(limit: int = 5) -> list[dict]:
"""Return a slice of demo annotation samples for model testing."""
return _SAMPLES[:max(1, min(limit, len(_SAMPLES)))]

View file

@ -22,7 +22,7 @@ class SessionResponse(BaseModel):
@router.post("/start", response_model=SessionResponse) @router.post("/start", response_model=SessionResponse)
async def start_session(req: StartRequest = StartRequest()) -> SessionResponse: async def start_session(req: StartRequest = StartRequest()) -> SessionResponse:
"""Start a new annotation session and begin streaming VoiceFrames.""" """Start a new annotation session and begin streaming VoiceFrames."""
session = session_store.create_session(elcor=req.elcor) session = await session_store.create_session(elcor=req.elcor)
return SessionResponse( return SessionResponse(
session_id=session.session_id, session_id=session.session_id,
state=session.state, state=session.state,
@ -33,7 +33,7 @@ async def start_session(req: StartRequest = StartRequest()) -> SessionResponse:
@router.delete("/{session_id}/end") @router.delete("/{session_id}/end")
async def end_session(session_id: str) -> dict: async def end_session(session_id: str) -> dict:
"""Stop a session and release its classifier.""" """Stop a session and release its classifier."""
removed = session_store.end_session(session_id) removed = await session_store.end_session(session_id)
if not removed: if not removed:
raise HTTPException(status_code=404, detail=f"Session {session_id} not found") raise HTTPException(status_code=404, detail=f"Session {session_id} not found")
return {"session_id": session_id, "state": "stopped"} return {"session_id": session_id, "state": "stopped"}

View file

@ -15,6 +15,12 @@ class Settings:
# DEMO: auto-kill sessions after this many seconds of inactivity # DEMO: auto-kill sessions after this many seconds of inactivity
demo_session_ttl_s: int = int(os.getenv("DEMO_SESSION_TTL_S", "300")) # 5 min demo_session_ttl_s: int = int(os.getenv("DEMO_SESSION_TTL_S", "300")) # 5 min
# All modes: kill a session this many seconds after its last SSE subscriber
# disconnects. Covers mobile tab timeout / screen lock / crash.
# Set generously enough to survive a brief screen-off without losing the session,
# but short enough that zombie sessions don't accumulate.
session_idle_ttl_s: int = int(os.getenv("SESSION_IDLE_TTL_S", "90"))
# CLOUD: where Caddy injects the cf_session user token # CLOUD: where Caddy injects the cf_session user token
cloud_session_header: str = os.getenv("CLOUD_SESSION_HEADER", "X-CF-Session") cloud_session_header: str = os.getenv("CLOUD_SESSION_HEADER", "X-CF-Session")
cloud_data_root: str = os.getenv("CLOUD_DATA_ROOT", "/devl/linnet-cloud-data") cloud_data_root: str = os.getenv("CLOUD_DATA_ROOT", "/devl/linnet-cloud-data")
@ -25,5 +31,17 @@ class Settings:
linnet_frontend_port: int = int(os.getenv("LINNET_FRONTEND_PORT", "8521")) linnet_frontend_port: int = int(os.getenv("LINNET_FRONTEND_PORT", "8521"))
linnet_base_url: str = os.getenv("LINNET_BASE_URL", "") linnet_base_url: str = os.getenv("LINNET_BASE_URL", "")
# cf-orch coordinator URL — used to request a managed cf-voice instance per session.
# When set, each session allocates a cf-voice sidecar via the coordinator REST API.
cf_orch_url: str = os.getenv("CF_ORCH_URL", "")
# Static cf-voice sidecar URL — legacy override (bypasses cf-orch, points directly
# at a running cf-voice process). Takes precedence over cf-orch allocation when set.
cf_voice_url: str = os.getenv("CF_VOICE_URL", "")
# Local SQLite DB for corrections storage. Shared across users on a single instance;
# corrections contain text only (no audio). Cloud mode uses cloud_data_root prefix.
linnet_db: str = os.getenv("LINNET_DB", "data/linnet.db")
settings = Settings() settings = Settings()

37
app/db.py Normal file
View file

@ -0,0 +1,37 @@
# app/db.py — SQLite connection and migration runner for Linnet
#
# Used only for corrections storage (tone annotation training data).
# Session state is in-memory; this DB persists user-submitted corrections.
from __future__ import annotations
import sqlite3
from collections.abc import Iterator
from pathlib import Path
from circuitforge_core.db import run_migrations
from app.config import settings
_MIGRATIONS_DIR = Path(__file__).parent / "migrations"
def _db_path() -> Path:
path = Path(settings.linnet_db)
path.parent.mkdir(parents=True, exist_ok=True)
return path
def get_connection() -> sqlite3.Connection:
conn = sqlite3.connect(str(_db_path()), check_same_thread=False)
conn.execute("PRAGMA journal_mode=WAL")
run_migrations(conn, _MIGRATIONS_DIR)
return conn
def get_db() -> Iterator[sqlite3.Connection]:
"""FastAPI dependency — yields a connection, closes on teardown."""
conn = get_connection()
try:
yield conn
finally:
conn.close()

View file

@ -3,22 +3,35 @@ from __future__ import annotations
import logging import logging
import os import os
from contextlib import asynccontextmanager
from fastapi import FastAPI from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from app.api import audio, events, export, history, sessions from app.api import audio, corrections, events, export, history, samples, sessions
from app.config import settings from app.config import settings
from app.services import session_store
logging.basicConfig( logging.basicConfig(
level=logging.INFO, level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s%(message)s", format="%(asctime)s %(levelname)s %(name)s%(message)s",
) )
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup: begin idle session reaper
session_store.start_reaper()
yield
# Shutdown: cancel the reaper cleanly
await session_store.stop_reaper()
app = FastAPI( app = FastAPI(
title="Linnet", title="Linnet",
description="Real-time tone annotation — Elcor-style subtext for ND/autistic users", description="Real-time tone annotation — tonal subtext labels for ND/autistic users",
version="0.1.0", version="0.1.0",
lifespan=lifespan,
) )
# ── Mode middleware (applied before CORS so headers are always present) ────── # ── Mode middleware (applied before CORS so headers are always present) ──────
@ -58,6 +71,8 @@ app.include_router(events.router)
app.include_router(history.router) app.include_router(history.router)
app.include_router(audio.router) app.include_router(audio.router)
app.include_router(export.router) app.include_router(export.router)
app.include_router(samples.router)
app.include_router(corrections.router, prefix="/corrections", tags=["corrections"])
@app.get("/health") @app.get("/health")

View file

@ -0,0 +1,21 @@
-- Migration 001: Corrections table for tone annotation training data.
-- Users can rate annotations (up/down) and submit corrected versions.
-- Only opted_in=1 rows are exported to the Avocet SFT pipeline.
CREATE TABLE IF NOT EXISTS corrections (
id INTEGER PRIMARY KEY AUTOINCREMENT,
item_id TEXT NOT NULL DEFAULT '', -- session_id or event_id
product TEXT NOT NULL DEFAULT 'linnet',
correction_type TEXT NOT NULL DEFAULT 'annotation',
input_text TEXT NOT NULL, -- the utterance that was annotated
original_output TEXT NOT NULL, -- the LLM annotation produced
corrected_output TEXT NOT NULL DEFAULT '', -- user's correction (empty = thumbs up)
rating TEXT NOT NULL DEFAULT 'down', -- 'up' | 'down'
context TEXT NOT NULL DEFAULT '{}', -- JSON: session_id, model, elcor flag, etc.
opted_in INTEGER NOT NULL DEFAULT 0, -- 1 = user consented to share for training
created_at TEXT NOT NULL DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_corrections_product ON corrections (product);
CREATE INDEX IF NOT EXISTS idx_corrections_opted_in ON corrections (opted_in);
CREATE INDEX IF NOT EXISTS idx_corrections_item_id ON corrections (item_id);

32
app/models/queue_event.py Normal file
View file

@ -0,0 +1,32 @@
# app/models/queue_event.py — call queue state and environment events
from __future__ import annotations
import json
from dataclasses import dataclass
@dataclass
class QueueEvent:
"""
Call queue state or environmental classification from cf-voice AST.
event_type is either "queue" or "environ".
Broadcasts via SSE as `event: queue-event` or `event: environ-event`.
Not stored in session history.
"""
session_id: str
event_type: str # "queue" or "environ"
label: str # e.g. "hold_music", "ringback", "call_center", "quiet"
confidence: float
timestamp: float
def to_sse(self) -> str:
payload = {
"event_type": self.event_type,
"session_id": self.session_id,
"label": self.label,
"confidence": self.confidence,
"timestamp": self.timestamp,
}
sse_name = "queue-event" if self.event_type == "queue" else "environ-event"
return f"event: {sse_name}\ndata: {json.dumps(payload)}\n\n"

View file

@ -5,13 +5,19 @@ import asyncio
import time import time
import uuid import uuid
from dataclasses import dataclass, field from dataclasses import dataclass, field
from typing import Literal from typing import Literal, Protocol, runtime_checkable
from app.models.tone_event import ToneEvent from app.models.tone_event import ToneEvent
SessionState = Literal["starting", "running", "stopped"] SessionState = Literal["starting", "running", "stopped"]
@runtime_checkable
class SessionEvent(Protocol):
"""Any event that can be broadcast over SSE from a session."""
def to_sse(self) -> str: ...
@dataclass @dataclass
class Session: class Session:
""" """
@ -26,14 +32,30 @@ class Session:
created_at: float = field(default_factory=time.monotonic) created_at: float = field(default_factory=time.monotonic)
elcor: bool = False elcor: bool = False
# cf-voice sidecar — populated by session_store._allocate_voice() when cf-orch
# is configured. Empty string means in-process fallback is active.
cf_voice_url: str = ""
cf_voice_allocation_id: str = ""
# History: last 50 events retained for GET /session/{id}/history # History: last 50 events retained for GET /session/{id}/history
history: list[ToneEvent] = field(default_factory=list) history: list[ToneEvent] = field(default_factory=list)
_subscribers: list[asyncio.Queue] = field(default_factory=list, repr=False) _subscribers: list[asyncio.Queue] = field(default_factory=list, repr=False)
# Idle tracking: monotonic timestamp of when the last subscriber left.
# None means at least one subscriber is currently active, or the session
# has never had one yet (just started). The reaper uses this to detect
# abandoned sessions after a mobile tab timeout.
last_subscriber_left_at: float | None = field(default=None, repr=False)
def subscriber_count(self) -> int:
return len(self._subscribers)
def subscribe(self) -> asyncio.Queue: def subscribe(self) -> asyncio.Queue:
"""Add an SSE subscriber. Returns its dedicated queue.""" """Add an SSE subscriber. Returns its dedicated queue."""
q: asyncio.Queue = asyncio.Queue(maxsize=100) q: asyncio.Queue = asyncio.Queue(maxsize=100)
self._subscribers.append(q) self._subscribers.append(q)
# Reset idle clock — someone is watching again
self.last_subscriber_left_at = None
return q return q
def unsubscribe(self, q: asyncio.Queue) -> None: def unsubscribe(self, q: asyncio.Queue) -> None:
@ -41,9 +63,13 @@ class Session:
self._subscribers.remove(q) self._subscribers.remove(q)
except ValueError: except ValueError:
pass pass
# Start the idle clock when the last subscriber leaves
if not self._subscribers:
self.last_subscriber_left_at = time.monotonic()
def broadcast(self, event: ToneEvent) -> None: def broadcast(self, event: SessionEvent) -> None:
"""Fan out a ToneEvent to all current SSE subscribers.""" """Fan out any SessionEvent (tone, speaker, etc.) to all SSE subscribers."""
if isinstance(event, ToneEvent):
if len(self.history) >= 50: if len(self.history) >= 50:
self.history.pop(0) self.history.pop(0)
self.history.append(event) self.history.append(event)

View file

@ -0,0 +1,29 @@
# app/models/speaker_event.py — speaker classification event
from __future__ import annotations
import json
from dataclasses import dataclass
@dataclass
class SpeakerEvent:
"""
Speaker/environment classification from cf-voice.
Broadcasts via SSE as `event: speaker-event` alongside tone-events.
Not stored in session history (tone history only).
"""
session_id: str
label: str # e.g. "human", "no_speaker", "ivr_synth"
confidence: float
timestamp: float
def to_sse(self) -> str:
payload = {
"event_type": "speaker",
"session_id": self.session_id,
"label": self.label,
"confidence": self.confidence,
"timestamp": self.timestamp,
}
return f"event: speaker-event\ndata: {json.dumps(payload)}\n\n"

View file

@ -0,0 +1,29 @@
# app/models/transcript_event.py — live STT transcript event
from __future__ import annotations
import json
from dataclasses import dataclass
@dataclass
class TranscriptEvent:
"""
Live transcription segment from cf-voice Whisper STT.
Broadcasts via SSE as `event: transcript-event`. Not stored in history.
The label field carries the raw transcript text.
"""
session_id: str
text: str
speaker_id: str # diarization label, e.g. "SPEAKER_00" or "speaker_a"
timestamp: float
def to_sse(self) -> str:
payload = {
"event_type": "transcript",
"session_id": self.session_id,
"text": self.text,
"speaker_id": self.speaker_id,
"timestamp": self.timestamp,
}
return f"event: transcript-event\ndata: {json.dumps(payload)}\n\n"

View file

@ -8,16 +8,22 @@ from cf_voice.models import VoiceFrame
from app.models.tone_event import ToneEvent from app.models.tone_event import ToneEvent
def annotate(frame: VoiceFrame, session_id: str, elcor: bool = False) -> ToneEvent | None: def annotate(
frame: VoiceFrame,
session_id: str,
elcor: bool = False,
threshold: float = 0.10,
) -> ToneEvent | None:
""" """
Convert a VoiceFrame into a session ToneEvent. Convert a VoiceFrame into a session ToneEvent.
Returns None if the frame is below the reliability threshold (confidence Returns None if the frame is below the reliability threshold. The default
too low to annotate confidently). Callers should skip None results. 0.25 is calibrated for real wav2vec2 SER inference, which routinely scores
0.15-0.45 on conversational audio. The mock used 0.6 (mock confidence was
artificially inflated).
elcor=True switches subtext to Mass Effect Elcor prefix format. elcor=True switches subtext to bracketed tone-prefix format (easter egg).
This is an easter egg -- do not pass elcor=True by default.
""" """
if not frame.is_reliable(): if not frame.is_reliable(threshold=threshold):
return None return None
return ToneEvent.from_voice_frame(frame, session_id=session_id, elcor=elcor) return ToneEvent.from_voice_frame(frame, session_id=session_id, elcor=elcor)

View file

@ -3,9 +3,9 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import time
from cf_voice.context import ContextClassifier from app.config import settings
from app.models.session import Session from app.models.session import Session
from app.models.tone_event import ToneEvent from app.models.tone_event import ToneEvent
from app.services.annotator import annotate from app.services.annotator import annotate
@ -15,19 +15,37 @@ logger = logging.getLogger(__name__)
# Module-level singleton store — one per process # Module-level singleton store — one per process
_sessions: dict[str, Session] = {} _sessions: dict[str, Session] = {}
_tasks: dict[str, asyncio.Task] = {} _tasks: dict[str, asyncio.Task] = {}
_reaper_task: asyncio.Task | None = None
# Audio accumulation buffer per session.
# We accumulate 100ms PCM chunks until we have CLASSIFY_WINDOW_MS of audio,
# then fire a single /classify call. Real emotion models need ≥500ms context.
_CLASSIFY_WINDOW_MS = 1000 # ms of audio per classify call; wav2vec2 needs ≥1s context
_CHUNK_MS = 100 # AudioWorklet sends 1600 samples @ 16kHz = 100ms
_CHUNKS_PER_WINDOW = _CLASSIFY_WINDOW_MS // _CHUNK_MS # 10 chunks
_audio_buffers: dict[str, list[bytes]] = {}
def create_session(elcor: bool = False) -> Session: async def create_session(elcor: bool = False) -> Session:
"""Create a new session and start its ContextClassifier background task.""" """Create a new session and start its ContextClassifier background task.
If CF_ORCH_URL is configured, requests a managed cf-voice instance before
starting the classifier. Falls back to in-process mock if allocation fails.
"""
session = Session(elcor=elcor) session = Session(elcor=elcor)
_sessions[session.session_id] = session _sessions[session.session_id] = session
await _allocate_voice(session)
task = asyncio.create_task( task = asyncio.create_task(
_run_classifier(session), _run_classifier(session),
name=f"classifier-{session.session_id}", name=f"classifier-{session.session_id}",
) )
_tasks[session.session_id] = task _tasks[session.session_id] = task
session.state = "running" session.state = "running"
logger.info("Session %s started", session.session_id) logger.info(
"Session %s started (voice=%s)",
session.session_id,
session.cf_voice_url or "in-process",
)
return session return session
@ -40,7 +58,7 @@ def active_session_count() -> int:
return sum(1 for s in _sessions.values() if s.state == "running") return sum(1 for s in _sessions.values() if s.state == "running")
def end_session(session_id: str) -> bool: async def end_session(session_id: str) -> bool:
"""Stop and remove a session. Returns True if it existed.""" """Stop and remove a session. Returns True if it existed."""
session = _sessions.pop(session_id, None) session = _sessions.pop(session_id, None)
if session is None: if session is None:
@ -49,17 +67,83 @@ def end_session(session_id: str) -> bool:
task = _tasks.pop(session_id, None) task = _tasks.pop(session_id, None)
if task and not task.done(): if task and not task.done():
task.cancel() task.cancel()
_audio_buffers.pop(session_id, None)
await _release_voice(session)
logger.info("Session %s ended", session_id) logger.info("Session %s ended", session_id)
return True return True
async def _allocate_voice(session: Session) -> None:
"""Request a managed cf-voice instance from cf-orch. No-op if CF_ORCH_URL is unset."""
# Static override takes precedence — skip cf-orch allocation
if settings.cf_voice_url:
session.cf_voice_url = settings.cf_voice_url
return
if not settings.cf_orch_url:
return
import httpx
url = settings.cf_orch_url.rstrip("/") + "/api/services/cf-voice/allocate"
try:
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.post(url, json={"caller": "linnet", "ttl_s": 3600.0})
resp.raise_for_status()
data = resp.json()
session.cf_voice_url = data["url"]
session.cf_voice_allocation_id = data["allocation_id"]
logger.info(
"cf-orch allocated cf-voice for session %s: %s (alloc=%s)",
session.session_id, data["url"], data["allocation_id"],
)
except Exception as exc:
logger.warning(
"cf-orch allocation failed for session %s: %s — falling back to in-process",
session.session_id, exc,
)
async def _release_voice(session: Session) -> None:
"""Release a cf-voice allocation back to cf-orch. No-op if not orch-managed."""
if not session.cf_voice_allocation_id or not settings.cf_orch_url:
return
import httpx
url = (
settings.cf_orch_url.rstrip("/")
+ f"/api/services/cf-voice/allocations/{session.cf_voice_allocation_id}"
)
try:
async with httpx.AsyncClient(timeout=5.0) as client:
await client.delete(url)
logger.info("Released cf-voice allocation for session %s", session.session_id)
except Exception as exc:
logger.warning(
"cf-orch release failed for session %s: %s", session.session_id, exc
)
async def _run_classifier(session: Session) -> None: async def _run_classifier(session: Session) -> None:
""" """
Background task: stream VoiceFrames from cf-voice and broadcast ToneEvents. Background task: stream VoiceFrames and broadcast ToneEvents.
Starts the ContextClassifier (mock or real depending on CF_VOICE_MOCK), Two modes:
converts each frame via annotator.annotate(), and fans out to subscribers. - Sidecar (cf-orch allocated or CF_VOICE_URL set): holds the session open
while audio chunks are forwarded to cf-voice via forward_audio_chunk().
- In-process (no sidecar): runs ContextClassifier.from_env() directly.
Used for local dev and mock mode when CF_ORCH_URL is unset or allocation failed.
""" """
if session.cf_voice_url:
await _run_classifier_sidecar(session)
else:
await _run_classifier_inprocess(session)
async def _run_classifier_inprocess(session: Session) -> None:
"""In-process path: ContextClassifier.stream() — used when CF_VOICE_URL is unset."""
from cf_voice.context import ContextClassifier
classifier = ContextClassifier.from_env() classifier = ContextClassifier.from_env()
try: try:
async for frame in classifier.stream(): async for frame in classifier.stream():
@ -74,3 +158,199 @@ async def _run_classifier(session: Session) -> None:
await classifier.stop() await classifier.stop()
session.state = "stopped" session.state = "stopped"
logger.info("Classifier stopped for session %s", session.session_id) logger.info("Classifier stopped for session %s", session.session_id)
async def _run_classifier_sidecar(session: Session) -> None:
"""
Sidecar path: wait for audio chunks forwarded via forward_audio_chunk().
The sidecar does not self-generate frames audio arrives from the browser
WebSocket and is sent to cf-voice /classify. This task holds the session
open and handles cleanup.
"""
try:
while session.state == "running":
await asyncio.sleep(1.0)
except asyncio.CancelledError:
pass
finally:
session.state = "stopped"
logger.info("Sidecar session ended for session %s", session.session_id)
async def forward_audio_chunk(
session: Session,
audio_b64: str,
timestamp: float,
) -> None:
"""
Accumulate PCM chunks into 500ms windows, then forward to cf-voice /classify.
The AudioWorklet sends 100ms chunks; we buffer 10 of them (_CHUNKS_PER_WINDOW)
before firing a classify call. This gives wav2vec2 a full second of context
and keeps the classify rate at 1/s instead of 10/s.
No-op when no cf-voice sidecar is allocated (in-process path handles its own input).
"""
voice_url = session.cf_voice_url
if not voice_url:
return
import base64 as _b64
import httpx
from app.models.speaker_event import SpeakerEvent
from cf_voice.models import VoiceFrame
# Decode incoming chunk and append to per-session buffer
raw = _b64.b64decode(audio_b64)
buf = _audio_buffers.setdefault(session.session_id, [])
buf.append(raw)
if len(buf) < _CHUNKS_PER_WINDOW:
return # not enough audio yet — wait for more chunks
# Flush: concatenate window, reset buffer
window_bytes = b"".join(buf)
buf.clear()
window_b64 = _b64.b64encode(window_bytes).decode()
url = voice_url.rstrip("/") + "/classify"
payload = {
"audio_chunk": window_b64,
"timestamp": timestamp,
"elcor": session.elcor,
"session_id": session.session_id,
}
try:
async with httpx.AsyncClient(timeout=5.0) as client:
resp = await client.post(url, json=payload)
resp.raise_for_status()
data = resp.json()
except Exception as exc:
logger.warning("cf-voice sidecar call failed for session %s: %s", session.session_id, exc)
return
from app.models.queue_event import QueueEvent
from app.models.transcript_event import TranscriptEvent
for ev in data.get("events", []):
etype = ev.get("event_type")
if etype == "tone":
frame = VoiceFrame(
label=ev["label"],
confidence=ev["confidence"],
speaker_id=ev.get("speaker_id", ""),
shift_magnitude=ev.get("shift_magnitude", 0.0),
timestamp=ev["timestamp"],
)
tone = annotate(frame, session_id=session.session_id, elcor=session.elcor)
if tone is not None:
session.broadcast(tone)
elif etype == "speaker":
speaker = SpeakerEvent(
session_id=session.session_id,
label=ev["label"],
confidence=ev.get("confidence", 1.0),
timestamp=ev["timestamp"],
)
session.broadcast(speaker)
elif etype == "transcript":
transcript = TranscriptEvent(
session_id=session.session_id,
text=ev["label"],
speaker_id=ev.get("speaker_id", "speaker_a"),
timestamp=ev["timestamp"],
)
session.broadcast(transcript)
elif etype in ("queue", "environ"):
queue_ev = QueueEvent(
session_id=session.session_id,
event_type=etype,
label=ev["label"],
confidence=ev.get("confidence", 1.0),
timestamp=ev["timestamp"],
)
session.broadcast(queue_ev)
# ── Idle session reaper ───────────────────────────────────────────────────────
async def _reaper_loop() -> None:
"""
Periodically kill sessions with no active SSE subscribers.
A session becomes eligible for reaping when:
- Its last SSE subscriber disconnected more than SESSION_IDLE_TTL_S seconds ago
- It has not been explicitly ended (state != "stopped")
This covers the common mobile pattern: screen locks browser suspends tab
EventSource closes SSE subscriber count drops to zero. If the user doesn't
return within the TTL window, the session is cleaned up automatically.
The reaper runs every REAP_INTERVAL_S seconds (half the TTL, so the worst-case
overshoot is TTL + REAP_INTERVAL_S).
"""
ttl = settings.session_idle_ttl_s
interval = max(15, ttl // 2)
logger.info("Session reaper started (TTL=%ds, check every %ds)", ttl, interval)
while True:
await asyncio.sleep(interval)
now = time.monotonic()
to_reap = [
sid
for sid, session in list(_sessions.items())
if (
session.state == "running"
and session.subscriber_count() == 0
and session.last_subscriber_left_at is not None
and (now - session.last_subscriber_left_at) > ttl
)
]
for sid in to_reap:
logger.info(
"Reaping idle session %s (no subscribers for >%ds)", sid, ttl
)
await end_session(sid)
async def _reaper_loop_once() -> None:
"""Single reaper pass — used by tests to avoid sleeping."""
ttl = settings.session_idle_ttl_s
now = time.monotonic()
to_reap = [
sid
for sid, session in list(_sessions.items())
if (
session.state == "running"
and session.subscriber_count() == 0
and session.last_subscriber_left_at is not None
and (now - session.last_subscriber_left_at) > ttl
)
]
for sid in to_reap:
logger.info("Reaping idle session %s (no subscribers for >%ds)", sid, ttl)
await end_session(sid)
def start_reaper() -> None:
"""Start the idle session reaper background task. Called from app lifespan."""
global _reaper_task
if _reaper_task is None or _reaper_task.done():
_reaper_task = asyncio.create_task(_reaper_loop(), name="session-reaper")
async def stop_reaper() -> None:
"""Cancel the reaper task cleanly. Called from app lifespan shutdown."""
global _reaper_task
if _reaper_task and not _reaper_task.done():
_reaper_task.cancel()
try:
await _reaper_task
except asyncio.CancelledError:
pass
_reaper_task = None

View file

@ -17,6 +17,8 @@ services:
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
args:
GIT_TOKEN: ${GIT_TOKEN:-}
restart: unless-stopped restart: unless-stopped
env_file: .env env_file: .env
environment: environment:
@ -31,6 +33,8 @@ services:
volumes: volumes:
- /devl/linnet-cloud-data:/devl/linnet-cloud-data - /devl/linnet-cloud-data:/devl/linnet-cloud-data
- ${HOME}/.config/circuitforge:/root/.config/circuitforge:ro - ${HOME}/.config/circuitforge:/root/.config/circuitforge:ro
# Live-reload app code without rebuilding during active dev
- ./app:/app/app:ro
networks: networks:
- linnet-cloud-net - linnet-cloud-net

View file

@ -1,12 +1,21 @@
services: services:
linnet-api: linnet-api:
build: . build:
context: .
dockerfile: Dockerfile
args:
GIT_TOKEN: ${GIT_TOKEN:-}
ports: ports:
- "${LINNET_PORT:-8522}:8522" - "${LINNET_PORT:-8522}:8522"
env_file: env_file:
- .env - .env
environment: environment:
CF_VOICE_MOCK: "${CF_VOICE_MOCK:-1}" CF_VOICE_MOCK: "${CF_VOICE_MOCK:-1}"
volumes:
# Live-reload app code without rebuilding the image in dev
- ./app:/app/app:ro
extra_hosts:
- "host.docker.internal:host-gateway"
restart: unless-stopped restart: unless-stopped
linnet-frontend: linnet-frontend:

BIN
data/linnet.db Normal file

Binary file not shown.

3148
frontend/package-lock.json generated Normal file

File diff suppressed because it is too large Load diff

View file

@ -19,15 +19,27 @@
<span class="session-id">{{ store.sessionId?.slice(0, 8) }}</span> <span class="session-id">{{ store.sessionId?.slice(0, 8) }}</span>
</div> </div>
<label class="elcor-toggle" title="Elcor subtext mode"> <label class="elcor-toggle" title="Tone prefix mode">
<input type="checkbox" v-model="elcorLocal" disabled /> <input type="checkbox" v-model="elcorLocal" disabled />
Elcor Prefix
</label> </label>
<button
class="btn-mic"
:class="{ active: capturing }"
@click="handleMicToggle"
:aria-label="capturing ? 'Stop microphone' : 'Start microphone'"
>
{{ capturing ? "Mic on" : "Mic off" }}
</button>
<button class="btn-stop" @click="handleStop">End session</button> <button class="btn-stop" @click="handleStop">End session</button>
</template> </template>
<p v-if="error" class="compose-error">{{ error }}</p> <p v-if="expired" class="compose-notice">
Session timed out after inactivity. Start a new one to continue.
</p>
<p v-else-if="error" class="compose-error">{{ error }}</p>
</div> </div>
</template> </template>
@ -35,9 +47,11 @@
import { ref } from "vue"; import { ref } from "vue";
import { useSessionStore } from "../stores/session"; import { useSessionStore } from "../stores/session";
import { useToneStream } from "../composables/useToneStream"; import { useToneStream } from "../composables/useToneStream";
import { useAudioCapture } from "../composables/useAudioCapture";
const store = useSessionStore(); const store = useSessionStore();
const { connect, disconnect } = useToneStream(); const { connect, disconnect, expired } = useToneStream();
const { start: startMic, stop: stopMic, capturing } = useAudioCapture();
const starting = ref(false); const starting = ref(false);
const error = ref<string | null>(null); const error = ref<string | null>(null);
@ -57,9 +71,23 @@ async function handleStart() {
} }
async function handleStop() { async function handleStop() {
if (capturing.value) stopMic();
disconnect(); disconnect();
await store.endSession(); await store.endSession();
} }
async function handleMicToggle() {
error.value = null;
try {
if (capturing.value) {
stopMic();
} else {
await startMic(store.sessionId!);
}
} catch (e: unknown) {
error.value = e instanceof Error ? e.message : "Mic access failed";
}
}
</script> </script>
<script lang="ts"> <script lang="ts">
@ -121,10 +149,34 @@ export default { name: "ComposeBar" };
opacity: 0.5; opacity: 0.5;
} }
.btn-mic {
padding: 0.5rem 1rem;
border-radius: 0.5rem;
font-size: 0.875rem;
font-weight: 500;
cursor: pointer;
border: 1px solid var(--color-border, #2a2d3a);
background: transparent;
color: var(--color-muted, #6b7280);
transition: background 0.15s, color 0.15s, border-color 0.15s;
}
.btn-mic.active {
background: #dc262622;
border-color: #dc2626;
color: #dc2626;
}
.compose-error { .compose-error {
width: 100%; width: 100%;
font-size: 0.75rem; font-size: 0.75rem;
color: #ef4444; color: #ef4444;
margin: 0; margin: 0;
} }
.compose-notice {
width: 100%;
font-size: 0.75rem;
color: var(--color-muted, #6b7280);
margin: 0;
}
</style> </style>

View file

@ -0,0 +1,316 @@
<template>
<div class="correction-widget" :class="{ 'correction-widget--expanded': expanded }">
<!-- Idle: thumbs row -->
<div v-if="state === 'idle'" class="rating-row">
<span class="rating-label">Was this annotation accurate?</span>
<button
type="button"
class="thumb-btn thumb-btn--up"
aria-label="Accurate"
@click="rate('up')"
>👍</button>
<button
type="button"
class="thumb-btn thumb-btn--down"
aria-label="Inaccurate — submit correction"
@click="rate('down')"
>👎</button>
</div>
<!-- Thumbs-down: correction form -->
<form v-else-if="state === 'correcting'" class="correction-form" @submit.prevent="submit">
<label class="field-label" for="corrected-output">
What should the annotation say?
</label>
<textarea
id="corrected-output"
v-model="correctedOutput"
class="correction-textarea"
rows="3"
placeholder="e.g. [RELUCTANTLY] [MASKING_STRESS] I'll have it done by end of day."
required
autofocus
/>
<label class="optin-label">
<input v-model="optedIn" type="checkbox" class="optin-check" />
Share this correction anonymously to improve the model
</label>
<div class="form-actions">
<button type="button" class="btn-cancel" @click="cancel">Cancel</button>
<button
type="submit"
class="btn-submit"
:disabled="submitting || !correctedOutput.trim()"
>
{{ submitting ? 'Saving…' : 'Submit correction' }}
</button>
</div>
<p v-if="error" class="correction-error" role="alert">{{ error }}</p>
</form>
<!-- Thumbs-up: optional praise form -->
<form v-else-if="state === 'praising'" class="correction-form" @submit.prevent="submitPraise">
<label class="field-label" for="praise-input">
What worked well? <span class="field-optional">(optional)</span>
</label>
<textarea
id="praise-input"
v-model="praiseText"
class="correction-textarea"
rows="2"
placeholder="e.g. Caught the sarcasm without over-labeling it."
autofocus
/>
<div class="form-actions">
<button type="button" class="btn-cancel" @click="submitPraise">Skip</button>
<button type="submit" class="btn-submit" :disabled="submitting">
{{ submitting ? 'Saving…' : 'Submit' }}
</button>
</div>
<p v-if="error" class="correction-error" role="alert">{{ error }}</p>
</form>
<!-- Thumbs-up submitted -->
<div v-else-if="state === 'done-up'" class="done-msg done-msg--up">
Thanks glad that was helpful.
</div>
<!-- Correction submitted -->
<div v-else-if="state === 'done-down'" class="done-msg done-msg--down">
Correction saved{{ optedIn ? ' and flagged for training' : '' }}.
</div>
</div>
</template>
<script setup lang="ts">
import { ref, computed } from 'vue'
const props = defineProps<{
itemId?: string
inputText: string
originalOutput: string
correctionType?: string
apiPath?: string
context?: Record<string, unknown>
}>()
type State = 'idle' | 'praising' | 'correcting' | 'done-up' | 'done-down'
const state = ref<State>('idle')
const correctedOutput = ref('')
const praiseText = ref('')
const optedIn = ref(false)
const submitting = ref(false)
const error = ref<string | null>(null)
const expanded = computed(() => state.value === 'correcting' || state.value === 'praising')
const apiBase = import.meta.env.VITE_API_BASE ?? "";
const endpoint = computed(() => `${apiBase}${props.apiPath ?? '/corrections'}`)
function rate(rating: 'up' | 'down') {
if (rating === 'up') {
state.value = 'praising'
} else {
state.value = 'correcting'
}
}
function cancel() {
state.value = 'idle'
correctedOutput.value = ''
praiseText.value = ''
error.value = null
}
async function submit() {
await submitRating('down', correctedOutput.value.trim())
}
async function submitPraise() {
const praise = praiseText.value.trim()
const context = { ...props.context ?? {}, ...(praise ? { praise } : {}) }
await submitRating('up', '', context)
}
async function submitRating(
rating: 'up' | 'down',
correction: string,
contextOverride?: Record<string, unknown>,
) {
submitting.value = true
error.value = null
// Thumbs-up rows contain no user text auto-consent so they export for training.
const effectiveOptIn = rating === 'up' ? true : optedIn.value
const context = contextOverride ?? props.context ?? {}
try {
const res = await fetch(endpoint.value, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
item_id: props.itemId ?? '',
product: 'linnet',
correction_type: props.correctionType ?? 'annotation',
input_text: props.inputText,
original_output: props.originalOutput,
corrected_output: correction,
rating,
context,
opted_in: effectiveOptIn,
}),
})
if (!res.ok) {
const body = await res.json().catch(() => ({}))
throw new Error(body?.detail ?? `HTTP ${res.status}`)
}
state.value = rating === 'up' ? 'done-up' : 'done-down'
} catch (err) {
error.value = err instanceof Error ? err.message : 'Something went wrong — please try again.'
} finally {
submitting.value = false
}
}
</script>
<style scoped>
.correction-widget {
display: flex;
flex-direction: column;
gap: var(--space-2, 0.5rem);
padding: var(--space-2, 0.5rem) var(--space-3, 0.75rem);
border-top: 1px solid var(--color-border, #2a2d3a);
font-size: 0.8rem;
color: var(--color-muted, #6b7280);
transition: padding 0.15s ease;
}
.correction-widget--expanded {
padding: var(--space-3, 0.75rem);
}
/* Rating row */
.rating-row {
display: flex;
align-items: center;
gap: var(--space-2, 0.5rem);
}
.rating-label {
flex: 1;
font-size: 0.75rem;
color: var(--color-muted, #6b7280);
}
.thumb-btn {
background: none;
border: 1px solid var(--color-border, #2a2d3a);
border-radius: var(--radius-sm, 4px);
padding: 0.2rem 0.45rem;
font-size: 1rem;
cursor: pointer;
line-height: 1;
transition: background 0.1s ease, border-color 0.1s ease;
}
.thumb-btn:hover {
background: var(--color-surface, #1a1d27);
border-color: var(--color-accent, #7c6af7);
}
/* Correction form */
.correction-form {
display: flex;
flex-direction: column;
gap: var(--space-2, 0.5rem);
}
.field-label {
font-size: 0.75rem;
font-weight: 600;
color: var(--color-text, #e2e8f0);
}
.field-optional {
font-weight: 400;
color: var(--color-muted, #6b7280);
}
.correction-textarea {
width: 100%;
resize: vertical;
background: var(--color-bg, #0f1117);
border: 1px solid var(--color-border, #2a2d3a);
border-radius: var(--radius-sm, 4px);
color: var(--color-text, #e2e8f0);
font-family: var(--font-mono, monospace);
font-size: 0.78rem;
padding: var(--space-2, 0.5rem);
box-sizing: border-box;
}
.correction-textarea:focus {
outline: none;
border-color: var(--color-accent, #7c6af7);
}
.optin-label {
display: flex;
align-items: center;
gap: var(--space-2, 0.5rem);
font-size: 0.72rem;
color: var(--color-muted, #6b7280);
cursor: pointer;
}
.optin-check {
accent-color: var(--color-accent, #7c6af7);
}
.form-actions {
display: flex;
gap: var(--space-2, 0.5rem);
justify-content: flex-end;
}
.btn-cancel {
background: none;
border: 1px solid var(--color-border, #2a2d3a);
border-radius: var(--radius-sm, 4px);
color: var(--color-muted, #6b7280);
font-size: 0.78rem;
padding: var(--space-1, 0.25rem) var(--space-3, 0.75rem);
cursor: pointer;
}
.btn-submit {
background: var(--color-accent, #7c6af7);
border: none;
border-radius: var(--radius-sm, 4px);
color: #fff;
font-size: 0.78rem;
font-weight: 600;
padding: var(--space-1, 0.25rem) var(--space-3, 0.75rem);
cursor: pointer;
transition: opacity 0.1s ease;
}
.btn-submit:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.correction-error {
font-size: 0.72rem;
color: var(--color-error, #dc2626);
margin: 0;
}
/* Done states */
.done-msg {
font-size: 0.75rem;
font-weight: 600;
}
.done-msg--up { color: var(--color-success, #16a34a); }
.done-msg--down { color: var(--color-accent, #7c6af7); }
</style>

View file

@ -1,5 +1,24 @@
<template> <template>
<div class="now-panel" :data-affect="affect"> <div class="now-panel" :data-affect="affect">
<!-- Row 1: speaker type + queue state badges -->
<div class="now-badges">
<div v-if="speakerLabel" class="now-speaker" :data-speaker="speakerKind">
{{ speakerLabel }}
</div>
<div v-if="queueLabel" class="now-queue-badge" :data-queue="queueKind">
{{ queueLabel }}
</div>
<div v-if="environLabel" class="now-environ-badge" :data-environ="environKind">
{{ environLabel }}
</div>
</div>
<!-- Live transcript strip -->
<div v-if="transcriptText" class="now-transcript">
"{{ transcriptText }}"
</div>
<!-- Tone annotation -->
<div class="now-label">{{ label }}</div> <div class="now-label">{{ label }}</div>
<div v-if="subtext" class="now-subtext">{{ subtext }}</div> <div v-if="subtext" class="now-subtext">{{ subtext }}</div>
<div class="now-meta"> <div class="now-meta">
@ -14,16 +33,29 @@
<span v-else>~</span> <span v-else>~</span>
shift {{ (shiftMagnitude * 100).toFixed(0) }}% shift {{ (shiftMagnitude * 100).toFixed(0) }}%
</div> </div>
<CorrectionWidget
v-if="props.event"
:item-id="props.event.session_id"
:input-text="''"
:original-output="annotationOutput"
correction-type="annotation"
api-path="/corrections"
:context="{ session_id: props.event.session_id, label: props.event.label }"
/>
</div> </div>
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import type { ToneEvent } from "../stores/session"; import type { ToneEvent } from "../stores/session";
import { useSessionStore } from "../stores/session";
import CorrectionWidget from "./CorrectionWidget.vue";
const props = defineProps<{ const props = defineProps<{
event: ToneEvent | null; event: ToneEvent | null;
}>(); }>();
const store = useSessionStore();
const label = computed(() => props.event?.label ?? "—"); const label = computed(() => props.event?.label ?? "—");
const confidence = computed(() => props.event?.confidence ?? 0); const confidence = computed(() => props.event?.confidence ?? 0);
const subtext = computed(() => props.event?.subtext ?? null); const subtext = computed(() => props.event?.subtext ?? null);
@ -31,6 +63,62 @@ const affect = computed(() => props.event?.affect ?? "neutral");
const prosodyFlags = computed(() => props.event?.prosody_flags ?? []); const prosodyFlags = computed(() => props.event?.prosody_flags ?? []);
const shiftMagnitude = computed(() => props.event?.shift_magnitude ?? 0); const shiftMagnitude = computed(() => props.event?.shift_magnitude ?? 0);
const shiftDirection = computed(() => props.event?.shift_direction ?? "stable"); const shiftDirection = computed(() => props.event?.shift_direction ?? "stable");
const annotationOutput = computed(() => {
if (!props.event) return "";
const parts = [`[${props.event.label}]`];
if (props.event.subtext) parts.push(props.event.subtext);
return parts.join(" ");
});
// Speaker badge hidden when no_speaker or no session
const SPEAKER_LABELS: Record<string, string> = {
human_single: "Human",
human_multi: "Group",
ivr_synth: "IVR",
no_speaker: "",
transfer: "Transfer",
};
const speakerLabel = computed(() => {
const raw = store.currentSpeaker?.label;
if (!raw) return null;
const mapped = SPEAKER_LABELS[raw] ?? raw;
return mapped || null;
});
const speakerKind = computed(() => store.currentSpeaker?.label ?? "");
// Queue state badge (hold_music, ringback, silence, etc.)
const QUEUE_LABELS: Record<string, string> = {
hold_music: "On Hold",
ringback: "Ringing",
busy: "Busy",
dtmf_tone: "DTMF",
silence: "Silence",
dead_air: "Dead Air",
};
const queueLabel = computed(() => {
const raw = store.currentQueue?.label;
if (!raw || raw === "silence") return null;
return QUEUE_LABELS[raw] ?? raw;
});
const queueKind = computed(() => store.currentQueue?.label ?? "");
// Environment badge (call_center, music, noise, etc.)
const ENVIRON_LABELS: Record<string, string> = {
call_center: "Call Centre",
music: "Music",
background_shift: "Shift",
noise_floor_change: "Noise",
quiet: "",
};
const environLabel = computed(() => {
const raw = store.currentEnviron?.label;
if (!raw || raw === "quiet") return null;
return ENVIRON_LABELS[raw] ?? raw;
});
const environKind = computed(() => store.currentEnviron?.label ?? "");
// Live transcript strip
const transcriptText = computed(() => store.currentTranscript?.text ?? null);
</script> </script>
<script lang="ts"> <script lang="ts">
@ -60,6 +148,64 @@ export default { name: "NowPanel" };
.now-panel[data-affect="dismissive"] { border-color: #a78bfa44; } .now-panel[data-affect="dismissive"] { border-color: #a78bfa44; }
.now-panel[data-affect="urgent"] { border-color: #fbbf2444; } .now-panel[data-affect="urgent"] { border-color: #fbbf2444; }
.now-badges {
display: flex;
flex-wrap: wrap;
gap: 0.4rem;
align-items: center;
min-height: 1.4rem;
}
/* Shared pill base for speaker / queue / environ badges */
.now-speaker,
.now-queue-badge,
.now-environ-badge {
display: inline-block;
font-size: 0.65rem;
font-family: var(--font-mono, monospace);
text-transform: uppercase;
letter-spacing: 0.08em;
padding: 0.15em 0.6em;
border-radius: 999px;
background: var(--color-border, #2a2d3a);
color: var(--color-muted, #6b7280);
border: 1px solid transparent;
transition: background 0.3s, color 0.3s;
}
/* Speaker type colours */
.now-speaker[data-speaker="human_single"] { background: #1e3a2f; color: #34d399; border-color: #34d39933; }
.now-speaker[data-speaker="human_multi"] { background: #1e3a2f; color: #34d399; border-color: #34d39933; }
.now-speaker[data-speaker="ivr_synth"] { background: #1e2a3a; color: #60a5fa; border-color: #60a5fa33; }
.now-speaker[data-speaker="transfer"] { background: #2a2a1e; color: #fbbf24; border-color: #fbbf2433; }
/* Queue state colours */
.now-queue-badge[data-queue="hold_music"] { background: #2a1e3a; color: #a78bfa; border-color: #a78bfa33; }
.now-queue-badge[data-queue="ringback"] { background: #2a2a1e; color: #fbbf24; border-color: #fbbf2433; }
.now-queue-badge[data-queue="busy"] { background: #3a1e1e; color: #f87171; border-color: #f8717133; }
.now-queue-badge[data-queue="dtmf_tone"] { background: #1e2a3a; color: #60a5fa; border-color: #60a5fa33; }
.now-queue-badge[data-queue="dead_air"] { background: #1e1e1e; color: #4b5563; border-color: #4b556333; }
/* Environment colours */
.now-environ-badge[data-environ="call_center"] { background: #1e2a3a; color: #60a5fa; border-color: #60a5fa22; }
.now-environ-badge[data-environ="music"] { background: #2a1e3a; color: #a78bfa; border-color: #a78bfa22; }
.now-environ-badge[data-environ="background_shift"] { background: #2a2a1e; color: #fbbf24; border-color: #fbbf2422; }
.now-environ-badge[data-environ="noise_floor_change"] { background: #2a1e1e; color: #f87171; border-color: #f8717122; }
/* Live transcript strip */
.now-transcript {
font-size: 0.8rem;
color: var(--color-muted, #6b7280);
font-style: italic;
border-left: 2px solid var(--color-border, #2a2d3a);
padding-left: 0.6rem;
line-height: 1.4;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
max-width: 100%;
}
.now-label { .now-label {
font-size: 1.35rem; font-size: 1.35rem;
font-weight: 600; font-weight: 600;

View file

@ -21,12 +21,13 @@ export function useAudioCapture() {
stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false }); stream = await navigator.mediaDevices.getUserMedia({ audio: true, video: false });
ctx = new AudioContext({ sampleRate: 16000 }); ctx = new AudioContext({ sampleRate: 16000 });
await ctx.audioWorklet.addModule("/audio-processor.js"); await ctx.audioWorklet.addModule(`${import.meta.env.BASE_URL}audio-processor.js`);
source = ctx.createMediaStreamSource(stream); source = ctx.createMediaStreamSource(stream);
const processor = new AudioWorkletNode(ctx, "pcm-processor"); const processor = new AudioWorkletNode(ctx, "pcm-processor");
const wsUrl = `${location.protocol === "https:" ? "wss" : "ws"}://${location.host}/session/${sessionId}/audio`; const apiBase = import.meta.env.VITE_API_BASE ?? "";
const wsUrl = `${location.protocol === "https:" ? "wss" : "ws"}://${location.host}${apiBase}/session/${sessionId}/audio`;
ws = new WebSocket(wsUrl); ws = new WebSocket(wsUrl);
ws.binaryType = "arraybuffer"; ws.binaryType = "arraybuffer";

View file

@ -1,43 +1,117 @@
/** /**
* useToneStream manages the SSE connection for a session's tone-event stream. * useToneStream SSE connection + wake lock + visibility reconnect.
* *
* Usage: * On page hide (screen lock, tab switch): closes EventSource without ending
* const { connect, disconnect, connected } = useToneStream(); * the session. The backend reaper gives the session a 90s grace window.
* connect(sessionId); // opens EventSource *
* disconnect(); // closes it * On page show: re-acquires the wake lock, checks whether the session is
* still alive on the server, then either reconnects SSE or surfaces an
* "expired" flag so the UI can prompt the user to start a new session.
*/ */
import { ref, onUnmounted } from "vue"; import { ref, onUnmounted } from "vue";
import { useSessionStore } from "../stores/session"; import {
useSessionStore,
type SpeakerEvent,
type TranscriptEvent,
type QueueEvent,
} from "../stores/session";
import { useWakeLock } from "./useWakeLock";
export function useToneStream() { export function useToneStream() {
const connected = ref(false); const connected = ref(false);
const expired = ref(false); // true when the session was reaped while hidden
let source: EventSource | null = null; let source: EventSource | null = null;
let activeSessionId: string | null = null;
const store = useSessionStore(); const store = useSessionStore();
const wakeLock = useWakeLock();
function connect(sessionId: string) { // ── SSE wiring ──────────────────────────────────────────────────────────────
if (source) disconnect();
source = new EventSource(`/session/${sessionId}/stream`); function _attachListeners() {
if (!source) return;
source.addEventListener("tone-event", (e: MessageEvent) => { source.addEventListener("tone-event", (e: MessageEvent) => {
try { try { store.pushEvent(JSON.parse(e.data)); } catch { /* malformed */ }
const evt = JSON.parse(e.data); });
store.pushEvent(evt); source.addEventListener("speaker-event", (e: MessageEvent) => {
} catch { try { store.updateSpeaker(JSON.parse(e.data) as SpeakerEvent); } catch { /* */ }
// malformed frame — ignore });
} source.addEventListener("transcript-event", (e: MessageEvent) => {
try { store.updateTranscript(JSON.parse(e.data) as TranscriptEvent); } catch { /* */ }
});
source.addEventListener("queue-event", (e: MessageEvent) => {
try { store.updateQueue(JSON.parse(e.data) as QueueEvent); } catch { /* */ }
});
source.addEventListener("environ-event", (e: MessageEvent) => {
try { store.updateQueue(JSON.parse(e.data) as QueueEvent); } catch { /* */ }
}); });
source.onopen = () => { connected.value = true; }; source.onopen = () => { connected.value = true; };
source.onerror = () => { connected.value = false; }; source.onerror = () => { connected.value = false; };
} }
function disconnect() { function connect(sessionId: string) {
if (source) _closeSource();
activeSessionId = sessionId;
expired.value = false;
const apiBase = import.meta.env.VITE_API_BASE ?? "";
source = new EventSource(`${apiBase}/session/${sessionId}/stream`);
_attachListeners();
wakeLock.acquire();
}
function _closeSource() {
source?.close(); source?.close();
source = null; source = null;
connected.value = false; connected.value = false;
} }
onUnmounted(disconnect); function disconnect() {
_closeSource();
return { connect, disconnect, connected }; activeSessionId = null;
wakeLock.release();
}
// ── Visibility handling ─────────────────────────────────────────────────────
async function _handleVisibilityChange() {
if (document.visibilityState === "hidden") {
// Close the SSE connection without ending the backend session.
// The reaper gives us SESSION_IDLE_TTL_S (default 90s) before cleanup.
_closeSource();
return;
}
// Page became visible again — check if session survived the absence.
if (!activeSessionId) return;
const apiBase = import.meta.env.VITE_API_BASE ?? "";
try {
const resp = await fetch(`${apiBase}/session/${activeSessionId}`);
if (resp.ok) {
// Session still alive — reconnect SSE and wake lock
const apiBase2 = import.meta.env.VITE_API_BASE ?? "";
source = new EventSource(`${apiBase2}/session/${activeSessionId}/stream`);
_attachListeners();
wakeLock.acquire();
} else {
// Session was reaped (404) or errored — surface to UI
expired.value = true;
store.reset();
activeSessionId = null;
}
} catch {
// Network error on return — don't immediately expire, user may be offline
// briefly. They can retry manually.
}
}
document.addEventListener("visibilitychange", _handleVisibilityChange);
onUnmounted(() => {
disconnect();
document.removeEventListener("visibilitychange", _handleVisibilityChange);
});
return { connect, disconnect, connected, expired };
} }

View file

@ -0,0 +1,57 @@
/**
* useWakeLock Screen Wake Lock API wrapper.
*
* Prevents the screen from dimming/sleeping while an annotation session is
* active. The browser *always* releases the lock on page hide (tab switch,
* screen lock), so we re-acquire on visibilitychange: visible.
*
* Degrades silently when the API is unavailable (Firefox < 126, iOS Safari,
* low-power mode). The session still works; screen may time out.
*/
import { ref, onUnmounted } from "vue";
export function useWakeLock() {
const supported = "wakeLock" in navigator;
const active = ref(false);
let lock: WakeLockSentinel | null = null;
async function acquire() {
if (!supported) return;
try {
lock = await navigator.wakeLock.request("screen");
active.value = true;
lock.addEventListener("release", () => {
active.value = false;
lock = null;
});
} catch {
// Denied: battery saver, low-power mode, permissions policy.
// Silent fail — the session continues without it.
active.value = false;
}
}
function release() {
lock?.release();
lock = null;
active.value = false;
}
// Re-acquire after returning to the tab — the browser releases the lock
// automatically on page hide, so we must request it again on page show.
async function handleVisibility() {
if (document.visibilityState === "visible" && lock === null && active.value === false) {
// Only re-acquire if we were previously active (i.e. a session is running).
// The caller controls this by only calling acquire() when a session starts.
}
}
// NOTE: the caller is responsible for calling acquire() when a session starts
// and release() when it ends. The visibility re-acquire is handled inside
// the SSE composable which coordinates both wake lock and reconnect together.
onUnmounted(release);
return { supported, active, acquire, release, handleVisibility };
}

View file

@ -15,15 +15,45 @@ export interface ToneEvent {
prosody_flags: string[]; prosody_flags: string[];
} }
export interface SpeakerEvent {
event_type: "speaker";
session_id: string;
label: string;
confidence: number;
timestamp: number;
}
export interface TranscriptEvent {
event_type: "transcript";
session_id: string;
text: string;
speaker_id: string;
timestamp: number;
}
export interface QueueEvent {
event_type: "queue" | "environ";
session_id: string;
label: string;
confidence: number;
timestamp: number;
}
export const useSessionStore = defineStore("session", () => { export const useSessionStore = defineStore("session", () => {
const sessionId = ref<string | null>(null); const sessionId = ref<string | null>(null);
const elcor = ref(false); const elcor = ref(false);
const state = ref<"idle" | "running" | "stopped">("idle"); const state = ref<"idle" | "running" | "stopped">("idle");
const events = ref<ToneEvent[]>([]); const events = ref<ToneEvent[]>([]);
const latest = computed(() => events.value.at(-1) ?? null); const latest = computed(() => events.value.at(-1) ?? null);
const currentSpeaker = ref<SpeakerEvent | null>(null);
const currentTranscript = ref<TranscriptEvent | null>(null);
const currentQueue = ref<QueueEvent | null>(null);
const currentEnviron = ref<QueueEvent | null>(null);
const apiBase = import.meta.env.VITE_API_BASE ?? "";
async function startSession(withElcor = false) { async function startSession(withElcor = false) {
const resp = await fetch("/session/start", { const resp = await fetch(`${apiBase}/session/start`, {
method: "POST", method: "POST",
headers: { "Content-Type": "application/json" }, headers: { "Content-Type": "application/json" },
body: JSON.stringify({ elcor: withElcor }), body: JSON.stringify({ elcor: withElcor }),
@ -38,7 +68,7 @@ export const useSessionStore = defineStore("session", () => {
async function endSession() { async function endSession() {
if (!sessionId.value) return; if (!sessionId.value) return;
await fetch(`/session/${sessionId.value}/end`, { method: "DELETE" }); await fetch(`${apiBase}/session/${sessionId.value}/end`, { method: "DELETE" });
state.value = "stopped"; state.value = "stopped";
} }
@ -48,12 +78,38 @@ export const useSessionStore = defineStore("session", () => {
if (events.value.length > 200) events.value.shift(); if (events.value.length > 200) events.value.shift();
} }
function updateSpeaker(evt: SpeakerEvent) {
currentSpeaker.value = evt;
}
function updateTranscript(evt: TranscriptEvent) {
currentTranscript.value = evt;
}
function updateQueue(evt: QueueEvent) {
if (evt.event_type === "queue") {
currentQueue.value = evt;
} else {
currentEnviron.value = evt;
}
}
function reset() { function reset() {
sessionId.value = null; sessionId.value = null;
state.value = "idle"; state.value = "idle";
events.value = []; events.value = [];
elcor.value = false; elcor.value = false;
currentSpeaker.value = null;
currentTranscript.value = null;
currentQueue.value = null;
currentEnviron.value = null;
} }
return { sessionId, elcor, state, events, latest, startSession, endSession, pushEvent, reset }; return {
sessionId, elcor, state, events, latest,
currentSpeaker, currentTranscript, currentQueue, currentEnviron,
startSession, endSession, pushEvent,
updateSpeaker, updateTranscript, updateQueue,
reset,
};
}); });

View file

@ -3,8 +3,10 @@ import vue from "@vitejs/plugin-vue";
import UnoCSS from "unocss/vite"; import UnoCSS from "unocss/vite";
export default defineConfig({ export default defineConfig({
base: process.env.VITE_BASE_URL ?? "/",
plugins: [vue(), UnoCSS()], plugins: [vue(), UnoCSS()],
server: { server: {
host: "0.0.0.0",
port: 8521, port: 8521,
proxy: { proxy: {
"/session": { "/session": {
@ -16,6 +18,10 @@ export default defineConfig({
target: "http://localhost:8522", target: "http://localhost:8522",
changeOrigin: true, changeOrigin: true,
}, },
"/corrections": {
target: "http://localhost:8522",
changeOrigin: true,
},
}, },
}, },
}); });

11
requirements.txt Normal file
View file

@ -0,0 +1,11 @@
# Install local sibling packages from Forgejo before the linnet package itself.
# cf-voice is public. circuitforge-core is private — token injected at build time
# via GIT_TOKEN build arg (see Dockerfile + compose.cloud.yml).
git+https://git.opensourcesolarpunk.com/Circuit-Forge/cf-voice.git@main
# Runtime deps (mirrors pyproject.toml — keep in sync)
fastapi>=0.111
uvicorn[standard]>=0.29
websockets>=12.0
python-dotenv>=1.0
httpx>=0.27

124
tests/test_reaper.py Normal file
View file

@ -0,0 +1,124 @@
# tests/test_reaper.py — idle session reaper and subscriber idle tracking
from __future__ import annotations
import asyncio
import time
import pytest
from app.models.session import Session
from app.services import session_store
@pytest.fixture(autouse=True)
def clean_sessions():
"""Ensure the session store is empty before and after each test."""
session_store._sessions.clear()
session_store._tasks.clear()
session_store._audio_buffers.clear()
yield
session_store._sessions.clear()
session_store._tasks.clear()
session_store._audio_buffers.clear()
# ── Session model: subscriber idle tracking ───────────────────────────────────
def test_last_subscriber_left_at_none_on_start():
s = Session()
assert s.last_subscriber_left_at is None
def test_last_subscriber_left_at_set_on_unsubscribe():
s = Session()
q = s.subscribe()
assert s.last_subscriber_left_at is None # someone is watching
s.unsubscribe(q)
assert s.last_subscriber_left_at is not None
def test_last_subscriber_left_at_cleared_on_resubscribe():
s = Session()
q = s.subscribe()
s.unsubscribe(q)
assert s.last_subscriber_left_at is not None
# New subscriber arrives — idle clock resets
q2 = s.subscribe()
assert s.last_subscriber_left_at is None
s.unsubscribe(q2)
def test_subscriber_count():
s = Session()
assert s.subscriber_count() == 0
q1 = s.subscribe()
q2 = s.subscribe()
assert s.subscriber_count() == 2
s.unsubscribe(q1)
assert s.subscriber_count() == 1
s.unsubscribe(q2)
assert s.subscriber_count() == 0
# ── Reaper logic ──────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_reaper_kills_idle_session(monkeypatch):
"""
A session with no subscribers for longer than the TTL should be reaped.
We monkeypatch SESSION_IDLE_TTL_S to 0 and manually invoke _reaper_loop
for a single cycle.
"""
monkeypatch.setattr(session_store.settings, "session_idle_ttl_s", 0)
session = await session_store.create_session()
sid = session.session_id
# Simulate a subscriber connecting then leaving
q = session.subscribe()
session.unsubscribe(q)
# Force last_subscriber_left_at into the past
session.last_subscriber_left_at = time.monotonic() - 1.0
assert sid in session_store._sessions
# Run one reaper cycle directly
await session_store._reaper_loop_once()
assert sid not in session_store._sessions
@pytest.mark.asyncio
async def test_reaper_spares_session_with_active_subscriber(monkeypatch):
"""A session that still has an active SSE subscriber must not be reaped."""
monkeypatch.setattr(session_store.settings, "session_idle_ttl_s", 0)
session = await session_store.create_session()
sid = session.session_id
# Subscriber is still connected — idle clock is None
_q = session.subscribe()
assert session.last_subscriber_left_at is None
await session_store._reaper_loop_once()
assert sid in session_store._sessions
session.unsubscribe(_q)
@pytest.mark.asyncio
async def test_reaper_spares_session_within_ttl(monkeypatch):
"""A session that lost its subscriber recently (within TTL) must survive."""
monkeypatch.setattr(session_store.settings, "session_idle_ttl_s", 3600)
session = await session_store.create_session()
sid = session.session_id
q = session.subscribe()
session.unsubscribe(q)
# last_subscriber_left_at is just now — well within TTL
await session_store._reaper_loop_once()
assert sid in session_store._sessions