rdbms-playground/docs/adr/0057-nfr-verification-strategy.md

ADR-0057: Non-functional-requirement verification strategy

Status

Accepted (2026-06-22).

Context

The non-functional requirements (requirements.md NFR-1..7 — startup, input latency, memory, distinctive design, colour use, cross-platform parity, light/dark legibility) were quality bars that had never been formally verified, even though v0.2.0 ships public binaries. With users now pulling updates, we want each NFR either gated by an automated test (so a regression fails CI) or measured-and-documented with a recorded figure and a reviewer judgement — not left as an aspiration.

The decision below was taken with the user: contrast is a hard gate; startup and memory are measured against generous bounds (not the literal target — a tight timing/RSS gate would flake on a shared runner); the qualitative bars are verified by argument and reviewer note.

Decision

NFR-5 / NFR-7 — colour contrast & legibility: gated

Unit tests in src/theme.rs (Tier 1, run by the existing gate):

• all_text_colours_meet_wcag_aa_contrast — every text-bearing foreground (fg, muted, mode labels, system/error/warning, plan_efficient, and all tok_*) must clear WCAG-AA 4.5:1 against bg, in both themes.
• advanced_mode_border_meets_ui_contrast — the advanced-mode border carries a mode-warning signal, so it meets the 3:1 non-text / UI-component threshold. The plain border is decorative structural chrome and is deliberately exempt (it sits at 2.20:1 dark / 1.82:1 light by design, so panel borders recede behind content — recorded here rather than silently tolerated).
• syntax_token_colours_are_perceptually_distinct — contrast against the background is necessary but not sufficient: two token colours can each clear 4.5:1 yet be hard to tell apart on a good screen. Every pair of syntax-token classes that can share a line must clear CIEDE2000 (ΔE2000) ≥ 15 (post-tuning minimum is ~18). The metric is itself validated against the Sharma et al. reference vectors (delta_e_2000_matches_reference_vectors).

Computing the real ratios while writing these tests uncovered two shipped defects in the v0.2.0 light theme — tok_string at 4.42:1 and tok_flag at 3.15:1, both below the 4.5:1 the scheme promises — plus two dark-theme token pairs that were perceptually close despite distinct hex (tok_type↔tok_keyword, tok_function↔tok_identifier, ΔE2000 ~14). All four were fixed in the same change (commit 65eab71); the gates now hold the palette to both bars permanently. A dev tool, scripts/palette-preview.py, renders the live palette with contrast + ΔE2000 for future palette work.

NFR-1 / NFR-3 — startup & memory: measured, generously bounded

The Tier-4 PTY harness (ADR-0008 Amendment 1) measures both:

• startup_to_first_frame_is_reasonable — spawn → first rendered frame.
• idle_memory_footprint_is_reasonable — child /proc/<pid> VmRSS after the app is idle (Linux only).

These run against the debug binary, so they assert generous bounds (startup < 2000 ms, RSS < 150 MB) for gross-regression detection, not the literal NFR targets — a tight 500 ms / 50 MB gate on a debug binary under a loaded shared runner would flake. The real release figures (measured 2026-06-22, 5 trials each, this dev machine) are the documented verification:

NFR	Target	Release median	Margin
NFR-1 startup → first frame	< 500 ms	~29 ms	~17× under
NFR-3 idle RSS	< 50 MB	~10 MB	~5× under

(Measurement granularity is the harness poll interval, ~20 ms — figures are "tens of ms", not sub-ms precise.) Per the user decision these are not wired as tight CI gates; the generous debug-bound tests are the ongoing regression signal.

NFR-2 — input responsiveness: verified by architecture

"Long-running queries execute off the UI thread so the interface stays responsive." This is structural and already true: database work runs on a dedicated worker thread (ADR-0010), and terminal input is read on a separate Tokio task (spawn_event_reader in src/runtime.rs), so a running query cannot block keystroke handling or rendering. There is no automated keystroke-latency test — a deterministic sub-16 ms measurement in a PTY harness would be flaky and low-value — so NFR-2 is verified by this architectural argument rather than a gate.

NFR-4 / NFR-6 — distinctive design & cross-platform parity: reviewer note

Qualitative bars. NFR-4 (deliberate, identifiable palette/layout using box-drawing) is satisfied by the bespoke two-theme palette, the three-region framed layout, and the annotated plan/relationship rendering — reviewer judgement, not automatable. NFR-6 is partly evidenced by the CI matrix (Linux + macOS execute the suite; Windows is build-only, no runner) and is otherwise a reviewer judgement; one documented divergence: on terminals/multiplexers without true-colour passthrough (e.g. a tmux session missing Tc/RGB), the 24-bit palette is quantised to the 256-colour cube and near hues can collide — a terminal-capability issue, not a palette one.

Consequences

• Contrast and perceptual distinctness are now permanently gated — the palette cannot regress below WCAG-AA or into look-alike token colours without failing the build.
• Startup and memory have recorded figures and a generous gross-regression test; the literal targets are met with large margins.
• The qualitative and architectural NFRs are documented with evidence rather than asserted.
• A latent behaviour found during this work — fast DDL→insert misparsing against a stale schema cache — is tracked as issue #39 (deferred; no interactive-user impact).

Requirements bookkeeping

NFR-1, NFR-3, NFR-5, NFR-7 → [x] (gated test and/or measured with evidence). NFR-2 → [x] (architectural). NFR-4, NFR-6 → [/] (reviewer note; not fully automatable).