docs(tt4,nfr): record Tier-4 + NFR verification (ADR-0008 A1, ADR-0057)
Document the regression-hardening work from this session: - ADR-0008 Amendment 1 — Tier 4 realized: the portable-pty + vt100 harness, expectrl dropped, serial/fail-fast design, the four flows, sidebar-region reads, command pacing (issue #39), CI default-target (advances TT5). - ADR-0057 (new) — NFR verification strategy: contrast + ΔE2000 gates (NFR-5/7), startup/RSS measured (NFR-1/3: release ~29 ms / ~10 MB), NFR-2 by architecture, NFR-4/6 reviewer note. Records the two shipped light-theme contrast defects this work caught and fixed. - requirements.md: TT4 [~]->[x], TT5 [/] (only a Windows exec runner left), NFR-1/2/3/5/7 ->[x], NFR-4/6 ->[/], each with evidence. - CHANGELOG.md (new): Keep a Changelog + SemVer, [Unreleased] + [0.2.0]. - README ADR index updated; handoff-76; the working plan. Docs-only (CI gate skips via paths-ignore).
This commit is contained in:
claude@clouddev1
@@ -0,0 +1,122 @@
|
||||
# ADR-0057: Non-functional-requirement verification strategy
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (2026-06-22).
|
||||
|
||||
## Context
|
||||
|
||||
The non-functional requirements (`requirements.md` NFR-1..7 — startup,
|
||||
input latency, memory, distinctive design, colour use, cross-platform
|
||||
parity, light/dark legibility) were quality bars that had **never been
|
||||
formally verified**, even though v0.2.0 ships public binaries. With users
|
||||
now pulling updates, we want each NFR either **gated by an automated test**
|
||||
(so a regression fails CI) or **measured-and-documented** with a recorded
|
||||
figure and a reviewer judgement — not left as an aspiration.
|
||||
|
||||
The decision below was taken with the user: contrast is a hard gate;
|
||||
startup and memory are measured against generous bounds (not the literal
|
||||
target — a tight timing/RSS gate would flake on a shared runner); the
|
||||
qualitative bars are verified by argument and reviewer note.
|
||||
|
||||
## Decision
|
||||
|
||||
### NFR-5 / NFR-7 — colour contrast & legibility: **gated**
|
||||
|
||||
Unit tests in `src/theme.rs` (Tier 1, run by the existing gate):
|
||||
|
||||
- **`all_text_colours_meet_wcag_aa_contrast`** — every text-bearing
|
||||
foreground (`fg`, `muted`, mode labels, `system`/`error`/`warning`,
|
||||
`plan_efficient`, and all `tok_*`) must clear **WCAG-AA 4.5:1** against
|
||||
`bg`, in **both** themes.
|
||||
- **`advanced_mode_border_meets_ui_contrast`** — the advanced-mode border
|
||||
carries a mode-warning signal, so it meets the **3:1** non-text /
|
||||
UI-component threshold. The plain `border` is decorative structural
|
||||
chrome and is **deliberately exempt** (it sits at 2.20:1 dark / 1.82:1
|
||||
light by design, so panel borders recede behind content — recorded here
|
||||
rather than silently tolerated).
|
||||
- **`syntax_token_colours_are_perceptually_distinct`** — contrast against
|
||||
the background is necessary but not sufficient: two token colours can
|
||||
each clear 4.5:1 yet be hard to tell *apart* on a good screen. Every pair
|
||||
of syntax-token classes that can share a line must clear **CIEDE2000
|
||||
(ΔE2000) ≥ 15** (post-tuning minimum is ~18). The metric is itself
|
||||
validated against the Sharma et al. reference vectors
|
||||
(`delta_e_2000_matches_reference_vectors`).
|
||||
|
||||
Computing the real ratios while writing these tests **uncovered two
|
||||
shipped defects** in the v0.2.0 light theme — `tok_string` at 4.42:1 and
|
||||
`tok_flag` at 3.15:1, both below the 4.5:1 the scheme promises — plus two
|
||||
dark-theme token pairs that were perceptually close despite distinct hex
|
||||
(`tok_type`↔`tok_keyword`, `tok_function`↔`tok_identifier`, ΔE2000 ~14).
|
||||
All four were fixed in the same change (commit `65eab71`); the gates now
|
||||
hold the palette to both bars permanently. A dev tool,
|
||||
`scripts/palette-preview.py`, renders the live palette with contrast +
|
||||
ΔE2000 for future palette work.
|
||||
|
||||
### NFR-1 / NFR-3 — startup & memory: **measured, generously bounded**
|
||||
|
||||
The Tier-4 PTY harness (ADR-0008 Amendment 1) measures both:
|
||||
|
||||
- **`startup_to_first_frame_is_reasonable`** — spawn → first rendered frame.
|
||||
- **`idle_memory_footprint_is_reasonable`** — child `/proc/<pid>` VmRSS
|
||||
after the app is idle (Linux only).
|
||||
|
||||
These run against the **debug** binary, so they assert *generous* bounds
|
||||
(startup < 2000 ms, RSS < 150 MB) for **gross-regression detection**, not
|
||||
the literal NFR targets — a tight 500 ms / 50 MB gate on a debug binary
|
||||
under a loaded shared runner would flake. The **real release figures**
|
||||
(measured 2026-06-22, 5 trials each, this dev machine) are the documented
|
||||
verification:
|
||||
|
||||
| NFR | Target | Release median | Margin |
|
||||
| --- | --- | --- | --- |
|
||||
| NFR-1 startup → first frame | < 500 ms | **~29 ms** | ~17× under |
|
||||
| NFR-3 idle RSS | < 50 MB | **~10 MB** | ~5× under |
|
||||
|
||||
(Measurement granularity is the harness poll interval, ~20 ms — figures
|
||||
are "tens of ms", not sub-ms precise.) Per the user decision these are
|
||||
**not** wired as tight CI gates; the generous debug-bound tests are the
|
||||
ongoing regression signal.
|
||||
|
||||
### NFR-2 — input responsiveness: **verified by architecture**
|
||||
|
||||
"Long-running queries execute off the UI thread so the interface stays
|
||||
responsive." This is structural and already true: database work runs on a
|
||||
dedicated **worker thread** (ADR-0010), and terminal input is read on a
|
||||
**separate Tokio task** (`spawn_event_reader` in `src/runtime.rs`), so a
|
||||
running query cannot block keystroke handling or rendering. There is no
|
||||
automated keystroke-latency test — a deterministic sub-16 ms measurement
|
||||
in a PTY harness would be flaky and low-value — so NFR-2 is verified by
|
||||
this architectural argument rather than a gate.
|
||||
|
||||
### NFR-4 / NFR-6 — distinctive design & cross-platform parity: **reviewer note**
|
||||
|
||||
Qualitative bars. **NFR-4** (deliberate, identifiable palette/layout using
|
||||
box-drawing) is satisfied by the bespoke two-theme palette, the
|
||||
three-region framed layout, and the annotated plan/relationship rendering —
|
||||
reviewer judgement, not automatable. **NFR-6** is partly evidenced by the
|
||||
CI matrix (Linux + macOS **execute** the suite; Windows is **build-only**,
|
||||
no runner) and is otherwise a reviewer judgement; one documented divergence:
|
||||
on terminals/multiplexers without true-colour passthrough (e.g. a tmux
|
||||
session missing `Tc`/`RGB`), the 24-bit palette is quantised to the
|
||||
256-colour cube and near hues can collide — a terminal-capability issue,
|
||||
not a palette one.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Contrast and perceptual distinctness are now **permanently gated** — the
|
||||
palette cannot regress below WCAG-AA or into look-alike token colours
|
||||
without failing the build.
|
||||
- Startup and memory have **recorded figures** and a generous
|
||||
gross-regression test; the literal targets are met with large margins.
|
||||
- The qualitative and architectural NFRs are **documented with evidence**
|
||||
rather than asserted.
|
||||
- A latent behaviour found during this work — fast DDL→insert misparsing
|
||||
against a stale schema cache — is tracked as **issue #39** (deferred; no
|
||||
interactive-user impact).
|
||||
|
||||
## Requirements bookkeeping
|
||||
|
||||
NFR-1, NFR-3, NFR-5, NFR-7 → `[x]` (gated test and/or measured with
|
||||
evidence). NFR-2 → `[x]` (architectural). NFR-4, NFR-6 → `[/]` (reviewer
|
||||
note; not fully automatable).
|
||||
Reference in New Issue
Block a user