docs(tt4,nfr): record Tier-4 + NFR verification (ADR-0008 A1, ADR-0057)
Document the regression-hardening work from this session: - ADR-0008 Amendment 1 — Tier 4 realized: the portable-pty + vt100 harness, expectrl dropped, serial/fail-fast design, the four flows, sidebar-region reads, command pacing (issue #39), CI default-target (advances TT5). - ADR-0057 (new) — NFR verification strategy: contrast + ΔE2000 gates (NFR-5/7), startup/RSS measured (NFR-1/3: release ~29 ms / ~10 MB), NFR-2 by architecture, NFR-4/6 reviewer note. Records the two shipped light-theme contrast defects this work caught and fixed. - requirements.md: TT4 [~]->[x], TT5 [/] (only a Windows exec runner left), NFR-1/2/3/5/7 ->[x], NFR-4/6 ->[/], each with evidence. - CHANGELOG.md (new): Keep a Changelog + SemVer, [Unreleased] + [0.2.0]. - README ADR index updated; handoff-76; the working plan. Docs-only (CI gate skips via paths-ignore).
This commit is contained in:
claude@clouddev1
@@ -0,0 +1,189 @@
|
||||
# Plan — TT4 (Tier-4 PTY tests) + NFR verification + CHANGELOG
|
||||
|
||||
Status: **approved 2026-06-22**, implementation pending. Supersedes the
|
||||
transient harness plan-mode copy. This is the persistent, version-controlled
|
||||
plan per project convention (`docs/plans/`).
|
||||
|
||||
## Context
|
||||
|
||||
RDBMS Playground now ships public binaries (v0.2.0 on crates.io, plus
|
||||
Scoop/Homebrew/winget). With real users pulling updates, the priority is
|
||||
**regression protection before the next release**. Three tracked gaps are taken
|
||||
up together:
|
||||
|
||||
- **TT4** (`requirements.md`, ADR-0008 Tier 4): the four critical end-to-end
|
||||
flows are specified but **no PTY harness exists** — no deps, no tests. Tiers
|
||||
1–3 never exercise the *real built binary* through a terminal.
|
||||
- **NFR-1..7**: quality bars (startup, latency, memory, contrast, distinctive
|
||||
design, parity) that were **never formally verified** despite shipping.
|
||||
- **CHANGELOG**: none exists; useful now that releases are public (and lets the
|
||||
CI `winget` job pass `--release-notes-url` later).
|
||||
|
||||
Baseline (Phase 1): `cargo test` = **2509 passed / 0 failed / 1 ignored**;
|
||||
clippy + fmt clean. This is the number Phase 5 verifies against (new tests add
|
||||
to it; nothing may regress).
|
||||
|
||||
User decisions (Phase 3):
|
||||
- **TT4 in CI:** own `e2e_pty` test target, runs by default in `cargo test`
|
||||
(so the Linux gate exercises it every push). **Tight, fail-fast timeouts** —
|
||||
the runner is self-hosted, low-parallelism, low expected flakiness (so a
|
||||
failure surfaces fast rather than hanging on a generous timeout).
|
||||
- **NFR:** WCAG contrast = hard gated test; startup + idle-memory = measured via
|
||||
the PTY harness against *generous* bounds + numbers recorded; NFR-2/4/6 =
|
||||
verify-by-argument + reviewer note. Each requirement → `[x]`/`[/]` with
|
||||
evidence.
|
||||
- **CHANGELOG:** Keep a Changelog + SemVer; `[Unreleased]` + a single
|
||||
consolidated `[0.2.0]` only (0.1.0 treated as pre-history).
|
||||
|
||||
---
|
||||
|
||||
## Workstream 1 — TT4: Tier-4 PTY harness + 4 flows
|
||||
|
||||
### Tooling (refines ADR-0008's stated `portable-pty`+`expectrl`+`vt100`)
|
||||
- **`portable-pty`** (wezterm, actively maintained) — spawn the real binary in a
|
||||
PTY with a fixed window size.
|
||||
- **`vt100`** — parse the PTY output stream into an inspectable cell grid.
|
||||
- **Drop `expectrl`** — it bundles its *own* PTY abstraction (conflicts with
|
||||
portable-pty) and is line-oriented, a poor fit for a full-screen TUI. Replace
|
||||
it with a small hand-rolled `wait_for(predicate, timeout)` polling the vt100
|
||||
screen. This deviation is recorded in the ADR-0008 amendment.
|
||||
- **Checkpoint at implementation:** verify `vt100` is still maintained/current
|
||||
(global currency rule). If stale, evaluate `avt` as the screen parser before
|
||||
committing the dep. `cargo add` picks latest; pin sensibly.
|
||||
|
||||
### Harness (`tests/e2e_pty.rs` — new `[[test]]` target)
|
||||
A `support` module providing:
|
||||
- `struct PtyApp` — opens a PTY (e.g. **100×30**, the wide three-region
|
||||
layout), spawns `env!("CARGO_BIN_EXE_rdbms-playground")` with a **per-test
|
||||
temp `--data-dir`** (`tempfile::TempDir`), `--theme dark`, `TERM=xterm-256color`;
|
||||
a reader thread feeds bytes into a `vt100::Parser`.
|
||||
- `send(bytes)` (raw; `\r` = Enter, `\x03` = Ctrl-C), `send_line(s)` =
|
||||
`s` + `\r`.
|
||||
- `wait_for(&str, Duration)` — poll the vt100 screen text until the substring
|
||||
appears or the (tight, ~**3 s**) timeout fires; on timeout **panic with a full
|
||||
screen dump** (fail-fast + debuggable). A `screen_text()` helper for assertions.
|
||||
- `quit()` — `send("\x03")`, wait for child exit, assert success.
|
||||
- Run **serially** (`serial_test` dep, or a shared `Mutex`/single-threaded
|
||||
target) — PTYs + temp dirs shouldn't race; also keeps timing stable.
|
||||
- **Instrument** with `eprintln!`/screen dumps on every wait so a CI failure is
|
||||
diagnosable from logs (per the project's logging discipline).
|
||||
|
||||
### The four flows (one `#[test]` each, mirroring ADR-0008 §Tier-4)
|
||||
1. **Cold launch → DDL → quit.** Launch fresh; `wait_for("SIMPLE")` +
|
||||
`wait_for("(none yet)")`; `send_line("create table Customers with pk
|
||||
id(serial)")`; `wait_for("Customers")` in the Tables panel; `quit()`.
|
||||
2. **Save → restart → reopen.** Launch project path P under the temp data-dir;
|
||||
create a table; `quit()` (autosave already persisted per-command); relaunch
|
||||
**same P**; `wait_for("Customers")` — schema restored from text/rebuild.
|
||||
3. **Export → import → rebuild.** Project A: create table + insert a row;
|
||||
`send_line("export A.zip")` (explicit path under the temp dir);
|
||||
`wait_for` export-success note; `quit()`. Fresh project B:
|
||||
`send_line("import <path>/A.zip")`; `wait_for` import note (rebuild auto-runs
|
||||
on missing `.db`); assert the table + row are present (`show data` /
|
||||
Tables panel).
|
||||
4. **Undo after DROP.** Create table; `send_line("drop table Customers")`;
|
||||
confirm gone; `send_line("undo")`; **`wait_for("Restore that earlier
|
||||
state?")`** (the real modal); `send("y")`; `wait_for("Customers")` restored.
|
||||
|
||||
### CI
|
||||
No workflow edit needed for execution: `e2e_pty` is a default target, so the
|
||||
existing `gate` job's `cargo test --no-fail-fast` runs it on the Linux
|
||||
container. **Validate locally first** that a PTY opens inside the
|
||||
`rdbms-playground-ci` image (container `/dev/ptmx`); if not, add `--init`/pts
|
||||
mount notes to the ADR. Advances **TT5** (Tier-4-in-CI on Linux); Windows
|
||||
execution runner stays out of scope (no runner).
|
||||
|
||||
---
|
||||
|
||||
## Workstream 2 — NFR verification
|
||||
|
||||
### Gated: WCAG contrast (NFR-5, NFR-7) — `src/theme.rs` `#[cfg(test)]`
|
||||
- Add a `relative_luminance(Color) -> f64` + `contrast_ratio(a,b) -> f64`
|
||||
helper (WCAG 2.x sRGB formula) in the test module; a `match Color::Rgb` channel
|
||||
extractor (panics on non-RGB — also a guard against a future named-colour
|
||||
regression).
|
||||
- For **both** `Theme::light()` and `Theme::dark()`, assert **≥ 4.5:1** for
|
||||
normal-text foregrounds on `bg`: `fg`, `error`, `warning`, `system`,
|
||||
`plan_efficient`, `mode_simple`, `mode_advanced`, and the info-carrying syntax
|
||||
tokens (`tok_keyword/identifier/type/number/string/flag/function/error`).
|
||||
- **`muted`/`tok_punct`** (deliberately dim secondary text) and **borders**
|
||||
(`border`/`border_advanced`, non-text UI): assert the WCAG **non-text 3:1**
|
||||
threshold, with a comment citing the WCAG large-text/non-text allowance. If
|
||||
any *fails even 3:1*, that's a real finding → surface to the user, don't
|
||||
silently relax. (Supersedes the existing inequality-only theme tests.)
|
||||
- This runs in the existing gate — permanent regression protection on the palette.
|
||||
|
||||
### Measured + documented: startup (NFR-1) + idle memory (NFR-3)
|
||||
Two measurement tests in `e2e_pty.rs`:
|
||||
- **Startup:** time from spawn to `wait_for("SIMPLE")` (first rendered frame);
|
||||
assert a **generous** bound (e.g. < 1500 ms locally) and `eprintln!` the actual.
|
||||
- **Idle memory:** after the app is idle, read the child's `/proc/<pid>/status`
|
||||
`VmRSS` (Linux); assert a generous bound and print actual. (Linux-gated via
|
||||
`#[cfg(target_os = "linux")]`.)
|
||||
The point is gross-regression detection, not a tight SLA gate (user decision).
|
||||
|
||||
### Verify-by-argument + reviewer note (NFR-2, NFR-4, NFR-6)
|
||||
- **NFR-2** (input off-thread): already architecturally true — DB runs on the
|
||||
worker thread (ADR-0010) and input on a separate Tokio task (confirmed in
|
||||
`runtime.rs`). Record the evidence; optionally a test that a query doesn't
|
||||
block a subsequent keystroke render.
|
||||
- **NFR-4** (distinctive design) / **NFR-6** (cross-platform parity): qualitative
|
||||
/ partly-CI. Written reviewer verification (DA-hat) in the NFR verification doc;
|
||||
parity partly evidenced by the CI matrix (Linux+macOS execute, Windows builds).
|
||||
|
||||
### Requirements bookkeeping
|
||||
Move NFR-5/7 → `[x]` (gated test), NFR-1/3 → `[x]` (measured + bounded test +
|
||||
recorded numbers), NFR-2 → `[x]` (architectural evidence), NFR-4/6 → `[/]` with
|
||||
the reviewer note (honest: not fully automatable). Each with a commit/test ref.
|
||||
|
||||
---
|
||||
|
||||
## Workstream 3 — CHANGELOG
|
||||
|
||||
- New **`CHANGELOG.md`** at repo root, **Keep a Changelog** + SemVer.
|
||||
- `[Unreleased]` (empty/seeded) + one consolidated **`[0.2.0] - <release date>`**
|
||||
with Added/Changed/Fixed of the notable user-facing changes (install methods,
|
||||
packaging, `--version`/`version`, seed improvements, hint feature, readline
|
||||
keys, cross-mode history) mined from ADRs 0053–0056 + handoffs 73–75. 0.1.0 =
|
||||
pre-history (a one-line note).
|
||||
- Optional follow-up (flag, don't auto-do): wire `--release-notes-url` into the
|
||||
CI `winget`/release path — out of scope unless you want it now.
|
||||
|
||||
---
|
||||
|
||||
## Docs / ADR updates (project discipline)
|
||||
|
||||
- **Amend ADR-0008** (Tier-4 section): record the realized tooling
|
||||
(`portable-pty` + `vt100` + hand-rolled `wait_for`, `expectrl` dropped),
|
||||
serial execution, tight-timeout/fail-fast stance, the default-target CI wiring,
|
||||
and the four implemented flows. Update `docs/adr/README.md` in the same edit.
|
||||
- **New ADR-0057 — NFR verification strategy** (next free number; assigned at
|
||||
merge per ADR-0000): the contrast-gate + thresholds (incl. the 3:1
|
||||
muted/border allowance), the measured-not-tightly-gated stance for
|
||||
startup/memory, and the verify-by-argument treatment of NFR-2/4/6. Add a
|
||||
short **NFR verification record** (the measured numbers + reviewer notes) —
|
||||
either in the ADR or a `docs/` companion. Update the README index.
|
||||
- Update `docs/requirements.md`: TT4 `[~]`→`[x]`, TT5 `[/]` note (Tier-4 now in
|
||||
Linux CI; Windows-exec still pending), NFR rows as above.
|
||||
- A handoff note (next sequence number) per project convention.
|
||||
|
||||
---
|
||||
|
||||
## Verification (Phase 5)
|
||||
|
||||
1. `cargo test` — full suite green; new e2e_pty + contrast tests pass; **2509 +
|
||||
new, 0 failed, ≤1 ignored**; no regressions vs baseline.
|
||||
2. `cargo clippy --all-targets -- -D warnings` + `cargo fmt --check` clean.
|
||||
3. Run `e2e_pty` repeatedly (e.g. 5–10×) locally to confirm non-flaky under the
|
||||
tight timeouts before relying on the gate.
|
||||
4. Confirm a PTY opens inside the CI image (or document the fix).
|
||||
5. DA-hat pass over each workstream + the NFR reviewer judgements, written down.
|
||||
6. `cargo sweep` at the milestone (CLAUDE.md build hygiene).
|
||||
|
||||
## Risks / checkpoints
|
||||
- **vt100 currency** — verify before adding; fall back to `avt` if stale.
|
||||
- **PTY-in-container** — validate early; the whole TT4 CI value depends on it.
|
||||
- **muted/border contrast** — may fail even 3:1; if so it's a finding to escalate,
|
||||
not silently relax (could mean a small palette tweak — touches a decided area,
|
||||
so escalate).
|
||||
- **Commits** — confirm each message; no AI attribution; append-only; never push.
|
||||
Reference in New Issue
Block a user