rdbms-playground/docs/adr/0008-testing-approach.md

ADR-0008: Testing approach

Status

Accepted

Context

The project's working standards (CLAUDE.md, plus the user's global testing rules) require:

• Test coverage established before changes.
• Bugs reproduced with failing tests before fixes.
• Integration tests that exercise the full stack a user touches.
• "All green, zero skips" as the only acceptable end state.

A TUI application needs a credible testing story for those rules to be enforceable. Naïvely, "TUI testing" sounds like an afterthought of manual screenshots. In practice, the Rust + Ratatui ecosystem supports automation across every level that matters, and we want to commit to that explicitly rather than discover it ad-hoc.

Decision

Testing is structured in four tiers. Tiers 1–3 run on every commit in CI; tier 4 runs on every commit for a focused subset of critical flows and on a nightly schedule for broader coverage.

Tier 1 — Pure-logic unit tests

Standard cargo test against modules with no terminal dependency:

• DSL parser and command dispatcher
• Type mapping (user-facing → SQLite STRICT types)
• Project I/O (project.yaml + data/<table>.csv round-tripping)
• Database rebuild from authoritative text sources
• Snapshot ring buffer logic (ADR-0006)
• Replay log writer/reader

These are the bulk of the test count. Every behavioural unit has unit tests; modules with non-trivial logic also have property tests via proptest where the input space justifies it (parser inputs, type coercion, CSV escaping, etc.).

Tier 2 — Render assertions via Ratatui TestBackend

Ratatui's in-tree TestBackend renders into a Buffer (a 2D cell grid). Tests build an app state, render a frame, and assert on the resulting buffer.

• Cell-level assertions for narrow tests ("the status bar shows mode label Simple in the expected style").
• Snapshot tests via insta for whole-frame coverage of representative views (default screen, query result table, schema view, undo confirmation prompt). Snapshots are checked in and reviewed on diff.

Snapshot discipline:

• Snapshots cover stable, intentional UI surfaces; they are not added reflexively to every component.
• A snapshot diff is treated as a real review item, not a rubber-stamp. Reviewers must confirm the change is intended.

Tier 3 — Synthetic event-loop integration tests

The application's update function consumes crossterm::event::Event values. Tests feed sequences of synthetic events (KeyEvent, MouseEvent, resize) to the update function, then render via TestBackend and assert on both state and buffer.

This tier is the equivalent of react-testing-library for our TUI: it exercises the full input → state → render path without a real terminal, and is where the most valuable behavioural tests live. Examples:

• Typing a DSL command in simple mode, submitting with Ctrl-Enter, asserting the table list updates and the schema view re-renders.
• Triggering undo, asserting the confirmation prompt appears with the expected timestamp and change summary, confirming, asserting state restoration.
• Switching modes and verifying the prompt label and border colour change.

Tier 4 — PTY-based black-box end-to-end

A small number of critical flows are exercised against the actual built binary in a pseudo-terminal:

• Tooling: portable-pty for the PTY, expectrl for expect-style scripting, vt100 to parse the terminal output stream into an inspectable cell grid.
• These tests catch issues the lower tiers miss: TTY setup, signal handling, terminal mode transitions, real I/O timing.

Tier 4 is reserved for the highest-value flows, not blanket coverage. The initial scope is:

• Cold launch → first DDL command → graceful quit.
• Project save → process restart → reopen → identical state.
• Project export → import in a fresh project → rebuilt database matches the source.
• undo immediately after a DROP TABLE, including the confirmation prompt.

Tier 4 tests run in CI on every commit (the focused list above) and on a nightly schedule for any extended coverage.

Tooling commitments

• cargo test — Tier 1, Tier 2, Tier 3.
• proptest — property-based testing for parser and conversion layers.
• Ratatui's TestBackend — frame rendering for tests.
• insta — snapshot testing of rendered buffers.
• portable-pty, expectrl, vt100 — Tier 4 PTY-based tests.
• CI matrix covers Linux, macOS, and Windows on stable Rust.

Honest limits

• No cross-terminal-emulator regression coverage. Tier 4 exercises a PTY but not real terminal emulators (xterm, Alacritty, Windows Terminal, etc.). Crossterm abstracts these well in practice; if a real-emulator regression ever surfaces, we will revisit.
• No visual aesthetic checks. Tests assert cell contents and styles, not "this layout is pretty". Visual polish is reviewed manually by humans.
• Snapshot brittleness is a known failure mode. We mitigate by being selective about what gets a snapshot and by treating snapshot diffs as real review items.

Consequences

• Test discipline from CLAUDE.md is enforceable on every layer: parser bugs caught at Tier 1, UI flow bugs caught at Tier 3, real-binary regressions caught at Tier 4.
• Module boundaries are designed for testability from the start (pure logic modules separate from rendering, an explicit update function consuming events).
• CI cost is real but bounded: Tiers 1–3 are fast; Tier 4 is the only slow tier and is kept narrow.
• Adding a feature implies adding tests at the appropriate tier (or tiers); coverage is not retrofitted later.

Amendment 1 — 2026-06-22: Tier 4 realized (TT4)

Tier 4 was a specification only (no PTY deps, no tests) until this amendment. It is now implemented in tests/e2e_pty.rs, and the realized shape refines the original decision in three ways.

Tooling — expectrl dropped. The decision named portable-pty + expectrl + vt100. We kept portable-pty (0.9, to spawn the real binary in a PTY at a fixed size) and vt100 (0.16, to parse the output stream into an inspectable cell grid), both current and maintained. We dropped expectrl: it bundles its own PTY abstraction (which conflicts with portable-pty) and matches line-by-line, a poor fit for a full-screen TUI. A small hand-rolled wait_for(predicate, timeout) that polls the vt100 screen replaces it — fewer dependencies, and assertions read the actual rendered grid.

Harness shape. Each test spawns the binary (env!("CARGO_BIN_EXE_rdbms-playground")) under a fresh PTY at 100×30 (the wide three-region layout, ADR-0046), with its own temp --data-dir so it never touches real projects or resume state, and --theme dark for deterministic styling. Tests run serially (one PTY + child at a time) via a poison-tolerant global lock, so timing stays predictable on the low-parallelism self-hosted runner. Waits are tight and fail-fast (3 s): a slow wait is a genuine hang, and a timeout panics with a full screen dump for debuggability.

Two non-obvious robustness rules the harness encodes:

• Read table presence from the Tables sidebar region, not the whole screen — the Output panel echoes every command, so a table name typed in a command would pollute a whole-screen match.
• Pace each command to completion before the next (real-user / cast cadence). A command submitted while the previous one's worker rebuild is still in flight can be misread against a stale schema cache (issue #39, discovered here — interactive use is unaffected).

The four flows mirror this ADR's Tier-4 scope: (1) cold launch → first DDL → graceful quit (Ctrl-C); (2) create → quit → reopen via --resume, asserting the column (not just the table name) round-trips; (3) export → import into a fresh project → schema and data rebuilt; (4) undo after DROP TABLE, through the Y/N confirm modal.

CI. The target runs by default in cargo test, so the existing Linux gate exercises it on every push — this advances TT5 (Tier 4 now runs in CI on Linux). PTY-in-container was validated (openpty + child spawn work under the Docker runtime the Gitea runner uses); the first real CI run is the final confirmation. Windows execution remains out of scope (no Windows runner), and the original Tier-4 note about a nightly broader coverage schedule is not implemented — the focused four flows are the committed TT4 scope; broader coverage is deferred.

Piggybacked NFR measurement. The same harness measures NFR-1 (startup to first frame) and NFR-3 (idle RSS, Linux) — see ADR-0057.