rdbms-playground/docs/adr/0008-testing-approach.md

# ADR-0008: Testing approach

## Status

Accepted

## Context

The project's working standards (`CLAUDE.md`, plus the user's
global testing rules) require:

- Test coverage established before changes.
- Bugs reproduced with failing tests before fixes.
- Integration tests that exercise the full stack a user touches.
- "All green, zero skips" as the only acceptable end state.

A TUI application needs a credible testing story for those rules
to be enforceable. Naïvely, "TUI testing" sounds like an
afterthought of manual screenshots. In practice, the Rust +
Ratatui ecosystem supports automation across every level that
matters, and we want to commit to that explicitly rather than
discover it ad-hoc.

## Decision

Testing is structured in four tiers. Tiers 1–3 run on every
commit in CI; tier 4 runs on every commit for a focused subset of
critical flows and on a nightly schedule for broader coverage.

### Tier 1 — Pure-logic unit tests

Standard `cargo test` against modules with no terminal
dependency:

- DSL parser and command dispatcher
- Type mapping (user-facing → SQLite STRICT types)
- Project I/O (`project.yaml` + `data/<table>.csv` round-tripping)
- Database rebuild from authoritative text sources
- Snapshot ring buffer logic (ADR-0006)
- Replay log writer/reader

These are the bulk of the test count. Every behavioural unit has
unit tests; modules with non-trivial logic also have property
tests via `proptest` where the input space justifies it (parser
inputs, type coercion, CSV escaping, etc.).

### Tier 2 — Render assertions via Ratatui `TestBackend`

Ratatui's in-tree `TestBackend` renders into a `Buffer` (a 2D
cell grid). Tests build an app state, render a frame, and assert
on the resulting buffer.

- **Cell-level assertions** for narrow tests
  ("the status bar shows mode label `Simple` in the expected
  style").
- **Snapshot tests** via `insta` for whole-frame coverage of
  representative views (default screen, query result table,
  schema view, undo confirmation prompt). Snapshots are checked
  in and reviewed on diff.

Snapshot discipline:

- Snapshots cover stable, intentional UI surfaces; they are not
  added reflexively to every component.
- A snapshot diff is treated as a real review item, not a
  rubber-stamp. Reviewers must confirm the change is intended.

### Tier 3 — Synthetic event-loop integration tests

The application's update function consumes
`crossterm::event::Event` values. Tests feed sequences of
synthetic events (`KeyEvent`, `MouseEvent`, resize) to the update
function, then render via `TestBackend` and assert on both
state and buffer.

This tier is the equivalent of `react-testing-library` for our
TUI: it exercises the full input → state → render path without a
real terminal, and is where the most valuable behavioural tests
live. Examples:

- Typing a DSL command in simple mode, submitting with
  Ctrl-Enter, asserting the table list updates and the schema
  view re-renders.
- Triggering `undo`, asserting the confirmation prompt appears
  with the expected timestamp and change summary, confirming,
  asserting state restoration.
- Switching modes and verifying the prompt label and border
  colour change.

### Tier 4 — PTY-based black-box end-to-end

A small number of critical flows are exercised against the
**actual built binary** in a pseudo-terminal:

- Tooling: `portable-pty` for the PTY, `expectrl` for
  expect-style scripting, `vt100` to parse the terminal output
  stream into an inspectable cell grid.
- These tests catch issues the lower tiers miss: TTY setup,
  signal handling, terminal mode transitions, real I/O timing.

Tier 4 is **reserved for the highest-value flows**, not blanket
coverage. The initial scope is:

- Cold launch → first DDL command → graceful quit.
- Project save → process restart → reopen → identical state.
- Project export → import in a fresh project → rebuilt database
  matches the source.
- `undo` immediately after a `DROP TABLE`, including the
  confirmation prompt.

Tier 4 tests run in CI on every commit (the focused list above)
and on a nightly schedule for any extended coverage.

## Tooling commitments

- `cargo test` — Tier 1, Tier 2, Tier 3.
- `proptest` — property-based testing for parser and conversion
  layers.
- Ratatui's `TestBackend` — frame rendering for tests.
- `insta` — snapshot testing of rendered buffers.
- `portable-pty`, `expectrl`, `vt100` — Tier 4 PTY-based tests.
- CI matrix covers Linux, macOS, and Windows on stable Rust.

## Honest limits

- **No cross-terminal-emulator regression coverage.** Tier 4
  exercises a PTY but not real terminal emulators (xterm,
  Alacritty, Windows Terminal, etc.). Crossterm abstracts these
  well in practice; if a real-emulator regression ever surfaces,
  we will revisit.
- **No visual aesthetic checks.** Tests assert cell contents and
  styles, not "this layout is pretty". Visual polish is reviewed
  manually by humans.
- **Snapshot brittleness** is a known failure mode. We mitigate
  by being selective about what gets a snapshot and by treating
  snapshot diffs as real review items.

## Consequences

- Test discipline from `CLAUDE.md` is enforceable on every
  layer: parser bugs caught at Tier 1, UI flow bugs caught at
  Tier 3, real-binary regressions caught at Tier 4.
- Module boundaries are designed for testability from the start
  (pure logic modules separate from rendering, an explicit update
  function consuming events).
- CI cost is real but bounded: Tiers 1–3 are fast; Tier 4 is the
  only slow tier and is kept narrow.
- Adding a feature implies adding tests at the appropriate tier
  (or tiers); coverage is not retrofitted later.

## Amendment 1 — 2026-06-22: Tier 4 realized (TT4)

Tier 4 was a specification only (no PTY deps, no tests) until this
amendment. It is now implemented in `tests/e2e_pty.rs`, and the realized
shape refines the original decision in three ways.

**Tooling — `expectrl` dropped.** The decision named
`portable-pty` + `expectrl` + `vt100`. We kept **`portable-pty`** (0.9, to
spawn the real binary in a PTY at a fixed size) and **`vt100`** (0.16, to
parse the output stream into an inspectable cell grid), both current and
maintained. We **dropped `expectrl`**: it bundles its own PTY abstraction
(which conflicts with `portable-pty`) and matches line-by-line, a poor fit
for a full-screen TUI. A small hand-rolled `wait_for(predicate, timeout)`
that polls the vt100 screen replaces it — fewer dependencies, and assertions
read the actual rendered grid.

**Harness shape.** Each test spawns the binary
(`env!("CARGO_BIN_EXE_rdbms-playground")`) under a fresh PTY at **100×30**
(the wide three-region layout, ADR-0046), with its **own temp `--data-dir`**
so it never touches real projects or resume state, and `--theme dark` for
deterministic styling. Tests run **serially** (one PTY + child at a time)
via a poison-tolerant global lock, so timing stays predictable on the
low-parallelism self-hosted runner. Waits are **tight and fail-fast** (3 s):
a slow wait is a genuine hang, and a timeout panics with a full screen dump
for debuggability.

Two non-obvious robustness rules the harness encodes:
- **Read table presence from the Tables *sidebar* region, not the whole
  screen** — the Output panel echoes every command, so a table name typed
  in a command would pollute a whole-screen match.
- **Pace each command to completion before the next** (real-user / cast
  cadence). A command submitted while the previous one's worker rebuild is
  still in flight can be misread against a stale schema cache (issue #39,
  discovered here — interactive use is unaffected).

**The four flows** mirror this ADR's Tier-4 scope: (1) cold launch → first
DDL → graceful quit (Ctrl-C); (2) create → quit → reopen via `--resume`,
asserting the *column* (not just the table name) round-trips; (3) export →
import into a fresh project → schema **and** data rebuilt; (4) `undo` after
`DROP TABLE`, through the Y/N confirm modal.

**CI.** The target runs by default in `cargo test`, so the existing Linux
gate exercises it on every push — this **advances TT5** (Tier 4 now runs in
CI on Linux). PTY-in-container was validated (openpty + child spawn work
under the Docker runtime the Gitea runner uses); the first real CI run is
the final confirmation. **Windows execution remains out of scope** (no
Windows runner), and the original Tier-4 note about a *nightly broader
coverage* schedule is **not** implemented — the focused four flows are the
committed TT4 scope; broader coverage is deferred.

**Piggybacked NFR measurement.** The same harness measures NFR-1 (startup
to first frame) and NFR-3 (idle RSS, Linux) — see ADR-0057.