# ADR-0034: `history.log` as a complete command journal; replay reads success-only

## Status

Accepted

## Context

`history.log` serves three stated purposes (ADR-0006 "Replay
log"; ADR-0015 §5/§12):

1. **Persistent input history** — the in-memory Up/Down recall
   ring is hydrated from `history.log` on project open
   (ADR-0015 §12, "I2-persist").
2. **Replay / scripting** — `replay <path>` re-executes a file
   of commands; `history.log` itself was declared
   "replay-compatible" (ADR-0015 §13 OOS-2; ADR-0006).
3. **Reproducible bug-report artifact** when a project is
   shared.

Tracing the implementation surfaced three problems that pull
against each other:

### Problem 1 — failed commands are lost across sessions

`history.log` records **only successful** commands. The writer
(`src/persistence/history.rs`) hardcodes the status field to
`ok` and never appends a failed command (ADR-0015 §5: "Status
is always `ok` in v1, because failed commands are not
recorded").

But the **in-memory** recall ring (`App.history`) records
**every** submission, success or failure — `App::submit`
pushes the trimmed line "regardless of whether it parses, so
users can recall and edit typo'd commands". So:

- **Within a session**, Up/Down recalls a failed command (type
  bad SQL, see it fail, fix the table, Up-arrow twice to bring
  the failed command back and edit it). This works.
- **Across sessions**, the ring is re-seeded from
  `history.log`, which dropped the failures — so the failed
  command the user wanted to fix is gone after a restart.

This asymmetry is the core defect. The recall surface should
behave the same whether the command came from this session's
in-memory ring or a previous session's hydrated log.

(Note: ADR-0015 §12's claim that "new successful commands … are
pushed onto the in-memory stack as they are now" is itself
inaccurate about the code — the stack receives *all*
submissions, not only successful ones. This ADR's model makes
the persistent log match that existing in-memory behaviour.)

### Problem 2 — the two consumers want different inputs

- **Replay** wants the commands that *built* the current state
  — the successful ones. Replaying a command that failed is
  pointless (it re-fails, or worse, behaves differently against
  a now-different schema).
- **Recall** wants *everything the user typed*, including
  failures, so they can fix and resubmit.

A single success-only log cannot satisfy both; serving recall
forces failures into the file, which then pollutes replay. The
status field — present in the line format precisely so "future
use cases can carry additional values without a format break"
(ADR-0015 §5) — is the lever that lets one journal serve both:
record everything with a status, and let each consumer filter.

### Problem 3 — `replay history.log` does not actually work

`run_replay` (`src/runtime.rs`) reads the target file and
parses **each whole trimmed line** through the DSL parser,
skipping only blank lines and `#` comments. It has no
understanding of the `<timestamp>|<status>|<source>` record
shape. Feeding it a real `history.log` fails on line 1 (the
leading timestamp is not a command keyword), despite ADR-0006
listing "`history.log` itself can be replayed" and the
`resolve_replay_path` doc comment promising "`replay
history.log` works without ceremony".

No test covers this: every replay test feeds a hand-written
plain `.commands` script (one bare command per line). The test
named `replay_history_log_records_subcommands_only` only
verifies that replaying a plain script *writes* the right
entries *into* `history.log`; it never replays the pipe-format
file as input. So the gap between the stated capability and the
implementation is invisible to the suite.

## Decision

### 1. `history.log` is a complete command journal, status-tagged

Every submitted command is recorded, regardless of outcome,
keeping the existing three-field line shape:

```
<timestamp>|<status>|<source>
```

`status` takes at least these values:

- `ok` — the command executed successfully (the only value
  written today).
- `err` — the command did **not** succeed: it either failed to
  parse / validate, or it dispatched and the engine/worker
  returned an error.

Finer-grained values (e.g. distinguishing a parse error from an
execution error, or marking imported vs user-issued commands)
remain a non-breaking future extension — the field already
tolerates additional values, and readers ignore values they do
not recognise.

This supersedes ADR-0015 §5's "Status is always `ok` … failed
commands are not recorded" and ADR-0006's "Every *successfully
executed* command … is appended" — both are narrowed to: *every
submitted command is appended, tagged with its resulting
status.*

### 2. Each consumer filters by status

- **Input-history hydration** (ADR-0015 §12) reads **all**
  records — `ok` and `err` — so cross-session recall matches
  the in-memory ring's "record everything" behaviour. A failed
  command from a previous session is recallable and editable
  after restart, just as it is within the session it was typed.
- **Replay** reads **`ok` records only**. A journal carrying
  `err` lines stays directly replayable: the failures are
  skipped, leaving exactly the sequence that built the state.
- **Bug-report artifact**: the full journal (both statuses) is
  the most useful form — it shows what the user tried, not just
  what worked.

### 3. Replay learns the journal format (and keeps plain scripts)

`replay <path>` must accept **both** input shapes a user can
hand it:

- A **journal record** — a line matching the
  `<iso8601-timestamp>|<status>|<source>` prefix. Replay
  extracts `<source>` (unescaping it per §5), and **skips the
  line unless `status == ok`**.
- A **bare command** — any other non-blank, non-`#` line, as in
  a hand-written `.commands` script. Replayed as-is (current
  behaviour).

Detection is by the leading `<timestamp>|<status>|` prefix,
where `<status>` is one of the known tokens. A bare command
that happens to contain `|` (e.g. `select 'a|b' from t`) is not
misread, because it does not begin with a timestamp-and-status
prefix. This makes `replay history.log` work as ADR-0006
promised, without breaking `.commands` scripts.

### 4. Where the writes happen

- **Successful commands** continue to be journalled by the db
  worker, transactionally coupled to the state change
  (`finalize_persistence` before `tx.commit()`), exactly as
  today — the `ok` line and the committed state land together
  or not at all.
- **Failed commands** are journalled with `err` from the
  runtime/app error path, *not* the worker: a parse failure
  never reaches the worker, and an execution failure has no
  committed state to couple to. The `err` append is therefore
  best-effort (a failure to record a failure is logged and
  ignored — it must never escalate a user error into a fatal).

This split is an implementation consequence of the journal
decision, recorded here so the implementing sub-task does not
re-derive it.

### 5. Backward compatibility

Existing `history.log` files contain only `ok` lines and remain
valid: hydration reads them (all `ok`), replay replays them
(all `ok`), nothing migrates. The malformed-line skipping in
`read_recent_sources` already tolerates unrecognised shapes, so
a mixed-version file degrades gracefully.

## Consequences

- The persistence layer gains a status parameter on the history
  append; `format_record` stops hardcoding `ok`.
- The runtime/app error paths gain an `err` journal append
  (best-effort) for both parse failures and execution failures.
- `read_recent_sources` (hydration) returns all records
  regardless of status — no change if it already ignores the
  status field, but the intent is now explicit and must be
  test-pinned.
- `run_replay` gains journal-record parsing with an `ok`-only
  filter, while still accepting bare-command scripts. This is
  the fix for Problem 3 and **must ship with a test that feeds
  an actual `history.log` to replay** (the missing-coverage gap
  that hid the bug).
- The bug-report artifact becomes strictly more useful (it now
  shows attempted-and-failed commands).
- A shared/exported project still excludes `history.log`
  (ADR-0015 §11 / ADR-0007), so the journal's added failure
  records do not leak by default.
- ADR-0006 and ADR-0015 §5/§12 are amended as stated in
  Decision §1–§2; they remain canonical for everything outside
  the amended clauses.

## Implementation note

Per the decision to capture this now and implement later, the
code lands as two tracked sub-tasks (test-first):

1. **Journal failures + per-consumer filtering** — status
   parameter on the append; `err` writes from the error paths;
   hydration reads all; replay reads `ok` only. Tests: a failed
   command appears in `history.log` with `err`; it survives a
   hydration round-trip into the recall ring; replay skips it.
2. **Replay parses the journal format** — dual-shape input
   (journal record vs bare command), `ok`-only filter, source
   unescaping. Tests: replaying an actual `history.log` (pipe
   format, mixed `ok`/`err`) runs the `ok` commands and skips
   the `err` ones; a plain `.commands` script still replays
   unchanged; a bare command containing `|` is not misparsed.

Both shipped 2026-05-24 (test-first), along with Amendment 1.

**Deferred follow-up (user-confirmed, 2026-05-24).** `err`
journalling covers parse failures of *any* submitted line and
DSL/SQL *worker* execution failures (which surface as
`DslFailed`). An app-lifecycle command that *parses* and then
fails at the *runtime* stage (e.g. a `save as` / `import` that
fails on I/O) is **not** yet journalled `err` — those failures
surface as their own runtime events, not `DslFailed`. Recording
them is a tracked follow-up; the recall-typos motivation of §2 is
already met by parse-failure journalling.

## Amendment 1 — Replay filters out app-lifecycle commands (2026-05-24)

This amendment **extends Decision §2's "each consumer filters"** to a
second filter dimension on replay, and **supersedes the runtime's
prior nested-`replay` refusal**. It was written during implementation,
after the §3 change (replay learning the journal format) made a latent
problem reachable, and is recorded with explicit user approval.

### The finding — a working `replay history.log` exposes app commands

§3 makes `replay history.log` actually work. But the journal is a
*complete* record (§1), and the runtime journals successful
app-lifecycle commands too — `save as` / `load` / `new` (project
switches), `export` / `import`, `rebuild`, `mode`. Before §3 these were
unreachable (replay died on line 1); now replay reaches them, and:

- the ones that parse to `Command::App` (`export`, `mode`, `rebuild`,
  …) hit the worker dispatch's `unreachable!()` arm — a **panic**;
- the ones whose target comes from a modal (`save as <name>` /
  `load <name>` / `new <name>`) **fail to parse** on the command line
  and **abort** the whole replay.

Either way, a normal journal breaks replay. Executing these would also
be *wrong*: they are session/project orchestration, not schema/data
reconstruction — `load` / `new` would switch projects mid-replay.

### The decision — replay re-applies only state mutations

Replay re-applies **only the schema/data write commands** (create/drop/
alter table, add/drop/change column, add/drop relationship, add/drop
index, add/drop constraint, insert/update/delete — DSL and SQL). It
**skips** every `Command::App(_)` and a nested `Command::Replay`. Reads
(`show` / `select` / `explain`) still run (harmless; they never appear
in a journal anyway).

- **All skips continue** — replay never aborts on a skippable command.
  This reverses the prior nested-`replay` *refusal*: a journal that
  happens to contain a `replay` the user once ran must not force them
  to hand-edit the log. Skipping a nested `replay` also removes the
  infinite-loop footgun by construction (the nested file is never
  re-entered).
- **Two skips warn; the rest are silent.** Skipping `import` or a
  nested `replay` can leave the replayed state *incomplete* (the
  imported data / the nested file's commands are not reconstructed),
  so each emits a `[skip]` warning (`replay.skipped_import` /
  `replay.skipped_replay`) surfaced in the replay summary. `save` /
  `save as` / `load` / `new` / `export` / `mode` / `messages` /
  `help` / `quit` / `rebuild` skip silently — omitting them changes
  nothing about the reconstructed schema/data.

### Detection

A line is classified after the §3 status filter: parse it; a
`Command::App(_)` or `Command::Replay` is skipped (warned for `import`
/ `replay`); a write/read is dispatched. The modal forms (`save as` /
`load` / `new`) do not parse, so a parse failure whose entry word is
one of those is skipped rather than aborting — any *other* parse
failure is a genuine malformed command and still stops replay with a
line-numbered error (so a typo in a hand-built script fails loudly).

### Consequences

- `AppEvent::ReplayCompleted` carries `warnings: Vec<String>`; the App
  renders them after the `[ok] replay …` summary.
- The `replay.error_nested` catalog key is removed (nested replay is
  no longer an error); `replay.skipped_import` / `replay.skipped_replay`
  are added.
- A journal whose later write command depended on `import` /
  `rebuild`-created state may not fully replay (that command can fail
  on the missing dependency). This is accepted: replay reconstructs the
  *command sequence*, and externally-sourced data is not a reproducible
  command. The `import` warning flags exactly this risk.
- ADR-0006's "`history.log` can be replayed" is now true in practice
  for the state-building subset.

## See also

- ADR-0004 — the project format `history.log` lives in.
- ADR-0006 — undo snapshots + the replay log this ADR amends.
- ADR-0007 — `export` excludes `history.log`.
- ADR-0015 — project storage runtime; §5 (history format) and
  §12 (input-history hydration) amended here.
