rdbms-playground/docs/plans/20260613-issue-30-top-of-chain-journaling.md

# Plan — issue #30: mode-tagged history + top-of-chain journaling

**Status:** draft for `/runda` review (2026-06-13).
**Issue:** #30 — advanced history reusable in simple mode (prepend `:`),
and the bug: the `:` one-shot prefix is lost across sessions.
**ADR:** ADR-0052 (new); amends ADR-0015 §6, ADR-0034, ADR-0040;
references ADR-0003.

## 1. Goal & root cause

Two coupled needs, one root cause — **history entries carry no mode**:
- **Bug:** the in-memory ring stores the raw `:select 1`, but the worker
  journals the *stripped* `select 1`, so cross-session the `:` is lost
  and the command recalls bare (unusable in simple mode).
- **Feature:** persistent-advanced commands (`select 1` typed in advanced
  mode) can't be told apart from simple DSL, so they can't be offered
  back with a `:` in simple mode.

Fix: **record the submission mode per entry** (status tag `:adv`), keep
the on-disk `source` canonical, and have **recall prepend/strip `:`** for
the current mode.

## 2. The architecture insight (why this plan is shaped this way)

Journaling **success** lives deep in the worker: `finalize_persistence`
(db.rs:3096-3099) writes `history.log` *inside the db transaction, before
`tx.commit()`*, alongside yaml/csv — plus four no-op-skip sites and three
read-only helpers. **Failure** journaling already lives at the top
(runtime.rs:484-495, best-effort). Threading the mode *down* to the
worker would mean ~30 `Request` variants + `Database` methods +
`execute_command_typed` arms — because the journal write is far from
where the mode is known.

So instead: **move success journaling up to the dispatch layer**, next to
where failure journaling already is and where mode + outcome + source are
all in scope. The mode then needs no plumbing. This is the correct
separation anyway — `history.log` is an append-only *journal of what was
typed*, not *state*; the state sources (yaml/csv/db) stay atomic in the
worker.

### Semantic changes this entails (must be vetted)

1. **history.log leaves the worker transaction** (amends ADR-0015 §6).
   `commit-db-last` still governs yaml/csv/db (the state); the journal is
   written *after* the worker replies (i.e. after `tx.commit`), at the
   dispatch layer.
2. **Success-journal write failure: fatal → best-effort** (amends
   ADR-0040). Today a failed `history.log` write on a *successful*
   command rolls the command back and shows a fatal banner. After: the
   command stays committed; the journal write is best-effort (logged +
   ignored), exactly like the failure path already is. The two journal
   paths become *consistent*.
3. **Consequence:** on a rare journal-write failure (disk full /
   permissions) a successful command is applied but may be missing from
   `history.log` — not recallable next session, not replayable. The state
   (yaml/csv/db) is unaffected and consistent. This is a graceful
   degradation, not corruption, and is logged. (Today the same disk-full
   instead kills the app mid-command.)

**Open question for review/user:** is trading "fatal on journal-write
failure" for "best-effort, command still succeeds" acceptable? The plan
assumes **yes** (a journal is auxiliary; killing the app over it is worse
UX). If not, journaling must stay coupled in the worker and we pay the
~30-site mode plumbing instead.

## 3. On-disk format (mode tag in status — already chosen + partly built)

Record stays `<ts>|<status>|<source>`; the **status token** gains an
optional `:adv` suffix (ADR-0052). `source` stays canonical so replay is
unaffected.

| Submission | Success | Failure |
|---|---|---|
| Simple / app command | `ok` | `err` |
| Advanced (SQL, persistent or one-shot) | `ok:adv` | `err:adv` |

**Done already** (history.rs / mod.rs):
- `status_token(base, advanced)`, `parse_status(status) -> (is_ok, advanced)`.
- `parse_record_source` reconstructs `": {cmd}"` for `:adv` records.
- `parse_journal_record.status_is_ok` via `parse_status` (so `ok:adv` replays).
- `append_history(text, advanced)`, `append_history_failure(text, advanced)`.

Back-compat: old `ok`/`err` logs → simple; nothing migrates.

## 4. In-memory ring & recall (app.rs) — the #30 behaviour

The ring stays `Vec<String>`. An **advanced** entry is stored in its
`: `-prefixed simple-mode runnable form (matching the existing in-session
one-shot ring); a **simple** entry bare. A leading `:` unambiguously
marks advanced (simple DSL can never start with `:`).

- **`submit`** (app.rs:1704): compute `effective_input` + `submission_mode`,
  parse once for the app-command check (already done at 1751), then build
  the ring line. The **`advanced` flag excludes app commands** —
  `advanced = submission_mode.is_advanced() && !is_app_command` — because
  app commands (`undo`, `mode …`, `save as`, …) run in *any* mode and must
  **not** get a `:` on recall. Ring line: `": " + effective_input` if
  `advanced`, else `effective_input`; `push_history(&ring_line)`. (Today it
  pushes the raw `trimmed` *before* stripping; the reorder also drops a
  bare `:`, which executed nothing, and is what lets the app-command check
  precede the push.) `ExecuteDsl.source` stays the **canonical**
  `effective_input`.
  - *Why the app-command exclusion matters (DA finding):* without it,
    `: save as foo` (an app command via the one-shot) would store `: save
    as foo` in the ring but journal `save as foo` (app commands journal
    simple at their own sites, §5) — the very in-session-vs-cross-session
    divergence #30 is fixing, re-introduced for app commands. Excluding
    them keeps ring and disk agreeing (both bare).
- **`history_back` / `history_forward`**: after cloning the stored entry
  into `self.input`, strip a leading `:` **iff `self.mode == Advanced`**
  (so an advanced entry runs as bare SQL in advanced mode, and as `: …`
  one-shot in simple mode). A small helper `recall_display(stored)`.
- `seed_history` / `ProjectSwitched` payload: **unchanged** (`Vec<String>`);
  hydration already returns the `: `-prefixed form (§3).

Recall matrix:
| entry \ current mode | Simple | Advanced |
|---|---|---|
| advanced (`: select 1`) | `: select 1` (one-shot) | `select 1` (SQL) |
| simple (`create …`) | `create …` | `create …` |

## 5. Move success journaling worker → dispatch layer

**Remove** (worker stops journaling success):
- `finalize_persistence` history write (db.rs:3096-3099). Keep yaml/csv.
  The now-unused `source` param: remove it + drop the arg at its ~30
  callers (mechanical, compiler-guided). (Handlers keep their own
  `source` for `snapshot_then`.)
- The 4 no-op-skip `append_history` (db.rs:2267, 2311, 2524, 2560) — these
  outcomes (`SchemaSkipped` etc.) are `Ok` at the dispatch layer, so the
  new top-level journal covers them.
- The 3 read-only helper `append_history` (db.rs:8372 show table, 9996
  show data, 10014 select) — `Ok(Query)`/`Ok(ShowList)` at the top.

**Add** (dispatch-layer journaling, all best-effort + logged):
- **`spawn_dsl_dispatch`** (runtime.rs ~1433): pass `project_path` in;
  after `execute_command_typed`, `if outcome.is_ok() {
  Persistence::new(path).append_history(&source_for_journal,
  submission_mode.is_advanced()) }`. (Failures stay in the existing path,
  §6 — no double-journal, since Ok and Err are exclusive.)
- **`run_replay`** (runtime.rs ~2540): after each line's
  `execute_command_typed`, `if outcome.is_ok() { append_history(
  &command_text, false) }` — replay is mode-agnostic, journalled
  **simple**. (Preserves ADR-0034 §3 "replayed sub-commands land in
  history"; a replayed advanced command re-journals without `:adv` — a
  documented OOS, not a regression: today it re-journals as plain `ok`.)
- **`spawn_rebuild`** (runtime.rs ~503): after a successful rebuild,
  `append_history("rebuild"/source, false)`. (Rebuild journalled via
  `finalize_persistence` today; that write is gone, so add it here.)

**Unchanged** (already at the dispatch layer, app commands):
- `perform_switch` (974: save-as/load/new) and `spawn_export` (1043) —
  already best-effort `append_history(&source)`; add the new `advanced`
  arg as `false` (app commands run in any mode → no `:` needed on recall;
  this also fixes the would-be "redundant `: undo`" — app commands
  journal **simple** because they're dispatched here, never via
  `ExecuteDsl`/the spawn).
- `undo`/`redo`/`copy`/`help`/`quit`: not journalled today; unchanged.
- The **`replay` command itself**: dispatched as `Action::Replay`, never
  reaches the spawn → not journalled (preserves the ADR-0034 §3 exclusion
  without extra work); nested `replay` skip in `run_replay` unchanged.

### DA-confirmed design choice: split, don't unify

Success journals in the spawn (`Ok` arm); **all** failures stay in the
existing App→`JournalFailure`→runtime path (just gaining the mode).
Considered and rejected: moving worker-rejection failures into the spawn
too (to "unify"). It doesn't actually unify — parse failures never reach
the spawn, so they'd stay in the App path regardless — and it adds a
double-journal hazard (must also strip the App's `DslFailed`→
`JournalFailure` emission). The split keeps the failure path **untouched
in structure** (lowest risk); `Ok`/`Err` are exclusive so there is no
double-journal. **Verified safe:** undo/redo never touches `history.log`
(the snapshot copies db+yaml+csv only, undo.rs:15-16), and `snapshot_then`'s
redo-clear keys on `source.is_some()`, independent of journaling — so
removing the worker journal write does not perturb undo/snapshot at all.

## 6. Failure journaling — add the mode (location unchanged)

Keep both failure origins where they are (best-effort, dispatch/App
layer); thread the mode so they tag `err:adv`:
- **`Action::JournalFailure`** (action.rs:42): add `advanced: bool` (or
  `submission_mode`).
- **`AppEvent::DslFailed`** (event.rs): add `submission_mode` (the
  worker-rejection path — the App can't recover the mode from an async
  reply otherwise).
- **App**: the parse-failure path (`dispatch_dsl` Err arm) has
  `submission_mode` directly; the `DslFailed` handler reads it off the
  event. Both emit `JournalFailure { source, advanced }`.
- **runtime.rs:492**: `append_history_failure(&source, advanced)`.

## 7. Tests

- **history.rs (Tier-1):** `status_token`/`parse_status` round-trip;
  `read_recent_sources` reconstructs `": …"` for `:adv` and leaves
  `ok`/`err` bare; `status_is_ok` true for `ok` & `ok:adv`; old-log
  back-compat.
- **app.rs (Tier-1):** advanced submission stored `: `-prefixed; recall
  prepends in simple / strips in advanced; simple bare in both; bare `:`
  not stored; a parse-failure is still recallable; dedup/cap hold.
- **iteration6_resume_history (Tier-3) — headline regression:** journal
  an advanced command (`append_history(text, true)`), hydrate, recall in
  simple → `: …`; and the full bug repro through `submit` + journal +
  hydrate if feasible.
- **replay_command (Tier-3):** replayed commands still land in
  history.log (now via `run_replay`'s call); the `replay`-self-exclusion
  + nested-skip still hold; advanced lines replay (status `ok:adv`
  treated as ok).
- **Journaling relocation:** a success no longer fatals on a journal
  write failure (best-effort) — if cheaply testable; at minimum a worker
  test that previously asserted worker-side journaling is updated/removed.
- **Update mechanical call sites:** `append_history(_, advanced)` /
  `append_history_failure(_, advanced)` at the db.rs inline tests
  (8372/9996/10014/11324 — likely now removed with the production sites),
  iteration6 (144-170), mod.rs (600).

## 8. ADR work

- **ADR-0052 (new):** the #30 feature + bug, the status-tag format, the
  `: `-prefixed ring + recall, AND the journaling relocation (it's the
  enabling refactor). Forks: status-tag format; unified scope;
  dispatch-layer journaling (best-effort).
- **ADR-0015 §6 amendment:** history.log out of the worker transaction;
  commit-db-last now scopes yaml/csv/db; journal is a dispatch-layer
  best-effort side-record.
- **ADR-0034 amendment:** journaling location (dispatch layer);
  status-field `:adv` extension (it already reserved the field).
- **ADR-0040 amendment:** a success-path journal-write failure is no
  longer fatal — best-effort, consistent with the failure path.
- README index upkeep for every ADR touched.

## 9. Risks / watch-list

- **Double-journaling**: ensure Ok→spawn and Err→App-path stay exclusive;
  do NOT also leave a worker journal.
- **Under/over-journaling vs today**: top-level "journal on every Ok"
  must match today's "journal every command with a source" — verified:
  reads + skips are Ok outcomes, internal ops never reach the spawn.
- **finalize_persistence source-param removal**: 30 mechanical call-site
  edits; compiler-guided.
- **Replay re-journal mode fidelity**: replayed advanced commands
  re-journal as simple (OOS, not a regression).
- **best-effort journal**: rare write-failure leaves a command unjournaled
  (logged). User decision (§2 open question).
- **app-command mode**: journalled simple by construction (dispatched
  outside the spawn) — this is correct (they run in any mode), and
  resolves the earlier "redundant `: undo`" worry.