add handoff-10: ADR-0024 Phase F (full) steps 1-4 landed; remaining work catalogue

docs/handoff/20260515-handoff-10.md

@@ -0,0 +1,442 @@
+# Session handoff — 2026-05-15 (10)
+
+Tenth handover. This session executed **ADR-0024 Phase F (full)
+steps 1–4** on top of handoff-9: the walker now drives
+highlighting, usage rendering, and the schema-list / completion
+ident-source vocabulary. `dsl::lexer`, `dsl::keyword`,
+`dsl::ident_slot`, and `dsl::usage` are deleted; their
+catalog wrappers (`parse.token.keyword.*` and
+`parse.token.punct.*`) are deleted too. Four commits in clean
+incremental steps.
+
+What did NOT land — and what the next session picks up — is
+captured below.
+
+## State at handoff
+
+**Branch:** `main`. Working tree clean. **Local HEAD is
+`fa994cf`**, ahead of `origin/main` by five commits (user pushes
+asynchronously).
+
+Commits since handoff-9's baseline (`b3d3bdf`):
+
+```
+7bdd398 ADR-0024 Phase F (full) step 1: walker-driven highlighting
+a41400e ADR-0024 Phase F (full) step 2: usage via CommandNode.usage_ids
+266b4c2 ADR-0024 Phase F (full) step 3: delete legacy parser modules
+fa994cf ADR-0024 Phase F (full) step 4: catalog token-keyword cleanup
+```
+
+**Tests:** **806 passing, 0 failing, 1 ignored** (was 844 at
+handoff-9; the net drop reflects the removal of legacy-module
+tests — ~45 tests in `dsl::lexer`, `dsl::keyword`,
+`dsl::ident_slot`, `dsl::usage` — minus 16 new walker
+highlight tests). The ignored test is still the same `\`\`\`ignore`
+doc-test in `src/friendly/mod.rs`.
+
+**Clippy:** clean with `nursery` lints + `-D warnings`.
+
+**Cargo.toml:** unchanged (chumsky was already gone in handoff-9).
+
+## What shipped this session
+
+### Step 1 — Walker-driven highlighting (commit 7bdd398)
+
+`input_render::base_runs` no longer calls `lex()`. The new
+`walker::highlight::highlight_runs(source) -> Vec<ByteClass>`
+combines walker `per_byte_class` output (for matched portions)
+with a byte-shape fallback (over `lex_helpers`) for trailing
+junk / unknown-command words / unterminated strings. The
+fallback handles UTF-8 codepoints byte-exact and produces an
+`Error` class for unrecognised bytes — preserves the `$` →
+`tok_error` behaviour from the lexer-driven path.
+
+`Theme::highlight_class_color(HighlightClass) -> Color`
+replaces `Theme::token_color(&TokenKind)`. The renderer reads
+`HighlightClass` directly off `ByteClass`.
+
+`#[allow(dead_code)]` is off `HighlightClass` and
+`WalkResult::per_byte_class` — they are part of the production
+path now.
+
+16 new walker-highlight tests pin the byte-class output for
+walks, fallbacks, multi-byte UTF-8, and the trailing-token case.
+
+### Step 2 — Usage rendering via `CommandNode.usage_ids` (commit a41400e)
+
+`CommandNode.usage_id: Option<&'static str>` becomes
+`usage_ids: &'static [&'static str]`. Multi-form families
+(`drop`, `add`, `show`) carry every variant — the legacy
+`dsl::usage::matched_entry` returned multi-key Vec for those
+families and the walker registry now matches that.
+
+App-lifecycle commands had been pointed at non-existent
+`parse.usage.app.*` catalog keys (an unnoticed bug introduced
+in handoff-9 because the field was unused at the time); they
+now point at the real catalog entries.
+
+Two new helpers in `dsl::grammar`:
+
+- `usage_keys_for_input(source) -> Option<(entry_word_text, usage_ids)>`
+  resolves the input's first identifier-shape token to a
+  CommandNode and returns its usage_ids list. Used by
+  `app::render_usage_block` (parse-error rendering) and
+  `input_render::ambient_hint` (live hint panel).
+- `entry_words_alphabetised() -> Vec<&'static str>` replaces
+  `dsl::usage::entry_keywords_alphabetised`.
+
+`dsl::usage` is deleted entirely. The "available commands:"
+fallback in `render_usage_block` formats entry words as
+`` `<word>` `` directly (replacing the `parse.token.keyword.*`
+catalog lookup; equivalent rendering).
+
+`parse_command` and `parse_tokens` slim down: no pre-lex pass;
+the walker scans source bytes directly. `parse_tokens` (which
+had been kept `pub` "for future I3/I4 work") is folded into
+`parse_command`.
+
+### Step 3 — Legacy module deletion (commit 266b4c2)
+
+Deleted: `src/dsl/lexer.rs`, `src/dsl/keyword.rs`,
+`src/dsl/ident_slot.rs`.
+
+`IdentSource` (`dsl::grammar`) absorbs the schema-list /
+expected-label / round-trip semantics that previously lived on
+`IdentSlot`. The walker's `Expectation::Ident { source }` and
+the schema-lookup request on the database worker now share one
+enum:
+
+- `IdentSource::Tables`, `Columns`, `Relationships` are the
+  schema-listable sources (`completes_from_schema() == true`).
+- `IdentSource::NewName` is the user-invents kind.
+- `IdentSource::Types` is the closed-set source on column-type
+  slots — does not query the schema; the walker's content
+  validator handles type-name validity.
+- `IdentSource::Free` is the catch-all branch in `mode` /
+  `messages` value slots.
+
+`SchemaCache::for_slot(IdentSlot)` becomes
+`for_source(IdentSource)`; `Database::list_names_for` and
+the `Request::ListNamesFor` worker variant take `IdentSource`.
+
+`InvalidIdent.slot: IdentSlot` becomes
+`InvalidIdent.source: IdentSource`.
+
+Completion's keyword filter (was `Keyword::from_word`) becomes
+"backticked items whose payload is all ASCII alphabetic" —
+punct and digit literals (`,`, `1`) still surface through their
+own candidate sources (composite-literal, flag, schema-ident).
+The alphabetic filter excludes them from the keyword bucket.
+
+`friendly::keys::tests::keyword_and_punct_have_complete_token_vocabulary`
+is dropped (cross-checked enum vs. catalog completeness; both
+enums are gone).
+
+### Step 4 — Catalog token-keyword cleanup (commit fa994cf)
+
+Dropped the 47 `parse.token.keyword.*` and 6
+`parse.token.punct.*` catalog entries (yaml + keys.rs). Nothing
+consumes them: keyword wording is produced verbatim by
+`format!("\`{word}\`")`, sourced from grammar-tree Word
+literals. Punct wording surfaces the same way via
+`Expectation::Punct(ch)`.
+
+Structural-class labels (`parse.token.identifier`, `.number`,
+`.string_literal`, `.flag`, `.end_of_input`) and lex-error
+wordings (`parse.token.error.{bad_flag,unknown_char,
+unterminated_string}`) stay. None of these are derivable from
+the grammar tree.
+
+`keys_validate_against_catalog` continues to enforce catalog ↔
+`KEYS_AND_PLACEHOLDERS` bidirectional coverage on the trimmed
+set.
+
+## DEFERRED — work the next session needs to pick up
+
+Items grouped by priority. Roughly in order of decreasing payoff.
+
+### 1. Walker-driven completion (Phase F full step 5)
+
+The current completion path:
+
+```
+input → parse_command → ParseError::Invalid::expected: Vec<String>
+       → completion.rs parses each string back to IdentSource
+         via from_expected_label
+```
+
+This works but loses information. The walker knows the full
+`IdentSource` of every expected slot, the `role` of every slot
+(e.g. `parent_table` vs `child_table`), the `skipped`
+expectations from any Optional that didn't engage, the cursor's
+full `MatchedPath` so far — and the bridge throws all of that
+away to render strings.
+
+**Migration**: add a `expected_walker: Vec<Expectation>` field
+on `ParseError::Invalid` (additive, doesn't break consumers
+yet). Or a new `walker::candidates_at(input, cursor) ->
+WalkResult` API. Update `completion.rs::candidates_at_cursor`
+to read `Expectation::Ident { source, role }` directly.
+
+The walker already supports `WalkBound::Position(cursor)` (the
+variant exists in `outcome.rs`); no caller passes it today.
+Adopting it lets completion see the cursor's full context.
+
+**Estimated cost:** one session. Smaller than the steps that
+landed this round.
+
+### 2. Phase D (full): schema-aware value typing
+
+Unchanged from handoff-9 §2 — same scope, same blockers.
+Specifically:
+
+- `WalkContext::current_table`, `current_table_columns`,
+  `current_column` are declared but unwritten.
+- `Node::DynamicSubgrammar(fn(&WalkContext) -> Node)` is
+  declared; the walker driver returns `Failed { expected:
+  vec![] }` on that branch (deliberate — catches mis-declared
+  grammar). Walker dispatch needs implementing per ADR-0024
+  §sub-grammars (`Box::leak` per walk, or per-walk arena).
+- `SchemaCache` carries flat `columns: Vec<String>`; per-table
+  column-with-type info needs adding.
+- `Ident { source: Tables }` has no `writes_table: bool` flag;
+  walker can't populate `current_table` on match.
+- Typed value slots (`int_slot`, `decimal_slot`, …) are not
+  declared.
+
+**The user UX win this unlocks:** typed slots reject mis-shaped
+input at parse time with localised wording ("Type a date as
+'YYYY-MM-DD'") instead of bind-time errors; completion narrows
+per column type. Round-5 "value-literal hint by column type"
+becomes type-specific.
+
+**Sequence to implement** (from handoff-9 §2):
+
+1. Plumb a `SchemaCache` reference into `parse_command`; thread
+   through `WalkContext::new(schema)`.
+2. Implement `Node::DynamicSubgrammar` walker dispatch.
+3. Add `writes_table: bool` (or analogous) to `Node::Ident`
+   and have the walker populate `WalkContext::current_table` +
+   `current_table_columns` from the schema cache when it
+   matches.
+4. Implement typed value slots — content validators for each
+   `Type`.
+5. Wire `column_value_list` as a `DynamicSubgrammar` that reads
+   `current_table_columns` and emits a `Seq` of typed slots
+   separated by commas.
+6. Update `insert` shape to use `column_value_list`.
+7. Update `update` / `delete` to use the per-column value slot
+   based on `current_column`.
+
+**Side effect to watch:** `parse_command` becomes
+schema-dependent. Tests that exercised parse in isolation
+without a schema will need to either pass a schema cache or fall
+back to the type-unaware path (Option<&SchemaCache>).
+
+**Estimated cost:** 1–2 sessions, as last session estimated.
+
+### 3. HintMode annotations on grammar nodes
+
+`HintMode::{Default | ForceProse | ProseOnly | SuppressProse}`
+is declared in `grammar/mod.rs`. Every `CommandNode` and every
+`Node::Ident` sets it to `None`. The current ad-hoc hint cases
+in `input_render.rs::ambient_hint` (`value_literal_hint_at_cursor`,
+`typing_name_at_cursor`, `invalid_ident_at_cursor`) still drive
+the hint panel.
+
+ADR-0024 §HintMode-per-node says these migrate to node-attached
+annotations during Phase D. They didn't (handoff-9 confirmed,
+this session didn't move them either).
+
+To do once Phase D lands: annotate the value-literal slots with
+`HintMode::ProseOnly("value_literal_format_hint")`, NewName
+slots with the typing-name prose, and have the hint resolver
+dispatch on the walker's per-expected-node mode. The current
+stopgap `value_literal_hint_at_cursor` becomes
+"narrow to the column's type" instead of generic.
+
+### 4. Ranker hook (declared in ADR-0024, not implemented)
+
+ADR-0024 §ranker-layer specifies a `Ranker` function type
+between the walker's raw candidates and the hint-panel
+renderer. Default is `identity_ranker` (declaration-order
+preserved).
+
+**Status: not declared anywhere in code.** Future plug-in point
+for frequency-based ranking, content-aware priors, recency.
+Non-blocking.
+
+### 5. Differential-test scaffolding wasn't built (carried from handoff-9)
+
+ADR-0024 §test-discipline §3 specified a "differential check
+during the migration window" — a test helper running both
+parsers against the input corpus, asserting identical `Command`
+output. Since chumsky is gone (removed in Phase F minimal),
+running the differential retroactively would require
+reconstructing the chumsky path from git history. Not worth it
+— the hand-curated `dsl::walker::tests` (53 tests) and the
+existing integration test suite serve the same regression-net
+role.
+
+### 6. `WalkContext` writes during walk — design exists, not implemented
+
+Carried from handoff-9 §12. `Ident { source: Tables, writes_table:
+true }` semantics: when the ident matches, the walker writes
+`current_table` to context. Subsequent dynamic sub-grammars read
+it. Today no `Ident` node has a `writes_table` field. Adding it
+is part of Phase D §3 above.
+
+### 7. `CommandNode.help_id` not consumed
+
+Carried from handoff-9 §13. Every `CommandNode` declares
+`help_id: Option<&'static str>` pointing into the catalog.
+**No code reads it.** Wiring up in-app help to read this field
+(replacing hand-curated `help.in_app_body` lookups) is future
+work — not blocking, not user-facing.
+
+### 8. Dead `parse.token.*` catalog entries
+
+The five structural-class entries
+(`parse.token.identifier/number/string_literal/flag/end_of_input`)
+and three lex-error entries
+(`parse.token.error.{bad_flag,unknown_char,unterminated_string}`)
+remain in the catalog after step 4. They are unreferenced by
+production code today — the walker classifies bytes by shape
+for highlighting and emits structural / validation errors with
+catalog keys directly (e.g., `mode.unknown`).
+
+Conservative call this session: leave them in. They cost
+nothing, and re-introducing them would be cheap if a future
+need arises. If you decide they're truly dead, drop the entries
+from `friendly/strings/en-US.yaml` and the corresponding
+declarations from `friendly/keys.rs`.
+
+## Sharp edges (carried from handoff-9 with updates)
+
+- **Optional backtracking on partial-match** is intentional —
+  matches chumsky's `or_not` semantics. See handoff-9 for the
+  asymmetry between content (no rollback) and structural (roll
+  back). Unchanged.
+- **Walker's `Choice` is strictly greedy.** Unchanged.
+- **`Ident { source: Tables/Columns/Relationships }` does NOT
+  validate against the schema at parse time.** Still
+  shape-only. Phase D §3 unlocks schema-aware parse.
+- **`Literal(&'static str)` matches verbatim bytes with a
+  word-boundary lookahead.** Unchanged.
+- **`AST builder` failures surface as
+  `WalkOutcome::ValidationFailed`** with `at_eof = true`.
+  Unchanged.
+- **`unknown_command_error` is the sole catch-all** for inputs
+  whose first identifier-shape token isn't a registered entry
+  word. Now read from `dsl::grammar::entry_words_alphabetised()`
+  (was `usage::entry_keywords_alphabetised`).
+- **`q` quit alias remains gone.** Native walker alias support
+  still works: adding `q` back is `aliases: &["q"]` on
+  `QUIT.entry`. Walker matches either; completion surfaces only
+  the primary.
+- **Path-bearing UX (replay / import / export):** unchanged
+  from handoff-9 — paths with spaces use the quoted form.
+- **Highlight fallback semantics (new this session):** for
+  inputs the walker doesn't engage on (no registered entry
+  word, e.g., `frobulate widgets`) the byte-shape scanner
+  classifies each shape as Identifier, Number, String, Flag,
+  Punct, or Error. The user sees normal token colouring on the
+  unknown command before the `[error]` line fires on submit.
+  This matches the pre-walker behaviour.
+- **Stopgap `value_literal_hint_at_cursor` continues to fire**
+  for every value-literal slot regardless of column type. Same
+  wording as handoff-9. Replaced once Phase D §4 typed slots
+  land.
+
+## ADR index (delta vs. handoff-9)
+
+```
+0019 Friendly error layer and i18n catalog
+        — parse.token.keyword/punct.* entries collapsed (Phase F)
+0020 Tokenization layer for the DSL parser
+        — superseded; dsl::lexer module deleted (Phase F)
+0021 Parser-as-source-of-truth for H1a
+        — partial: usage info reads from CommandNode.usage_ids;
+          help_id wiring deferred (handoff-10 §7)
+0022 Ambient typing assistance
+        — completion still reads ParseError-string expected;
+          walker-direct path pending (handoff-10 §1)
+0024 Unified grammar tree: execution plan (ACCEPTED)
+        — A through F full steps 1–4 landed.
+          F step 5 (walker-direct completion) + D full deferred.
+```
+
+## Repository layout (delta vs. handoff-9)
+
+Files deleted:
+
+```
+src/dsl/lexer.rs        (598 lines)
+src/dsl/keyword.rs      (311 lines)
+src/dsl/ident_slot.rs   (140 lines)
+src/dsl/usage.rs        (318 lines)
+```
+
+Files added:
+
+```
+src/dsl/walker/highlight.rs   (319 lines)
+docs/handoff/20260515-handoff-10.md (this file)
+```
+
+Files significantly modified:
+
+```
+src/dsl/grammar/mod.rs   — usage_ids, usage_keys_for_input,
+                             entry_words_alphabetised,
+                             IdentSource expanded with helpers
+src/dsl/grammar/app.rs   — usage_ids tuples (corrected catalog keys)
+src/dsl/grammar/ddl.rs   — usage_ids tuples (drop/add family)
+src/dsl/grammar/data.rs  — usage_ids tuples (show family)
+src/dsl/parser.rs        — slimmed; chumsky-side parse_tokens folded in
+src/dsl/walker/mod.rs    — re-export highlight_runs
+src/dsl/walker/outcome.rs — ByteClass / WalkResult fields no longer
+                             #[allow(dead_code)]
+src/dsl/mod.rs           — removed legacy module declarations
+src/app.rs               — render_usage_block via walker registry
+src/input_render.rs      — base_runs via walker::highlight_runs;
+                             ambient_hint via usage_keys_for_input
+src/completion.rs        — IdentSource throughout; keyword filter
+                             via ASCII-alphabetic check
+src/db.rs                — Request::ListNamesFor takes IdentSource
+src/runtime.rs           — refresh_schema_cache uses IdentSource
+src/theme.rs             — highlight_class_color; token_color removed
+src/friendly/keys.rs     — parse.token.keyword/punct.* dropped
+src/friendly/strings/en-US.yaml — same drops
+```
+
+## How to take over
+
+1. **Read this file.**
+2. **Read `CLAUDE.md`** for the working-style rules.
+3. **Read handoff-9** (`20260515-handoff-9.md`) for context on the
+   ADR-0024 phases that landed in the prior session and any
+   carry-over sharp edges.
+4. **Read ADR-0024** for the design intent. F-step-5 (walker-direct
+   completion) and Phase D full are the unfinished work; their
+   sketches live in §migration of that ADR.
+5. **Skim `src/dsl/grammar/mod.rs`** — the Node / Word /
+   CommandNode / REGISTRY / IdentSource / HintMode contract.
+6. **Skim `src/dsl/walker/mod.rs` + `walker/highlight.rs`** — the
+   walk() entry, bridge logic, and the new highlight-runs API.
+7. **Run `cargo test`** — should report 806 passing, 0 failing,
+   1 ignored.
+8. **Run `cargo clippy --all-targets -- -D warnings`** — clean.
+9. **Pick a deferred item from §1–§4** and start. §1 (walker-
+   driven completion) is the natural next move; it removes the
+   string round-trip and unlocks richer hints. §2 (Phase D full)
+   is the largest item and the biggest user-visible payoff.
+
+### End-to-end smoke (current state)
+
+Same as handoff-9's smoke; nothing in user-visible behaviour
+changed this session. All four steps were silent refactors of
+internal machinery — same parse, same dispatch, same wording
+on errors, same colours on tokens. The catalog cleanup removed
+unused YAML entries; no string the user sees was sourced from
+them.