add handoff-10: ADR-0024 Phase F (full) steps 1-4 landed; remaining work catalogue

2026-05-15 08:39:56 +00:00
parent fa994cfb66
commit 044173bd39
1 changed files with 442 additions and 0 deletions
@@ -0,0 +1,442 @@
 # Session handoff — 2026-05-15 (10)
 Tenth handover. This session executed **ADR-0024 Phase F (full)
 steps 1–4** on top of handoff-9: the walker now drives
 highlighting, usage rendering, and the schema-list / completion
 ident-source vocabulary. `dsl::lexer`, `dsl::keyword`,
 `dsl::ident_slot`, and `dsl::usage` are deleted; their
 catalog wrappers (`parse.token.keyword.*` and
 `parse.token.punct.*`) are deleted too. Four commits in clean
 incremental steps.
 What did NOT land — and what the next session picks up — is
 captured below.
 ## State at handoff
 **Branch:** `main`. Working tree clean. **Local HEAD is
 `fa994cf`**, ahead of `origin/main` by five commits (user pushes
 asynchronously).
 Commits since handoff-9's baseline (`b3d3bdf`):
 ```
 7bdd398 ADR-0024 Phase F (full) step 1: walker-driven highlighting
 a41400e ADR-0024 Phase F (full) step 2: usage via CommandNode.usage_ids
 266b4c2 ADR-0024 Phase F (full) step 3: delete legacy parser modules
 fa994cf ADR-0024 Phase F (full) step 4: catalog token-keyword cleanup
 ```
 **Tests:** **806 passing, 0 failing, 1 ignored** (was 844 at
 handoff-9; the net drop reflects the removal of legacy-module
 tests — ~45 tests in `dsl::lexer`, `dsl::keyword`,
 `dsl::ident_slot`, `dsl::usage` — minus 16 new walker
 highlight tests). The ignored test is still the same `\`\`\`ignore`
 doc-test in `src/friendly/mod.rs`.
 **Clippy:** clean with `nursery` lints + `-D warnings`.
 **Cargo.toml:** unchanged (chumsky was already gone in handoff-9).
 ## What shipped this session
 ### Step 1 — Walker-driven highlighting (commit 7bdd398)
 `input_render::base_runs` no longer calls `lex()`. The new
 `walker::highlight::highlight_runs(source) -> Vec<ByteClass>`
 combines walker `per_byte_class` output (for matched portions)
 with a byte-shape fallback (over `lex_helpers`) for trailing
 junk / unknown-command words / unterminated strings. The
 fallback handles UTF-8 codepoints byte-exact and produces an
 `Error` class for unrecognised bytes — preserves the `$` →
 `tok_error` behaviour from the lexer-driven path.
 `Theme::highlight_class_color(HighlightClass) -> Color`
 replaces `Theme::token_color(&TokenKind)`. The renderer reads
 `HighlightClass` directly off `ByteClass`.
 `#[allow(dead_code)]` is off `HighlightClass` and
 `WalkResult::per_byte_class` — they are part of the production
 path now.
 16 new walker-highlight tests pin the byte-class output for
 walks, fallbacks, multi-byte UTF-8, and the trailing-token case.
 ### Step 2 — Usage rendering via `CommandNode.usage_ids` (commit a41400e)
 `CommandNode.usage_id: Option<&'static str>` becomes
 `usage_ids: &'static [&'static str]`. Multi-form families
 (`drop`, `add`, `show`) carry every variant — the legacy
 `dsl::usage::matched_entry` returned multi-key Vec for those
 families and the walker registry now matches that.
 App-lifecycle commands had been pointed at non-existent
 `parse.usage.app.*` catalog keys (an unnoticed bug introduced
 in handoff-9 because the field was unused at the time); they
 now point at the real catalog entries.
 Two new helpers in `dsl::grammar`:
 - `usage_keys_for_input(source) -> Option<(entry_word_text, usage_ids)>`
  resolves the input's first identifier-shape token to a
  CommandNode and returns its usage_ids list. Used by
  `app::render_usage_block` (parse-error rendering) and
  `input_render::ambient_hint` (live hint panel).
 - `entry_words_alphabetised() -> Vec<&'static str>` replaces
  `dsl::usage::entry_keywords_alphabetised`.
 `dsl::usage` is deleted entirely. The "available commands:"
 fallback in `render_usage_block` formats entry words as
 `` `<word>` `` directly (replacing the `parse.token.keyword.*`
 catalog lookup; equivalent rendering).
 `parse_command` and `parse_tokens` slim down: no pre-lex pass;
 the walker scans source bytes directly. `parse_tokens` (which
 had been kept `pub` "for future I3/I4 work") is folded into
 `parse_command`.
 ### Step 3 — Legacy module deletion (commit 266b4c2)
 Deleted: `src/dsl/lexer.rs`, `src/dsl/keyword.rs`,
 `src/dsl/ident_slot.rs`.
 `IdentSource` (`dsl::grammar`) absorbs the schema-list /
 expected-label / round-trip semantics that previously lived on
 `IdentSlot`. The walker's `Expectation::Ident { source }` and
 the schema-lookup request on the database worker now share one
 enum:
 - `IdentSource::Tables`, `Columns`, `Relationships` are the
  schema-listable sources (`completes_from_schema() == true`).
 - `IdentSource::NewName` is the user-invents kind.
 - `IdentSource::Types` is the closed-set source on column-type
  slots — does not query the schema; the walker's content
  validator handles type-name validity.
 - `IdentSource::Free` is the catch-all branch in `mode` /
  `messages` value slots.
 `SchemaCache::for_slot(IdentSlot)` becomes
 `for_source(IdentSource)`; `Database::list_names_for` and
 the `Request::ListNamesFor` worker variant take `IdentSource`.
 `InvalidIdent.slot: IdentSlot` becomes
 `InvalidIdent.source: IdentSource`.
 Completion's keyword filter (was `Keyword::from_word`) becomes
 "backticked items whose payload is all ASCII alphabetic" —
 punct and digit literals (`,`, `1`) still surface through their
 own candidate sources (composite-literal, flag, schema-ident).
 The alphabetic filter excludes them from the keyword bucket.
 `friendly::keys::tests::keyword_and_punct_have_complete_token_vocabulary`
 is dropped (cross-checked enum vs. catalog completeness; both
 enums are gone).
 ### Step 4 — Catalog token-keyword cleanup (commit fa994cf)
 Dropped the 47 `parse.token.keyword.*` and 6
 `parse.token.punct.*` catalog entries (yaml + keys.rs). Nothing
 consumes them: keyword wording is produced verbatim by
 `format!("\`{word}\`")`, sourced from grammar-tree Word
 literals. Punct wording surfaces the same way via
 `Expectation::Punct(ch)`.
 Structural-class labels (`parse.token.identifier`, `.number`,
 `.string_literal`, `.flag`, `.end_of_input`) and lex-error
 wordings (`parse.token.error.{bad_flag,unknown_char,
 unterminated_string}`) stay. None of these are derivable from
 the grammar tree.
 `keys_validate_against_catalog` continues to enforce catalog ↔
 `KEYS_AND_PLACEHOLDERS` bidirectional coverage on the trimmed
 set.
 ## DEFERRED — work the next session needs to pick up
 Items grouped by priority. Roughly in order of decreasing payoff.
 ### 1. Walker-driven completion (Phase F full step 5)
 The current completion path:
 ```
 input → parse_command → ParseError::Invalid::expected: Vec<String>
       → completion.rs parses each string back to IdentSource
         via from_expected_label
 ```
 This works but loses information. The walker knows the full
 `IdentSource` of every expected slot, the `role` of every slot
 (e.g. `parent_table` vs `child_table`), the `skipped`
 expectations from any Optional that didn't engage, the cursor's
 full `MatchedPath` so far — and the bridge throws all of that
 away to render strings.
 **Migration**: add a `expected_walker: Vec<Expectation>` field
 on `ParseError::Invalid` (additive, doesn't break consumers
 yet). Or a new `walker::candidates_at(input, cursor) ->
 WalkResult` API. Update `completion.rs::candidates_at_cursor`
 to read `Expectation::Ident { source, role }` directly.
 The walker already supports `WalkBound::Position(cursor)` (the
 variant exists in `outcome.rs`); no caller passes it today.
 Adopting it lets completion see the cursor's full context.
 **Estimated cost:** one session. Smaller than the steps that
 landed this round.
 ### 2. Phase D (full): schema-aware value typing
 Unchanged from handoff-9 §2 — same scope, same blockers.
 Specifically:
 - `WalkContext::current_table`, `current_table_columns`,
  `current_column` are declared but unwritten.
 - `Node::DynamicSubgrammar(fn(&WalkContext) -> Node)` is
  declared; the walker driver returns `Failed { expected:
  vec![] }` on that branch (deliberate — catches mis-declared
  grammar). Walker dispatch needs implementing per ADR-0024
  §sub-grammars (`Box::leak` per walk, or per-walk arena).
 - `SchemaCache` carries flat `columns: Vec<String>`; per-table
  column-with-type info needs adding.
 - `Ident { source: Tables }` has no `writes_table: bool` flag;
  walker can't populate `current_table` on match.
 - Typed value slots (`int_slot`, `decimal_slot`, …) are not
  declared.
 **The user UX win this unlocks:** typed slots reject mis-shaped
 input at parse time with localised wording ("Type a date as
 'YYYY-MM-DD'") instead of bind-time errors; completion narrows
 per column type. Round-5 "value-literal hint by column type"
 becomes type-specific.
 **Sequence to implement** (from handoff-9 §2):
 1. Plumb a `SchemaCache` reference into `parse_command`; thread
   through `WalkContext::new(schema)`.
 2. Implement `Node::DynamicSubgrammar` walker dispatch.
 3. Add `writes_table: bool` (or analogous) to `Node::Ident`
   and have the walker populate `WalkContext::current_table` +
   `current_table_columns` from the schema cache when it
   matches.
 4. Implement typed value slots — content validators for each
   `Type`.
 5. Wire `column_value_list` as a `DynamicSubgrammar` that reads
   `current_table_columns` and emits a `Seq` of typed slots
   separated by commas.
 6. Update `insert` shape to use `column_value_list`.
 7. Update `update` / `delete` to use the per-column value slot
   based on `current_column`.
 **Side effect to watch:** `parse_command` becomes
 schema-dependent. Tests that exercised parse in isolation
 without a schema will need to either pass a schema cache or fall
 back to the type-unaware path (Option<&SchemaCache>).
 **Estimated cost:** 1–2 sessions, as last session estimated.
 ### 3. HintMode annotations on grammar nodes
 `HintMode::{Default | ForceProse | ProseOnly | SuppressProse}`
 is declared in `grammar/mod.rs`. Every `CommandNode` and every
 `Node::Ident` sets it to `None`. The current ad-hoc hint cases
 in `input_render.rs::ambient_hint` (`value_literal_hint_at_cursor`,
 `typing_name_at_cursor`, `invalid_ident_at_cursor`) still drive
 the hint panel.
 ADR-0024 §HintMode-per-node says these migrate to node-attached
 annotations during Phase D. They didn't (handoff-9 confirmed,
 this session didn't move them either).
 To do once Phase D lands: annotate the value-literal slots with
 `HintMode::ProseOnly("value_literal_format_hint")`, NewName
 slots with the typing-name prose, and have the hint resolver
 dispatch on the walker's per-expected-node mode. The current
 stopgap `value_literal_hint_at_cursor` becomes
 "narrow to the column's type" instead of generic.
 ### 4. Ranker hook (declared in ADR-0024, not implemented)
 ADR-0024 §ranker-layer specifies a `Ranker` function type
 between the walker's raw candidates and the hint-panel
 renderer. Default is `identity_ranker` (declaration-order
 preserved).
 **Status: not declared anywhere in code.** Future plug-in point
 for frequency-based ranking, content-aware priors, recency.
 Non-blocking.
 ### 5. Differential-test scaffolding wasn't built (carried from handoff-9)
 ADR-0024 §test-discipline §3 specified a "differential check
 during the migration window" — a test helper running both
 parsers against the input corpus, asserting identical `Command`
 output. Since chumsky is gone (removed in Phase F minimal),
 running the differential retroactively would require
 reconstructing the chumsky path from git history. Not worth it
 — the hand-curated `dsl::walker::tests` (53 tests) and the
 existing integration test suite serve the same regression-net
 role.
 ### 6. `WalkContext` writes during walk — design exists, not implemented
 Carried from handoff-9 §12. `Ident { source: Tables, writes_table:
 true }` semantics: when the ident matches, the walker writes
 `current_table` to context. Subsequent dynamic sub-grammars read
 it. Today no `Ident` node has a `writes_table` field. Adding it
 is part of Phase D §3 above.
 ### 7. `CommandNode.help_id` not consumed
 Carried from handoff-9 §13. Every `CommandNode` declares
 `help_id: Option<&'static str>` pointing into the catalog.
 **No code reads it.** Wiring up in-app help to read this field
 (replacing hand-curated `help.in_app_body` lookups) is future
 work — not blocking, not user-facing.
 ### 8. Dead `parse.token.*` catalog entries
 The five structural-class entries
 (`parse.token.identifier/number/string_literal/flag/end_of_input`)
 and three lex-error entries
 (`parse.token.error.{bad_flag,unknown_char,unterminated_string}`)
 remain in the catalog after step 4. They are unreferenced by
 production code today — the walker classifies bytes by shape
 for highlighting and emits structural / validation errors with
 catalog keys directly (e.g., `mode.unknown`).
 Conservative call this session: leave them in. They cost
 nothing, and re-introducing them would be cheap if a future
 need arises. If you decide they're truly dead, drop the entries
 from `friendly/strings/en-US.yaml` and the corresponding
 declarations from `friendly/keys.rs`.
 ## Sharp edges (carried from handoff-9 with updates)
 - **Optional backtracking on partial-match** is intentional —
  matches chumsky's `or_not` semantics. See handoff-9 for the
  asymmetry between content (no rollback) and structural (roll
  back). Unchanged.
 - **Walker's `Choice` is strictly greedy.** Unchanged.
 - **`Ident { source: Tables/Columns/Relationships }` does NOT
  validate against the schema at parse time.** Still
  shape-only. Phase D §3 unlocks schema-aware parse.
 - **`Literal(&'static str)` matches verbatim bytes with a
  word-boundary lookahead.** Unchanged.
 - **`AST builder` failures surface as
  `WalkOutcome::ValidationFailed`** with `at_eof = true`.
  Unchanged.
 - **`unknown_command_error` is the sole catch-all** for inputs
  whose first identifier-shape token isn't a registered entry
  word. Now read from `dsl::grammar::entry_words_alphabetised()`
  (was `usage::entry_keywords_alphabetised`).
 - **`q` quit alias remains gone.** Native walker alias support
  still works: adding `q` back is `aliases: &["q"]` on
  `QUIT.entry`. Walker matches either; completion surfaces only
  the primary.
 - **Path-bearing UX (replay / import / export):** unchanged
  from handoff-9 — paths with spaces use the quoted form.
 - **Highlight fallback semantics (new this session):** for
  inputs the walker doesn't engage on (no registered entry
  word, e.g., `frobulate widgets`) the byte-shape scanner
  classifies each shape as Identifier, Number, String, Flag,
  Punct, or Error. The user sees normal token colouring on the
  unknown command before the `[error]` line fires on submit.
  This matches the pre-walker behaviour.
 - **Stopgap `value_literal_hint_at_cursor` continues to fire**
  for every value-literal slot regardless of column type. Same
  wording as handoff-9. Replaced once Phase D §4 typed slots
  land.
 ## ADR index (delta vs. handoff-9)
 ```
 0019 Friendly error layer and i18n catalog
        — parse.token.keyword/punct.* entries collapsed (Phase F)
 0020 Tokenization layer for the DSL parser
        — superseded; dsl::lexer module deleted (Phase F)
 0021 Parser-as-source-of-truth for H1a
        — partial: usage info reads from CommandNode.usage_ids;
          help_id wiring deferred (handoff-10 §7)
 0022 Ambient typing assistance
        — completion still reads ParseError-string expected;
          walker-direct path pending (handoff-10 §1)
 0024 Unified grammar tree: execution plan (ACCEPTED)
        — A through F full steps 1–4 landed.
          F step 5 (walker-direct completion) + D full deferred.
 ```
 ## Repository layout (delta vs. handoff-9)
 Files deleted:
 ```
 src/dsl/lexer.rs        (598 lines)
 src/dsl/keyword.rs      (311 lines)
 src/dsl/ident_slot.rs   (140 lines)
 src/dsl/usage.rs        (318 lines)
 ```
 Files added:
 ```
 src/dsl/walker/highlight.rs   (319 lines)
 docs/handoff/20260515-handoff-10.md (this file)
 ```
 Files significantly modified:
 ```
 src/dsl/grammar/mod.rs   — usage_ids, usage_keys_for_input,
                             entry_words_alphabetised,
                             IdentSource expanded with helpers
 src/dsl/grammar/app.rs   — usage_ids tuples (corrected catalog keys)
 src/dsl/grammar/ddl.rs   — usage_ids tuples (drop/add family)
 src/dsl/grammar/data.rs  — usage_ids tuples (show family)
 src/dsl/parser.rs        — slimmed; chumsky-side parse_tokens folded in
 src/dsl/walker/mod.rs    — re-export highlight_runs
 src/dsl/walker/outcome.rs — ByteClass / WalkResult fields no longer
                             #[allow(dead_code)]
 src/dsl/mod.rs           — removed legacy module declarations
 src/app.rs               — render_usage_block via walker registry
 src/input_render.rs      — base_runs via walker::highlight_runs;
                             ambient_hint via usage_keys_for_input
 src/completion.rs        — IdentSource throughout; keyword filter
                             via ASCII-alphabetic check
 src/db.rs                — Request::ListNamesFor takes IdentSource
 src/runtime.rs           — refresh_schema_cache uses IdentSource
 src/theme.rs             — highlight_class_color; token_color removed
 src/friendly/keys.rs     — parse.token.keyword/punct.* dropped
 src/friendly/strings/en-US.yaml — same drops
 ```
 ## How to take over
 1. **Read this file.**
 2. **Read `CLAUDE.md`** for the working-style rules.
 3. **Read handoff-9** (`20260515-handoff-9.md`) for context on the
   ADR-0024 phases that landed in the prior session and any
   carry-over sharp edges.
 4. **Read ADR-0024** for the design intent. F-step-5 (walker-direct
   completion) and Phase D full are the unfinished work; their
   sketches live in §migration of that ADR.
 5. **Skim `src/dsl/grammar/mod.rs`** — the Node / Word /
   CommandNode / REGISTRY / IdentSource / HintMode contract.
 6. **Skim `src/dsl/walker/mod.rs` + `walker/highlight.rs`** — the
   walk() entry, bridge logic, and the new highlight-runs API.
 7. **Run `cargo test`** — should report 806 passing, 0 failing,
   1 ignored.
 8. **Run `cargo clippy --all-targets -- -D warnings`** — clean.
 9. **Pick a deferred item from §1–§4** and start. §1 (walker-
   driven completion) is the natural next move; it removes the
   string round-trip and unlocks richer hints. §2 (Phase D full)
   is the largest item and the biggest user-visible payoff.
 ### End-to-end smoke (current state)
 Same as handoff-9's smoke; nothing in user-visible behaviour
 changed this session. All four steps were silent refactors of
 internal machinery — same parse, same dispatch, same wording
 on errors, same colours on tokens. The catalog cleanup removed
 unused YAML entries; no string the user sees was sourced from
 them.