diff --git a/docs/handoff/20260515-handoff-10.md b/docs/handoff/20260515-handoff-10.md new file mode 100644 index 0000000..20169b7 --- /dev/null +++ b/docs/handoff/20260515-handoff-10.md @@ -0,0 +1,442 @@ +# Session handoff — 2026-05-15 (10) + +Tenth handover. This session executed **ADR-0024 Phase F (full) +steps 1–4** on top of handoff-9: the walker now drives +highlighting, usage rendering, and the schema-list / completion +ident-source vocabulary. `dsl::lexer`, `dsl::keyword`, +`dsl::ident_slot`, and `dsl::usage` are deleted; their +catalog wrappers (`parse.token.keyword.*` and +`parse.token.punct.*`) are deleted too. Four commits in clean +incremental steps. + +What did NOT land — and what the next session picks up — is +captured below. + +## State at handoff + +**Branch:** `main`. Working tree clean. **Local HEAD is +`fa994cf`**, ahead of `origin/main` by five commits (user pushes +asynchronously). + +Commits since handoff-9's baseline (`b3d3bdf`): + +``` +7bdd398 ADR-0024 Phase F (full) step 1: walker-driven highlighting +a41400e ADR-0024 Phase F (full) step 2: usage via CommandNode.usage_ids +266b4c2 ADR-0024 Phase F (full) step 3: delete legacy parser modules +fa994cf ADR-0024 Phase F (full) step 4: catalog token-keyword cleanup +``` + +**Tests:** **806 passing, 0 failing, 1 ignored** (was 844 at +handoff-9; the net drop reflects the removal of legacy-module +tests — ~45 tests in `dsl::lexer`, `dsl::keyword`, +`dsl::ident_slot`, `dsl::usage` — minus 16 new walker +highlight tests). The ignored test is still the same `\`\`\`ignore` +doc-test in `src/friendly/mod.rs`. + +**Clippy:** clean with `nursery` lints + `-D warnings`. + +**Cargo.toml:** unchanged (chumsky was already gone in handoff-9). + +## What shipped this session + +### Step 1 — Walker-driven highlighting (commit 7bdd398) + +`input_render::base_runs` no longer calls `lex()`. The new +`walker::highlight::highlight_runs(source) -> Vec` +combines walker `per_byte_class` output (for matched portions) +with a byte-shape fallback (over `lex_helpers`) for trailing +junk / unknown-command words / unterminated strings. The +fallback handles UTF-8 codepoints byte-exact and produces an +`Error` class for unrecognised bytes — preserves the `$` → +`tok_error` behaviour from the lexer-driven path. + +`Theme::highlight_class_color(HighlightClass) -> Color` +replaces `Theme::token_color(&TokenKind)`. The renderer reads +`HighlightClass` directly off `ByteClass`. + +`#[allow(dead_code)]` is off `HighlightClass` and +`WalkResult::per_byte_class` — they are part of the production +path now. + +16 new walker-highlight tests pin the byte-class output for +walks, fallbacks, multi-byte UTF-8, and the trailing-token case. + +### Step 2 — Usage rendering via `CommandNode.usage_ids` (commit a41400e) + +`CommandNode.usage_id: Option<&'static str>` becomes +`usage_ids: &'static [&'static str]`. Multi-form families +(`drop`, `add`, `show`) carry every variant — the legacy +`dsl::usage::matched_entry` returned multi-key Vec for those +families and the walker registry now matches that. + +App-lifecycle commands had been pointed at non-existent +`parse.usage.app.*` catalog keys (an unnoticed bug introduced +in handoff-9 because the field was unused at the time); they +now point at the real catalog entries. + +Two new helpers in `dsl::grammar`: + +- `usage_keys_for_input(source) -> Option<(entry_word_text, usage_ids)>` + resolves the input's first identifier-shape token to a + CommandNode and returns its usage_ids list. Used by + `app::render_usage_block` (parse-error rendering) and + `input_render::ambient_hint` (live hint panel). +- `entry_words_alphabetised() -> Vec<&'static str>` replaces + `dsl::usage::entry_keywords_alphabetised`. + +`dsl::usage` is deleted entirely. The "available commands:" +fallback in `render_usage_block` formats entry words as +`` `` `` directly (replacing the `parse.token.keyword.*` +catalog lookup; equivalent rendering). + +`parse_command` and `parse_tokens` slim down: no pre-lex pass; +the walker scans source bytes directly. `parse_tokens` (which +had been kept `pub` "for future I3/I4 work") is folded into +`parse_command`. + +### Step 3 — Legacy module deletion (commit 266b4c2) + +Deleted: `src/dsl/lexer.rs`, `src/dsl/keyword.rs`, +`src/dsl/ident_slot.rs`. + +`IdentSource` (`dsl::grammar`) absorbs the schema-list / +expected-label / round-trip semantics that previously lived on +`IdentSlot`. The walker's `Expectation::Ident { source }` and +the schema-lookup request on the database worker now share one +enum: + +- `IdentSource::Tables`, `Columns`, `Relationships` are the + schema-listable sources (`completes_from_schema() == true`). +- `IdentSource::NewName` is the user-invents kind. +- `IdentSource::Types` is the closed-set source on column-type + slots — does not query the schema; the walker's content + validator handles type-name validity. +- `IdentSource::Free` is the catch-all branch in `mode` / + `messages` value slots. + +`SchemaCache::for_slot(IdentSlot)` becomes +`for_source(IdentSource)`; `Database::list_names_for` and +the `Request::ListNamesFor` worker variant take `IdentSource`. + +`InvalidIdent.slot: IdentSlot` becomes +`InvalidIdent.source: IdentSource`. + +Completion's keyword filter (was `Keyword::from_word`) becomes +"backticked items whose payload is all ASCII alphabetic" — +punct and digit literals (`,`, `1`) still surface through their +own candidate sources (composite-literal, flag, schema-ident). +The alphabetic filter excludes them from the keyword bucket. + +`friendly::keys::tests::keyword_and_punct_have_complete_token_vocabulary` +is dropped (cross-checked enum vs. catalog completeness; both +enums are gone). + +### Step 4 — Catalog token-keyword cleanup (commit fa994cf) + +Dropped the 47 `parse.token.keyword.*` and 6 +`parse.token.punct.*` catalog entries (yaml + keys.rs). Nothing +consumes them: keyword wording is produced verbatim by +`format!("\`{word}\`")`, sourced from grammar-tree Word +literals. Punct wording surfaces the same way via +`Expectation::Punct(ch)`. + +Structural-class labels (`parse.token.identifier`, `.number`, +`.string_literal`, `.flag`, `.end_of_input`) and lex-error +wordings (`parse.token.error.{bad_flag,unknown_char, +unterminated_string}`) stay. None of these are derivable from +the grammar tree. + +`keys_validate_against_catalog` continues to enforce catalog ↔ +`KEYS_AND_PLACEHOLDERS` bidirectional coverage on the trimmed +set. + +## DEFERRED — work the next session needs to pick up + +Items grouped by priority. Roughly in order of decreasing payoff. + +### 1. Walker-driven completion (Phase F full step 5) + +The current completion path: + +``` +input → parse_command → ParseError::Invalid::expected: Vec + → completion.rs parses each string back to IdentSource + via from_expected_label +``` + +This works but loses information. The walker knows the full +`IdentSource` of every expected slot, the `role` of every slot +(e.g. `parent_table` vs `child_table`), the `skipped` +expectations from any Optional that didn't engage, the cursor's +full `MatchedPath` so far — and the bridge throws all of that +away to render strings. + +**Migration**: add a `expected_walker: Vec` field +on `ParseError::Invalid` (additive, doesn't break consumers +yet). Or a new `walker::candidates_at(input, cursor) -> +WalkResult` API. Update `completion.rs::candidates_at_cursor` +to read `Expectation::Ident { source, role }` directly. + +The walker already supports `WalkBound::Position(cursor)` (the +variant exists in `outcome.rs`); no caller passes it today. +Adopting it lets completion see the cursor's full context. + +**Estimated cost:** one session. Smaller than the steps that +landed this round. + +### 2. Phase D (full): schema-aware value typing + +Unchanged from handoff-9 §2 — same scope, same blockers. +Specifically: + +- `WalkContext::current_table`, `current_table_columns`, + `current_column` are declared but unwritten. +- `Node::DynamicSubgrammar(fn(&WalkContext) -> Node)` is + declared; the walker driver returns `Failed { expected: + vec![] }` on that branch (deliberate — catches mis-declared + grammar). Walker dispatch needs implementing per ADR-0024 + §sub-grammars (`Box::leak` per walk, or per-walk arena). +- `SchemaCache` carries flat `columns: Vec`; per-table + column-with-type info needs adding. +- `Ident { source: Tables }` has no `writes_table: bool` flag; + walker can't populate `current_table` on match. +- Typed value slots (`int_slot`, `decimal_slot`, …) are not + declared. + +**The user UX win this unlocks:** typed slots reject mis-shaped +input at parse time with localised wording ("Type a date as +'YYYY-MM-DD'") instead of bind-time errors; completion narrows +per column type. Round-5 "value-literal hint by column type" +becomes type-specific. + +**Sequence to implement** (from handoff-9 §2): + +1. Plumb a `SchemaCache` reference into `parse_command`; thread + through `WalkContext::new(schema)`. +2. Implement `Node::DynamicSubgrammar` walker dispatch. +3. Add `writes_table: bool` (or analogous) to `Node::Ident` + and have the walker populate `WalkContext::current_table` + + `current_table_columns` from the schema cache when it + matches. +4. Implement typed value slots — content validators for each + `Type`. +5. Wire `column_value_list` as a `DynamicSubgrammar` that reads + `current_table_columns` and emits a `Seq` of typed slots + separated by commas. +6. Update `insert` shape to use `column_value_list`. +7. Update `update` / `delete` to use the per-column value slot + based on `current_column`. + +**Side effect to watch:** `parse_command` becomes +schema-dependent. Tests that exercised parse in isolation +without a schema will need to either pass a schema cache or fall +back to the type-unaware path (Option<&SchemaCache>). + +**Estimated cost:** 1–2 sessions, as last session estimated. + +### 3. HintMode annotations on grammar nodes + +`HintMode::{Default | ForceProse | ProseOnly | SuppressProse}` +is declared in `grammar/mod.rs`. Every `CommandNode` and every +`Node::Ident` sets it to `None`. The current ad-hoc hint cases +in `input_render.rs::ambient_hint` (`value_literal_hint_at_cursor`, +`typing_name_at_cursor`, `invalid_ident_at_cursor`) still drive +the hint panel. + +ADR-0024 §HintMode-per-node says these migrate to node-attached +annotations during Phase D. They didn't (handoff-9 confirmed, +this session didn't move them either). + +To do once Phase D lands: annotate the value-literal slots with +`HintMode::ProseOnly("value_literal_format_hint")`, NewName +slots with the typing-name prose, and have the hint resolver +dispatch on the walker's per-expected-node mode. The current +stopgap `value_literal_hint_at_cursor` becomes +"narrow to the column's type" instead of generic. + +### 4. Ranker hook (declared in ADR-0024, not implemented) + +ADR-0024 §ranker-layer specifies a `Ranker` function type +between the walker's raw candidates and the hint-panel +renderer. Default is `identity_ranker` (declaration-order +preserved). + +**Status: not declared anywhere in code.** Future plug-in point +for frequency-based ranking, content-aware priors, recency. +Non-blocking. + +### 5. Differential-test scaffolding wasn't built (carried from handoff-9) + +ADR-0024 §test-discipline §3 specified a "differential check +during the migration window" — a test helper running both +parsers against the input corpus, asserting identical `Command` +output. Since chumsky is gone (removed in Phase F minimal), +running the differential retroactively would require +reconstructing the chumsky path from git history. Not worth it +— the hand-curated `dsl::walker::tests` (53 tests) and the +existing integration test suite serve the same regression-net +role. + +### 6. `WalkContext` writes during walk — design exists, not implemented + +Carried from handoff-9 §12. `Ident { source: Tables, writes_table: +true }` semantics: when the ident matches, the walker writes +`current_table` to context. Subsequent dynamic sub-grammars read +it. Today no `Ident` node has a `writes_table` field. Adding it +is part of Phase D §3 above. + +### 7. `CommandNode.help_id` not consumed + +Carried from handoff-9 §13. Every `CommandNode` declares +`help_id: Option<&'static str>` pointing into the catalog. +**No code reads it.** Wiring up in-app help to read this field +(replacing hand-curated `help.in_app_body` lookups) is future +work — not blocking, not user-facing. + +### 8. Dead `parse.token.*` catalog entries + +The five structural-class entries +(`parse.token.identifier/number/string_literal/flag/end_of_input`) +and three lex-error entries +(`parse.token.error.{bad_flag,unknown_char,unterminated_string}`) +remain in the catalog after step 4. They are unreferenced by +production code today — the walker classifies bytes by shape +for highlighting and emits structural / validation errors with +catalog keys directly (e.g., `mode.unknown`). + +Conservative call this session: leave them in. They cost +nothing, and re-introducing them would be cheap if a future +need arises. If you decide they're truly dead, drop the entries +from `friendly/strings/en-US.yaml` and the corresponding +declarations from `friendly/keys.rs`. + +## Sharp edges (carried from handoff-9 with updates) + +- **Optional backtracking on partial-match** is intentional — + matches chumsky's `or_not` semantics. See handoff-9 for the + asymmetry between content (no rollback) and structural (roll + back). Unchanged. +- **Walker's `Choice` is strictly greedy.** Unchanged. +- **`Ident { source: Tables/Columns/Relationships }` does NOT + validate against the schema at parse time.** Still + shape-only. Phase D §3 unlocks schema-aware parse. +- **`Literal(&'static str)` matches verbatim bytes with a + word-boundary lookahead.** Unchanged. +- **`AST builder` failures surface as + `WalkOutcome::ValidationFailed`** with `at_eof = true`. + Unchanged. +- **`unknown_command_error` is the sole catch-all** for inputs + whose first identifier-shape token isn't a registered entry + word. Now read from `dsl::grammar::entry_words_alphabetised()` + (was `usage::entry_keywords_alphabetised`). +- **`q` quit alias remains gone.** Native walker alias support + still works: adding `q` back is `aliases: &["q"]` on + `QUIT.entry`. Walker matches either; completion surfaces only + the primary. +- **Path-bearing UX (replay / import / export):** unchanged + from handoff-9 — paths with spaces use the quoted form. +- **Highlight fallback semantics (new this session):** for + inputs the walker doesn't engage on (no registered entry + word, e.g., `frobulate widgets`) the byte-shape scanner + classifies each shape as Identifier, Number, String, Flag, + Punct, or Error. The user sees normal token colouring on the + unknown command before the `[error]` line fires on submit. + This matches the pre-walker behaviour. +- **Stopgap `value_literal_hint_at_cursor` continues to fire** + for every value-literal slot regardless of column type. Same + wording as handoff-9. Replaced once Phase D §4 typed slots + land. + +## ADR index (delta vs. handoff-9) + +``` +0019 Friendly error layer and i18n catalog + — parse.token.keyword/punct.* entries collapsed (Phase F) +0020 Tokenization layer for the DSL parser + — superseded; dsl::lexer module deleted (Phase F) +0021 Parser-as-source-of-truth for H1a + — partial: usage info reads from CommandNode.usage_ids; + help_id wiring deferred (handoff-10 §7) +0022 Ambient typing assistance + — completion still reads ParseError-string expected; + walker-direct path pending (handoff-10 §1) +0024 Unified grammar tree: execution plan (ACCEPTED) + — A through F full steps 1–4 landed. + F step 5 (walker-direct completion) + D full deferred. +``` + +## Repository layout (delta vs. handoff-9) + +Files deleted: + +``` +src/dsl/lexer.rs (598 lines) +src/dsl/keyword.rs (311 lines) +src/dsl/ident_slot.rs (140 lines) +src/dsl/usage.rs (318 lines) +``` + +Files added: + +``` +src/dsl/walker/highlight.rs (319 lines) +docs/handoff/20260515-handoff-10.md (this file) +``` + +Files significantly modified: + +``` +src/dsl/grammar/mod.rs — usage_ids, usage_keys_for_input, + entry_words_alphabetised, + IdentSource expanded with helpers +src/dsl/grammar/app.rs — usage_ids tuples (corrected catalog keys) +src/dsl/grammar/ddl.rs — usage_ids tuples (drop/add family) +src/dsl/grammar/data.rs — usage_ids tuples (show family) +src/dsl/parser.rs — slimmed; chumsky-side parse_tokens folded in +src/dsl/walker/mod.rs — re-export highlight_runs +src/dsl/walker/outcome.rs — ByteClass / WalkResult fields no longer + #[allow(dead_code)] +src/dsl/mod.rs — removed legacy module declarations +src/app.rs — render_usage_block via walker registry +src/input_render.rs — base_runs via walker::highlight_runs; + ambient_hint via usage_keys_for_input +src/completion.rs — IdentSource throughout; keyword filter + via ASCII-alphabetic check +src/db.rs — Request::ListNamesFor takes IdentSource +src/runtime.rs — refresh_schema_cache uses IdentSource +src/theme.rs — highlight_class_color; token_color removed +src/friendly/keys.rs — parse.token.keyword/punct.* dropped +src/friendly/strings/en-US.yaml — same drops +``` + +## How to take over + +1. **Read this file.** +2. **Read `CLAUDE.md`** for the working-style rules. +3. **Read handoff-9** (`20260515-handoff-9.md`) for context on the + ADR-0024 phases that landed in the prior session and any + carry-over sharp edges. +4. **Read ADR-0024** for the design intent. F-step-5 (walker-direct + completion) and Phase D full are the unfinished work; their + sketches live in §migration of that ADR. +5. **Skim `src/dsl/grammar/mod.rs`** — the Node / Word / + CommandNode / REGISTRY / IdentSource / HintMode contract. +6. **Skim `src/dsl/walker/mod.rs` + `walker/highlight.rs`** — the + walk() entry, bridge logic, and the new highlight-runs API. +7. **Run `cargo test`** — should report 806 passing, 0 failing, + 1 ignored. +8. **Run `cargo clippy --all-targets -- -D warnings`** — clean. +9. **Pick a deferred item from §1–§4** and start. §1 (walker- + driven completion) is the natural next move; it removes the + string round-trip and unlocks richer hints. §2 (Phase D full) + is the largest item and the biggest user-visible payoff. + +### End-to-end smoke (current state) + +Same as handoff-9's smoke; nothing in user-visible behaviour +changed this session. All four steps were silent refactors of +internal machinery — same parse, same dispatch, same wording +on errors, same colours on tokens. The catalog cleanup removed +unused YAML entries; no string the user sees was sourced from +them.