rdbms-playground/docs/handoff/20260515-handoff-10.md

Session handoff — 2026-05-15 (10)

Tenth handover. This session executed ADR-0024 Phase F (full) steps 1–4 on top of handoff-9: the walker now drives highlighting, usage rendering, and the schema-list / completion ident-source vocabulary. dsl::lexer, dsl::keyword, dsl::ident_slot, and dsl::usage are deleted; their catalog wrappers (parse.token.keyword.* and parse.token.punct.*) are deleted too. Four commits in clean incremental steps.

What did NOT land — and what the next session picks up — is captured below.

State at handoff

Branch: main. Working tree clean. Local HEAD is fa994cf, ahead of origin/main by five commits (user pushes asynchronously).

Commits since handoff-9's baseline (b3d3bdf):

7bdd398 ADR-0024 Phase F (full) step 1: walker-driven highlighting
a41400e ADR-0024 Phase F (full) step 2: usage via CommandNode.usage_ids
266b4c2 ADR-0024 Phase F (full) step 3: delete legacy parser modules
fa994cf ADR-0024 Phase F (full) step 4: catalog token-keyword cleanup

Tests: 806 passing, 0 failing, 1 ignored (was 844 at handoff-9; the net drop reflects the removal of legacy-module tests — ~45 tests in dsl::lexer, dsl::keyword, dsl::ident_slot, dsl::usage — minus 16 new walker highlight tests). The ignored test is still the same \``ignoredoc-test insrc/friendly/mod.rs`.

Clippy: clean with nursery lints + -D warnings.

Cargo.toml: unchanged (chumsky was already gone in handoff-9).

What shipped this session

Step 1 — Walker-driven highlighting (commit 7bdd398)

input_render::base_runs no longer calls lex(). The new walker::highlight::highlight_runs(source) -> Vec<ByteClass> combines walker per_byte_class output (for matched portions) with a byte-shape fallback (over lex_helpers) for trailing junk / unknown-command words / unterminated strings. The fallback handles UTF-8 codepoints byte-exact and produces an Error class for unrecognised bytes — preserves the $ → tok_error behaviour from the lexer-driven path.

Theme::highlight_class_color(HighlightClass) -> Color replaces Theme::token_color(&TokenKind). The renderer reads HighlightClass directly off ByteClass.

#[allow(dead_code)] is off HighlightClass and WalkResult::per_byte_class — they are part of the production path now.

16 new walker-highlight tests pin the byte-class output for walks, fallbacks, multi-byte UTF-8, and the trailing-token case.

Step 2 — Usage rendering via CommandNode.usage_ids (commit a41400e)

CommandNode.usage_id: Option<&'static str> becomes usage_ids: &'static [&'static str]. Multi-form families (drop, add, show) carry every variant — the legacy dsl::usage::matched_entry returned multi-key Vec for those families and the walker registry now matches that.

App-lifecycle commands had been pointed at non-existent parse.usage.app.* catalog keys (an unnoticed bug introduced in handoff-9 because the field was unused at the time); they now point at the real catalog entries.

Two new helpers in dsl::grammar:

• usage_keys_for_input(source) -> Option<(entry_word_text, usage_ids)> resolves the input's first identifier-shape token to a CommandNode and returns its usage_ids list. Used by app::render_usage_block (parse-error rendering) and input_render::ambient_hint (live hint panel).
• entry_words_alphabetised() -> Vec<&'static str> replaces dsl::usage::entry_keywords_alphabetised.

dsl::usage is deleted entirely. The "available commands:" fallback in render_usage_block formats entry words as `<word>` directly (replacing the parse.token.keyword.* catalog lookup; equivalent rendering).

parse_command and parse_tokens slim down: no pre-lex pass; the walker scans source bytes directly. parse_tokens (which had been kept pub "for future I3/I4 work") is folded into parse_command.

Step 3 — Legacy module deletion (commit 266b4c2)

Deleted: src/dsl/lexer.rs, src/dsl/keyword.rs, src/dsl/ident_slot.rs.

IdentSource (dsl::grammar) absorbs the schema-list / expected-label / round-trip semantics that previously lived on IdentSlot. The walker's Expectation::Ident { source } and the schema-lookup request on the database worker now share one enum:

• IdentSource::Tables, Columns, Relationships are the schema-listable sources (completes_from_schema() == true).
• IdentSource::NewName is the user-invents kind.
• IdentSource::Types is the closed-set source on column-type slots — does not query the schema; the walker's content validator handles type-name validity.
• IdentSource::Free is the catch-all branch in mode / messages value slots.

SchemaCache::for_slot(IdentSlot) becomes for_source(IdentSource); Database::list_names_for and the Request::ListNamesFor worker variant take IdentSource.

InvalidIdent.slot: IdentSlot becomes InvalidIdent.source: IdentSource.

Completion's keyword filter (was Keyword::from_word) becomes "backticked items whose payload is all ASCII alphabetic" — punct and digit literals (,, 1) still surface through their own candidate sources (composite-literal, flag, schema-ident). The alphabetic filter excludes them from the keyword bucket.

friendly::keys::tests::keyword_and_punct_have_complete_token_vocabulary is dropped (cross-checked enum vs. catalog completeness; both enums are gone).

Step 4 — Catalog token-keyword cleanup (commit fa994cf)

Dropped the 47 parse.token.keyword.* and 6 parse.token.punct.* catalog entries (yaml + keys.rs). Nothing consumes them: keyword wording is produced verbatim by format!("\{word}`"), sourced from grammar-tree Word literals. Punct wording surfaces the same way via Expectation::Punct(ch)`.

Structural-class labels (parse.token.identifier, .number, .string_literal, .flag, .end_of_input) and lex-error wordings (parse.token.error.{bad_flag,unknown_char, unterminated_string}) stay. None of these are derivable from the grammar tree.

keys_validate_against_catalog continues to enforce catalog ↔ KEYS_AND_PLACEHOLDERS bidirectional coverage on the trimmed set.

DEFERRED — work the next session needs to pick up

Items grouped by priority. Roughly in order of decreasing payoff.

1. Walker-driven completion (Phase F full step 5)

The current completion path:

input → parse_command → ParseError::Invalid::expected: Vec<String>
       → completion.rs parses each string back to IdentSource
         via from_expected_label

This works but loses information. The walker knows the full IdentSource of every expected slot, the role of every slot (e.g. parent_table vs child_table), the skipped expectations from any Optional that didn't engage, the cursor's full MatchedPath so far — and the bridge throws all of that away to render strings.

Migration: add a expected_walker: Vec<Expectation> field on ParseError::Invalid (additive, doesn't break consumers yet). Or a new walker::candidates_at(input, cursor) -> WalkResult API. Update completion.rs::candidates_at_cursor to read Expectation::Ident { source, role } directly.

The walker already supports WalkBound::Position(cursor) (the variant exists in outcome.rs); no caller passes it today. Adopting it lets completion see the cursor's full context.

Estimated cost: one session. Smaller than the steps that landed this round.

2. Phase D (full): schema-aware value typing

Unchanged from handoff-9 §2 — same scope, same blockers. Specifically:

• WalkContext::current_table, current_table_columns, current_column are declared but unwritten.
• Node::DynamicSubgrammar(fn(&WalkContext) -> Node) is declared; the walker driver returns Failed { expected: vec![] } on that branch (deliberate — catches mis-declared grammar). Walker dispatch needs implementing per ADR-0024 §sub-grammars (Box::leak per walk, or per-walk arena).
• SchemaCache carries flat columns: Vec<String>; per-table column-with-type info needs adding.
• Ident { source: Tables } has no writes_table: bool flag; walker can't populate current_table on match.
• Typed value slots (int_slot, decimal_slot, …) are not declared.

The user UX win this unlocks: typed slots reject mis-shaped input at parse time with localised wording ("Type a date as 'YYYY-MM-DD'") instead of bind-time errors; completion narrows per column type. Round-5 "value-literal hint by column type" becomes type-specific.

Sequence to implement (from handoff-9 §2):

1. Plumb a SchemaCache reference into parse_command; thread through WalkContext::new(schema).
2. Implement Node::DynamicSubgrammar walker dispatch.
3. Add writes_table: bool (or analogous) to Node::Ident and have the walker populate WalkContext::current_table + current_table_columns from the schema cache when it matches.
4. Implement typed value slots — content validators for each Type.
5. Wire column_value_list as a DynamicSubgrammar that reads current_table_columns and emits a Seq of typed slots separated by commas.
6. Update insert shape to use column_value_list.
7. Update update / delete to use the per-column value slot based on current_column.

Side effect to watch: parse_command becomes schema-dependent. Tests that exercised parse in isolation without a schema will need to either pass a schema cache or fall back to the type-unaware path (Option<&SchemaCache>).

Estimated cost: 1–2 sessions, as last session estimated.

3. HintMode annotations on grammar nodes

HintMode::{Default | ForceProse | ProseOnly | SuppressProse} is declared in grammar/mod.rs. Every CommandNode and every Node::Ident sets it to None. The current ad-hoc hint cases in input_render.rs::ambient_hint (value_literal_hint_at_cursor, typing_name_at_cursor, invalid_ident_at_cursor) still drive the hint panel.

ADR-0024 §HintMode-per-node says these migrate to node-attached annotations during Phase D. They didn't (handoff-9 confirmed, this session didn't move them either).

To do once Phase D lands: annotate the value-literal slots with HintMode::ProseOnly("value_literal_format_hint"), NewName slots with the typing-name prose, and have the hint resolver dispatch on the walker's per-expected-node mode. The current stopgap value_literal_hint_at_cursor becomes "narrow to the column's type" instead of generic.

4. Ranker hook (declared in ADR-0024, not implemented)

ADR-0024 §ranker-layer specifies a Ranker function type between the walker's raw candidates and the hint-panel renderer. Default is identity_ranker (declaration-order preserved).

Status: not declared anywhere in code. Future plug-in point for frequency-based ranking, content-aware priors, recency. Non-blocking.

5. Differential-test scaffolding wasn't built (carried from handoff-9)

ADR-0024 §test-discipline §3 specified a "differential check during the migration window" — a test helper running both parsers against the input corpus, asserting identical Command output. Since chumsky is gone (removed in Phase F minimal), running the differential retroactively would require reconstructing the chumsky path from git history. Not worth it — the hand-curated dsl::walker::tests (53 tests) and the existing integration test suite serve the same regression-net role.

6. WalkContext writes during walk — design exists, not implemented

Carried from handoff-9 §12. Ident { source: Tables, writes_table: true } semantics: when the ident matches, the walker writes current_table to context. Subsequent dynamic sub-grammars read it. Today no Ident node has a writes_table field. Adding it is part of Phase D §3 above.

7. CommandNode.help_id not consumed

Carried from handoff-9 §13. Every CommandNode declares help_id: Option<&'static str> pointing into the catalog. No code reads it. Wiring up in-app help to read this field (replacing hand-curated help.in_app_body lookups) is future work — not blocking, not user-facing.

8. Dead parse.token.* catalog entries

The five structural-class entries (parse.token.identifier/number/string_literal/flag/end_of_input) and three lex-error entries (parse.token.error.{bad_flag,unknown_char,unterminated_string}) remain in the catalog after step 4. They are unreferenced by production code today — the walker classifies bytes by shape for highlighting and emits structural / validation errors with catalog keys directly (e.g., mode.unknown).

Conservative call this session: leave them in. They cost nothing, and re-introducing them would be cheap if a future need arises. If you decide they're truly dead, drop the entries from friendly/strings/en-US.yaml and the corresponding declarations from friendly/keys.rs.

Sharp edges (carried from handoff-9 with updates)

• Optional backtracking on partial-match is intentional — matches chumsky's or_not semantics. See handoff-9 for the asymmetry between content (no rollback) and structural (roll back). Unchanged.
• Walker's Choice is strictly greedy. Unchanged.
• Ident { source: Tables/Columns/Relationships } does NOT validate against the schema at parse time. Still shape-only. Phase D §3 unlocks schema-aware parse.
• Literal(&'static str) matches verbatim bytes with a word-boundary lookahead. Unchanged.
• AST builder failures surface as WalkOutcome::ValidationFailed with at_eof = true. Unchanged.
• unknown_command_error is the sole catch-all for inputs whose first identifier-shape token isn't a registered entry word. Now read from dsl::grammar::entry_words_alphabetised() (was usage::entry_keywords_alphabetised).
• q quit alias remains gone. Native walker alias support still works: adding q back is aliases: &["q"] on QUIT.entry. Walker matches either; completion surfaces only the primary.
• Path-bearing UX (replay / import / export): unchanged from handoff-9 — paths with spaces use the quoted form.
• Highlight fallback semantics (new this session): for inputs the walker doesn't engage on (no registered entry word, e.g., frobulate widgets) the byte-shape scanner classifies each shape as Identifier, Number, String, Flag, Punct, or Error. The user sees normal token colouring on the unknown command before the [error] line fires on submit. This matches the pre-walker behaviour.
• Stopgap value_literal_hint_at_cursor continues to fire for every value-literal slot regardless of column type. Same wording as handoff-9. Replaced once Phase D §4 typed slots land.

ADR index (delta vs. handoff-9)

0019 Friendly error layer and i18n catalog
        — parse.token.keyword/punct.* entries collapsed (Phase F)
0020 Tokenization layer for the DSL parser
        — superseded; dsl::lexer module deleted (Phase F)
0021 Parser-as-source-of-truth for H1a
        — partial: usage info reads from CommandNode.usage_ids;
          help_id wiring deferred (handoff-10 §7)
0022 Ambient typing assistance
        — completion still reads ParseError-string expected;
          walker-direct path pending (handoff-10 §1)
0024 Unified grammar tree: execution plan (ACCEPTED)
        — A through F full steps 1–4 landed.
          F step 5 (walker-direct completion) + D full deferred.

Repository layout (delta vs. handoff-9)

Files deleted:

src/dsl/lexer.rs        (598 lines)
src/dsl/keyword.rs      (311 lines)
src/dsl/ident_slot.rs   (140 lines)
src/dsl/usage.rs        (318 lines)

Files added:

src/dsl/walker/highlight.rs   (319 lines)
docs/handoff/20260515-handoff-10.md (this file)

Files significantly modified:

src/dsl/grammar/mod.rs   — usage_ids, usage_keys_for_input,
                             entry_words_alphabetised,
                             IdentSource expanded with helpers
src/dsl/grammar/app.rs   — usage_ids tuples (corrected catalog keys)
src/dsl/grammar/ddl.rs   — usage_ids tuples (drop/add family)
src/dsl/grammar/data.rs  — usage_ids tuples (show family)
src/dsl/parser.rs        — slimmed; chumsky-side parse_tokens folded in
src/dsl/walker/mod.rs    — re-export highlight_runs
src/dsl/walker/outcome.rs — ByteClass / WalkResult fields no longer
                             #[allow(dead_code)]
src/dsl/mod.rs           — removed legacy module declarations
src/app.rs               — render_usage_block via walker registry
src/input_render.rs      — base_runs via walker::highlight_runs;
                             ambient_hint via usage_keys_for_input
src/completion.rs        — IdentSource throughout; keyword filter
                             via ASCII-alphabetic check
src/db.rs                — Request::ListNamesFor takes IdentSource
src/runtime.rs           — refresh_schema_cache uses IdentSource
src/theme.rs             — highlight_class_color; token_color removed
src/friendly/keys.rs     — parse.token.keyword/punct.* dropped
src/friendly/strings/en-US.yaml — same drops

How to take over

1. Read this file.
2. Read CLAUDE.md for the working-style rules.
3. Read handoff-9 (20260515-handoff-9.md) for context on the ADR-0024 phases that landed in the prior session and any carry-over sharp edges.
4. Read ADR-0024 for the design intent. F-step-5 (walker-direct completion) and Phase D full are the unfinished work; their sketches live in §migration of that ADR.
5. Skim src/dsl/grammar/mod.rs — the Node / Word / CommandNode / REGISTRY / IdentSource / HintMode contract.
6. Skim src/dsl/walker/mod.rs + walker/highlight.rs — the walk() entry, bridge logic, and the new highlight-runs API.
7. Run cargo test — should report 806 passing, 0 failing, 1 ignored.
8. Run cargo clippy --all-targets -- -D warnings — clean.
9. Pick a deferred item from §1–§4 and start. §1 (walker- driven completion) is the natural next move; it removes the string round-trip and unlocks richer hints. §2 (Phase D full) is the largest item and the biggest user-visible payoff.

End-to-end smoke (current state)

Same as handoff-9's smoke; nothing in user-visible behaviour changed this session. All four steps were silent refactors of internal machinery — same parse, same dispatch, same wording on errors, same colours on tokens. The catalog cleanup removed unused YAML entries; no string the user sees was sourced from them.