rdbms-playground/docs/handoff/20260515-handoff-10.md

# Session handoff — 2026-05-15 (10)

Tenth handover. This session executed **ADR-0024 Phase F (full)
steps 1–4** on top of handoff-9: the walker now drives
highlighting, usage rendering, and the schema-list / completion
ident-source vocabulary. `dsl::lexer`, `dsl::keyword`,
`dsl::ident_slot`, and `dsl::usage` are deleted; their
catalog wrappers (`parse.token.keyword.*` and
`parse.token.punct.*`) are deleted too. Four commits in clean
incremental steps.

What did NOT land — and what the next session picks up — is
captured below.

## State at handoff

**Branch:** `main`. Working tree clean. **Local HEAD is
`fa994cf`**, ahead of `origin/main` by five commits (user pushes
asynchronously).

Commits since handoff-9's baseline (`b3d3bdf`):

```
7bdd398 ADR-0024 Phase F (full) step 1: walker-driven highlighting
a41400e ADR-0024 Phase F (full) step 2: usage via CommandNode.usage_ids
266b4c2 ADR-0024 Phase F (full) step 3: delete legacy parser modules
fa994cf ADR-0024 Phase F (full) step 4: catalog token-keyword cleanup
```

**Tests:** **806 passing, 0 failing, 1 ignored** (was 844 at
handoff-9; the net drop reflects the removal of legacy-module
tests — ~45 tests in `dsl::lexer`, `dsl::keyword`,
`dsl::ident_slot`, `dsl::usage` — minus 16 new walker
highlight tests). The ignored test is still the same `\`\`\`ignore`
doc-test in `src/friendly/mod.rs`.

**Clippy:** clean with `nursery` lints + `-D warnings`.

**Cargo.toml:** unchanged (chumsky was already gone in handoff-9).

## What shipped this session

### Step 1 — Walker-driven highlighting (commit 7bdd398)

`input_render::base_runs` no longer calls `lex()`. The new
`walker::highlight::highlight_runs(source) -> Vec<ByteClass>`
combines walker `per_byte_class` output (for matched portions)
with a byte-shape fallback (over `lex_helpers`) for trailing
junk / unknown-command words / unterminated strings. The
fallback handles UTF-8 codepoints byte-exact and produces an
`Error` class for unrecognised bytes — preserves the `$` →
`tok_error` behaviour from the lexer-driven path.

`Theme::highlight_class_color(HighlightClass) -> Color`
replaces `Theme::token_color(&TokenKind)`. The renderer reads
`HighlightClass` directly off `ByteClass`.

`#[allow(dead_code)]` is off `HighlightClass` and
`WalkResult::per_byte_class` — they are part of the production
path now.

16 new walker-highlight tests pin the byte-class output for
walks, fallbacks, multi-byte UTF-8, and the trailing-token case.

### Step 2 — Usage rendering via `CommandNode.usage_ids` (commit a41400e)

`CommandNode.usage_id: Option<&'static str>` becomes
`usage_ids: &'static [&'static str]`. Multi-form families
(`drop`, `add`, `show`) carry every variant — the legacy
`dsl::usage::matched_entry` returned multi-key Vec for those
families and the walker registry now matches that.

App-lifecycle commands had been pointed at non-existent
`parse.usage.app.*` catalog keys (an unnoticed bug introduced
in handoff-9 because the field was unused at the time); they
now point at the real catalog entries.

Two new helpers in `dsl::grammar`:

- `usage_keys_for_input(source) -> Option<(entry_word_text, usage_ids)>`
  resolves the input's first identifier-shape token to a
  CommandNode and returns its usage_ids list. Used by
  `app::render_usage_block` (parse-error rendering) and
  `input_render::ambient_hint` (live hint panel).
- `entry_words_alphabetised() -> Vec<&'static str>` replaces
  `dsl::usage::entry_keywords_alphabetised`.

`dsl::usage` is deleted entirely. The "available commands:"
fallback in `render_usage_block` formats entry words as
`` `<word>` `` directly (replacing the `parse.token.keyword.*`
catalog lookup; equivalent rendering).

`parse_command` and `parse_tokens` slim down: no pre-lex pass;
the walker scans source bytes directly. `parse_tokens` (which
had been kept `pub` "for future I3/I4 work") is folded into
`parse_command`.

### Step 3 — Legacy module deletion (commit 266b4c2)

Deleted: `src/dsl/lexer.rs`, `src/dsl/keyword.rs`,
`src/dsl/ident_slot.rs`.

`IdentSource` (`dsl::grammar`) absorbs the schema-list /
expected-label / round-trip semantics that previously lived on
`IdentSlot`. The walker's `Expectation::Ident { source }` and
the schema-lookup request on the database worker now share one
enum:

- `IdentSource::Tables`, `Columns`, `Relationships` are the
  schema-listable sources (`completes_from_schema() == true`).
- `IdentSource::NewName` is the user-invents kind.
- `IdentSource::Types` is the closed-set source on column-type
  slots — does not query the schema; the walker's content
  validator handles type-name validity.
- `IdentSource::Free` is the catch-all branch in `mode` /
  `messages` value slots.

`SchemaCache::for_slot(IdentSlot)` becomes
`for_source(IdentSource)`; `Database::list_names_for` and
the `Request::ListNamesFor` worker variant take `IdentSource`.

`InvalidIdent.slot: IdentSlot` becomes
`InvalidIdent.source: IdentSource`.

Completion's keyword filter (was `Keyword::from_word`) becomes
"backticked items whose payload is all ASCII alphabetic" —
punct and digit literals (`,`, `1`) still surface through their
own candidate sources (composite-literal, flag, schema-ident).
The alphabetic filter excludes them from the keyword bucket.

`friendly::keys::tests::keyword_and_punct_have_complete_token_vocabulary`
is dropped (cross-checked enum vs. catalog completeness; both
enums are gone).

### Step 4 — Catalog token-keyword cleanup (commit fa994cf)

Dropped the 47 `parse.token.keyword.*` and 6
`parse.token.punct.*` catalog entries (yaml + keys.rs). Nothing
consumes them: keyword wording is produced verbatim by
`format!("\`{word}\`")`, sourced from grammar-tree Word
literals. Punct wording surfaces the same way via
`Expectation::Punct(ch)`.

Structural-class labels (`parse.token.identifier`, `.number`,
`.string_literal`, `.flag`, `.end_of_input`) and lex-error
wordings (`parse.token.error.{bad_flag,unknown_char,
unterminated_string}`) stay. None of these are derivable from
the grammar tree.

`keys_validate_against_catalog` continues to enforce catalog ↔
`KEYS_AND_PLACEHOLDERS` bidirectional coverage on the trimmed
set.

## DEFERRED — work the next session needs to pick up

Items grouped by priority. Roughly in order of decreasing payoff.

### 1. Walker-driven completion (Phase F full step 5)

The current completion path:

```
input → parse_command → ParseError::Invalid::expected: Vec<String>
       → completion.rs parses each string back to IdentSource
         via from_expected_label
```

This works but loses information. The walker knows the full
`IdentSource` of every expected slot, the `role` of every slot
(e.g. `parent_table` vs `child_table`), the `skipped`
expectations from any Optional that didn't engage, the cursor's
full `MatchedPath` so far — and the bridge throws all of that
away to render strings.

**Migration**: add a `expected_walker: Vec<Expectation>` field
on `ParseError::Invalid` (additive, doesn't break consumers
yet). Or a new `walker::candidates_at(input, cursor) ->
WalkResult` API. Update `completion.rs::candidates_at_cursor`
to read `Expectation::Ident { source, role }` directly.

The walker already supports `WalkBound::Position(cursor)` (the
variant exists in `outcome.rs`); no caller passes it today.
Adopting it lets completion see the cursor's full context.

**Estimated cost:** one session. Smaller than the steps that
landed this round.

### 2. Phase D (full): schema-aware value typing

Unchanged from handoff-9 §2 — same scope, same blockers.
Specifically:

- `WalkContext::current_table`, `current_table_columns`,
  `current_column` are declared but unwritten.
- `Node::DynamicSubgrammar(fn(&WalkContext) -> Node)` is
  declared; the walker driver returns `Failed { expected:
  vec![] }` on that branch (deliberate — catches mis-declared
  grammar). Walker dispatch needs implementing per ADR-0024
  §sub-grammars (`Box::leak` per walk, or per-walk arena).
- `SchemaCache` carries flat `columns: Vec<String>`; per-table
  column-with-type info needs adding.
- `Ident { source: Tables }` has no `writes_table: bool` flag;
  walker can't populate `current_table` on match.
- Typed value slots (`int_slot`, `decimal_slot`, …) are not
  declared.

**The user UX win this unlocks:** typed slots reject mis-shaped
input at parse time with localised wording ("Type a date as
'YYYY-MM-DD'") instead of bind-time errors; completion narrows
per column type. Round-5 "value-literal hint by column type"
becomes type-specific.

**Sequence to implement** (from handoff-9 §2):

1. Plumb a `SchemaCache` reference into `parse_command`; thread
   through `WalkContext::new(schema)`.
2. Implement `Node::DynamicSubgrammar` walker dispatch.
3. Add `writes_table: bool` (or analogous) to `Node::Ident`
   and have the walker populate `WalkContext::current_table` +
   `current_table_columns` from the schema cache when it
   matches.
4. Implement typed value slots — content validators for each
   `Type`.
5. Wire `column_value_list` as a `DynamicSubgrammar` that reads
   `current_table_columns` and emits a `Seq` of typed slots
   separated by commas.
6. Update `insert` shape to use `column_value_list`.
7. Update `update` / `delete` to use the per-column value slot
   based on `current_column`.

**Side effect to watch:** `parse_command` becomes
schema-dependent. Tests that exercised parse in isolation
without a schema will need to either pass a schema cache or fall
back to the type-unaware path (Option<&SchemaCache>).

**Estimated cost:** 1–2 sessions, as last session estimated.

### 3. HintMode annotations on grammar nodes

`HintMode::{Default | ForceProse | ProseOnly | SuppressProse}`
is declared in `grammar/mod.rs`. Every `CommandNode` and every
`Node::Ident` sets it to `None`. The current ad-hoc hint cases
in `input_render.rs::ambient_hint` (`value_literal_hint_at_cursor`,
`typing_name_at_cursor`, `invalid_ident_at_cursor`) still drive
the hint panel.

ADR-0024 §HintMode-per-node says these migrate to node-attached
annotations during Phase D. They didn't (handoff-9 confirmed,
this session didn't move them either).

To do once Phase D lands: annotate the value-literal slots with
`HintMode::ProseOnly("value_literal_format_hint")`, NewName
slots with the typing-name prose, and have the hint resolver
dispatch on the walker's per-expected-node mode. The current
stopgap `value_literal_hint_at_cursor` becomes
"narrow to the column's type" instead of generic.

### 4. Ranker hook (declared in ADR-0024, not implemented)

ADR-0024 §ranker-layer specifies a `Ranker` function type
between the walker's raw candidates and the hint-panel
renderer. Default is `identity_ranker` (declaration-order
preserved).

**Status: not declared anywhere in code.** Future plug-in point
for frequency-based ranking, content-aware priors, recency.
Non-blocking.

### 5. Differential-test scaffolding wasn't built (carried from handoff-9)

ADR-0024 §test-discipline §3 specified a "differential check
during the migration window" — a test helper running both
parsers against the input corpus, asserting identical `Command`
output. Since chumsky is gone (removed in Phase F minimal),
running the differential retroactively would require
reconstructing the chumsky path from git history. Not worth it
— the hand-curated `dsl::walker::tests` (53 tests) and the
existing integration test suite serve the same regression-net
role.

### 6. `WalkContext` writes during walk — design exists, not implemented

Carried from handoff-9 §12. `Ident { source: Tables, writes_table:
true }` semantics: when the ident matches, the walker writes
`current_table` to context. Subsequent dynamic sub-grammars read
it. Today no `Ident` node has a `writes_table` field. Adding it
is part of Phase D §3 above.

### 7. `CommandNode.help_id` not consumed

Carried from handoff-9 §13. Every `CommandNode` declares
`help_id: Option<&'static str>` pointing into the catalog.
**No code reads it.** Wiring up in-app help to read this field
(replacing hand-curated `help.in_app_body` lookups) is future
work — not blocking, not user-facing.

### 8. Dead `parse.token.*` catalog entries

The five structural-class entries
(`parse.token.identifier/number/string_literal/flag/end_of_input`)
and three lex-error entries
(`parse.token.error.{bad_flag,unknown_char,unterminated_string}`)
remain in the catalog after step 4. They are unreferenced by
production code today — the walker classifies bytes by shape
for highlighting and emits structural / validation errors with
catalog keys directly (e.g., `mode.unknown`).

Conservative call this session: leave them in. They cost
nothing, and re-introducing them would be cheap if a future
need arises. If you decide they're truly dead, drop the entries
from `friendly/strings/en-US.yaml` and the corresponding
declarations from `friendly/keys.rs`.

## Sharp edges (carried from handoff-9 with updates)

- **Optional backtracking on partial-match** is intentional —
  matches chumsky's `or_not` semantics. See handoff-9 for the
  asymmetry between content (no rollback) and structural (roll
  back). Unchanged.
- **Walker's `Choice` is strictly greedy.** Unchanged.
- **`Ident { source: Tables/Columns/Relationships }` does NOT
  validate against the schema at parse time.** Still
  shape-only. Phase D §3 unlocks schema-aware parse.
- **`Literal(&'static str)` matches verbatim bytes with a
  word-boundary lookahead.** Unchanged.
- **`AST builder` failures surface as
  `WalkOutcome::ValidationFailed`** with `at_eof = true`.
  Unchanged.
- **`unknown_command_error` is the sole catch-all** for inputs
  whose first identifier-shape token isn't a registered entry
  word. Now read from `dsl::grammar::entry_words_alphabetised()`
  (was `usage::entry_keywords_alphabetised`).
- **`q` quit alias remains gone.** Native walker alias support
  still works: adding `q` back is `aliases: &["q"]` on
  `QUIT.entry`. Walker matches either; completion surfaces only
  the primary.
- **Path-bearing UX (replay / import / export):** unchanged
  from handoff-9 — paths with spaces use the quoted form.
- **Highlight fallback semantics (new this session):** for
  inputs the walker doesn't engage on (no registered entry
  word, e.g., `frobulate widgets`) the byte-shape scanner
  classifies each shape as Identifier, Number, String, Flag,
  Punct, or Error. The user sees normal token colouring on the
  unknown command before the `[error]` line fires on submit.
  This matches the pre-walker behaviour.
- **Stopgap `value_literal_hint_at_cursor` continues to fire**
  for every value-literal slot regardless of column type. Same
  wording as handoff-9. Replaced once Phase D §4 typed slots
  land.

## ADR index (delta vs. handoff-9)

```
0019 Friendly error layer and i18n catalog
        — parse.token.keyword/punct.* entries collapsed (Phase F)
0020 Tokenization layer for the DSL parser
        — superseded; dsl::lexer module deleted (Phase F)
0021 Parser-as-source-of-truth for H1a
        — partial: usage info reads from CommandNode.usage_ids;
          help_id wiring deferred (handoff-10 §7)
0022 Ambient typing assistance
        — completion still reads ParseError-string expected;
          walker-direct path pending (handoff-10 §1)
0024 Unified grammar tree: execution plan (ACCEPTED)
        — A through F full steps 1–4 landed.
          F step 5 (walker-direct completion) + D full deferred.
```

## Repository layout (delta vs. handoff-9)

Files deleted:

```
src/dsl/lexer.rs        (598 lines)
src/dsl/keyword.rs      (311 lines)
src/dsl/ident_slot.rs   (140 lines)
src/dsl/usage.rs        (318 lines)
```

Files added:

```
src/dsl/walker/highlight.rs   (319 lines)
docs/handoff/20260515-handoff-10.md (this file)
```

Files significantly modified:

```
src/dsl/grammar/mod.rs   — usage_ids, usage_keys_for_input,
                             entry_words_alphabetised,
                             IdentSource expanded with helpers
src/dsl/grammar/app.rs   — usage_ids tuples (corrected catalog keys)
src/dsl/grammar/ddl.rs   — usage_ids tuples (drop/add family)
src/dsl/grammar/data.rs  — usage_ids tuples (show family)
src/dsl/parser.rs        — slimmed; chumsky-side parse_tokens folded in
src/dsl/walker/mod.rs    — re-export highlight_runs
src/dsl/walker/outcome.rs — ByteClass / WalkResult fields no longer
                             #[allow(dead_code)]
src/dsl/mod.rs           — removed legacy module declarations
src/app.rs               — render_usage_block via walker registry
src/input_render.rs      — base_runs via walker::highlight_runs;
                             ambient_hint via usage_keys_for_input
src/completion.rs        — IdentSource throughout; keyword filter
                             via ASCII-alphabetic check
src/db.rs                — Request::ListNamesFor takes IdentSource
src/runtime.rs           — refresh_schema_cache uses IdentSource
src/theme.rs             — highlight_class_color; token_color removed
src/friendly/keys.rs     — parse.token.keyword/punct.* dropped
src/friendly/strings/en-US.yaml — same drops
```

## How to take over

1. **Read this file.**
2. **Read `CLAUDE.md`** for the working-style rules.
3. **Read handoff-9** (`20260515-handoff-9.md`) for context on the
   ADR-0024 phases that landed in the prior session and any
   carry-over sharp edges.
4. **Read ADR-0024** for the design intent. F-step-5 (walker-direct
   completion) and Phase D full are the unfinished work; their
   sketches live in §migration of that ADR.
5. **Skim `src/dsl/grammar/mod.rs`** — the Node / Word /
   CommandNode / REGISTRY / IdentSource / HintMode contract.
6. **Skim `src/dsl/walker/mod.rs` + `walker/highlight.rs`** — the
   walk() entry, bridge logic, and the new highlight-runs API.
7. **Run `cargo test`** — should report 806 passing, 0 failing,
   1 ignored.
8. **Run `cargo clippy --all-targets -- -D warnings`** — clean.
9. **Pick a deferred item from §1–§4** and start. §1 (walker-
   driven completion) is the natural next move; it removes the
   string round-trip and unlocks richer hints. §2 (Phase D full)
   is the largest item and the biggest user-visible payoff.

### End-to-end smoke (current state)

Same as handoff-9's smoke; nothing in user-visible behaviour
changed this session. All four steps were silent refactors of
internal machinery — same parse, same dispatch, same wording
on errors, same colours on tokens. The catalog cleanup removed
unused YAML entries; no string the user sees was sourced from
them.