The validity-indicator debounce was two locals in the event
loop (indicator_pending + app.input_indicator) with no unit
coverage — ADR-0027's as-built notes flag it as untested async
glue. The decision logic is now an IndicatorDebounce struct:
note_event (a keystroke hides + arms; non-key events leave it
be), settle (the quiet window elapsed → show the verdict +
disarm), is_armed (drives the recv timeout), visible (mirrored
into app.input_indicator for the renderer).
No behaviour change — the tokio timer and terminal stay in the
loop. 7 unit tests cover the debounce contract: the keystroke /
settle cycle, clean verdicts, and that a background event
mid-typing does not cancel the owed recompute. 1125 passing,
clippy clean.
ambient_hint now reads the walker's schema-aware diagnostics.
input_diagnostics is non-empty only for a command that
structurally parses — so a non-empty result means "complete
and submittable, but wrong or dubious". That is checked early
(right after the Tab-cycle memo), ahead of slot hints and
completions: a command that parses but is flawed no longer
gets the misleading "Submit with Enter" prose, it gets the
diagnostic's why. pick_hint_diagnostic prefers the diagnostic
under the cursor, else the most severe.
The cursor-local invalid-ident hint is kept for genuinely
incomplete commands (no Match → no diagnostics).
5 ambient_hint tests (unknown table, type-mismatch over
submit-prose, LIKE-numeric, clean command still submittable,
cursor-following). The complex_and_or matrix cell referenced a
non-existent column `t`; fixed to a real column so it tests a
valid expression as intended. 1118 passing, clippy clean.
render_input_runs now overlays the walker's schema-aware
diagnostics: an unknown table/column ERROR is recoloured
tok_error, an expression WARNING (type mismatch, = NULL, LIKE
on a numeric column) recoloured theme.warning. New overlay_span
covers a token's whole byte range (overlay_error only hits the
run at a single byte). New walker::input_diagnostics is the
shared entry point.
The overlay is global — every flagged token is coloured
wherever it sits, not only under the cursor — which is exactly
ADR-0027's motivation. The existing cursor-local invalid-ident
overlay is kept (it covers in-progress idents diagnostics do
not); the two are additive and idempotent.
5 input_render tests (unknown table/column, type-mismatch
literal precise, LIKE-on-numeric, clean command). 1113 passing,
clippy clean.
LIKE is a text-pattern match; against a numeric column (int,
real, decimal, serial) it runs but is almost never intended.
predicate_warnings now emits a WARNING for it, spanned at the
target column. New Type::is_numeric; catalog key
diagnostic.like_numeric; ADR-0027 gains "Amendment 1" and the
adr/README index line is updated per the index-upkeep rule.
bool and the text-/blob-backed types are deliberately not
flagged — see the amendment for the rationale.
3 walker tests (int, decimal NOT LIKE, text-column clean).
1108 passing, clippy clean.
Expression WARNING diagnostics (type mismatch, = NULL) carried
a coarse span — the whole WHERE clause, from the `where`
keyword to end of input. They now span exactly the offending
literal operand, read from the Operand source span added in the
previous commit. predicate_warnings derives the span per
warning; pair_type_mismatch returns (message, literal-span);
the dead where_clause_span helper is removed.
5 walker tests assert the spans cover exactly the literal /
identifier (type mismatch, = NULL, BETWEEN bounds, IN item,
unknown-column ERROR). 1105 passing, clippy clean.
Each WHERE-expression Operand now records the byte span of the
terminal it was built from — the precise per-literal highlight
target for an expression WARNING (finishing ADR-0027 §2's
highlight/hint wiring). parse_operand captures MatchedItem::span;
the RowFilter::eq convenience constructor uses Operand::NO_SPAN.
PartialEq is hand-written to ignore the span — it is editor
metadata, so Command equality stays whitespace- and
position-independent, which the Expr test corpus relies on.
No behaviour change; 1100 tests still pass, clippy clean.
A parse error in `add index …` showed the `add column` usage:
`add` and `drop` are multi-form commands, and both the
ambient hint and the submit-time usage block picked the
first-listed form unconditionally.
New `grammar::usage_key_for_input` disambiguates by the form
word after the entry keyword — `column` / `index` / `table` /
`relationship`, or the leading digit of `add 1:n …`. The
ambient hint now shows that one form; `render_usage_block`
shows the committed form's usage and falls back to the whole
family only for a bare `add` / `drop` with no form chosen.
Sweep: input_verdict tests confirm the schema-existence check
fires across the identifier-taking commands — unknown table
on drop / show / add column, unknown column on drop column /
update — and that known references stay clean. The Step B
check is grammar-generic, so this is verification + coverage
rather than new code.
Docs: requirements.md S6 -> [x], baseline 1096; CLAUDE.md
deferred list reconciled (C5a and S6 are done — removed);
ADR-0026's as-built note updated (step 5 shipped via
ADR-0027); ADR-0027 gains an As-built notes section
recording the post-walk diagnostics realization, the
pre-rendered message, the timeout-based debounce, coarse
WARNING spans, and the deferred highlight/hint wiring.
The event loop now time-boxes `recv` while an indicator
recompute is owed: every keystroke hides the indicator and
arms an `INDICATOR_DEBOUNCE` (1s) window; once typing pauses
that long the runtime computes `App::input_validity_verdict`
and shows `[ERR]` / `[WRN]`. An idle session (nothing owed)
still blocks plainly on `recv` — no wake-ups.
`update()` stays pure — the debounce timer lives in the
runtime; `App` only holds the resulting `input_indicator`
state, which the runtime clears on a keystroke and sets when
the quiet interval elapses.
`App::input_validity_verdict` is tested directly (a
simple-mode verdict, and silence in advanced mode / the `:`
one-shot); the debounce timing itself is runtime-loop glue,
covered at the integration level.
Adds the `[ERR]` / `[WRN]` validity indicator to the input
row. `App` gains `input_indicator: Option<Severity>` (the
runtime owns its timing — step E) and a pure
`input_validity_verdict()` query that runs `input_verdict`
in simple mode only (advanced mode is raw SQL, ADR-0027 §7).
`render_input_panel` reserves the rightmost six columns of
the input row unconditionally (ADR-0027 §4) — a five-column
label plus a one-column gap — so the typed command never
shifts sideways when the indicator appears or hides. The
label renders only when `input_indicator` is set: `[ERR]` in
`theme.error`, `[WRN]` in the new amber `theme.warning`
(defined for both light and dark themes).
The indicator is not yet wired live — `input_indicator`
stays `None` until the debounce lands (step E). Covered by a
render test and the theme contrast test; the input-panel
snapshot is updated for the six-column reservation.
Type-mismatched comparisons and `= NULL` / `!= NULL` in a
WHERE expression now yield WARNING diagnostics — the command
still parses and runs (the ADR-0026 §7 permissive posture is
unchanged), but the validity indicator can flag it before
submission.
Computed post-walk from the built command's `Expr` against
the table's column types: a Compare / Between / In with a
column operand and a non-null literal whose type the column
cannot hold, or a Compare with `=` / `!=` against NULL. New
catalog keys `diagnostic.type_mismatch` / `diagnostic.eq_null`.
This is ADR-0026's deferred step 5, folded into ADR-0027's
diagnostics-severity model as the user requested.
`MatchedKind::Ident` now carries its `IdentSource`. A
post-walk pass over a structurally-valid parse flags a
matched `Tables` ident that is absent from the schema, or a
`Columns` ident absent from the table in scope, as an ERROR
diagnostic — the command parses but would fail at execution
(ADR-0027 §2). New behaviour: an unknown table / column used
to parse cleanly and fail only when run.
Column scope is resolved by one left-to-right pass over the
matched path (every command places its table ident before
the columns that belong to it); an unknown table clears the
scope, so its columns are not cascaded into a second
diagnostic. New catalog keys `diagnostic.unknown_table` /
`diagnostic.unknown_column`.
Adds `Severity` (Error / Warning, ordered so Error > Warning)
and `Diagnostic { severity, span, message }` in
`walker::outcome`, plus a `diagnostics` field on `WalkResult`
— the schema-aware findings layered on a structurally-valid
parse (ADR-0027 §2).
`input_verdict(source, schema)` is the validity-indicator
entry point: `None` when the input would run clean (and for
empty input), `Some(Error)` for a parse failure or unknown
command, `Some(Warning)` for the ADR-0026 expression flags.
The verdict is the highest severity across the parse outcome
and the diagnostics set.
`diagnostics` is empty at this step — the schema-existence
(ERROR) and expression (WARNING) passes that fill it land
next. Covered by `input_verdict` unit tests.
Adds tests/typing_surface/where_expression.rs — 9 matrix
cells for the complex WHERE / show-data limit typing surface:
operator candidates after an operand, AND / OR after a
predicate, NOT, BETWEEN / IN bounds, and `show data`
where / limit.
Writing the cells surfaced a grammar bug. `predicate_tail`'s
`[NOT] negatable` branch started with `Optional(not)`, and an
Optional-first `Seq` always "commits" — so on an incomplete
input the walker's `Choice` returned that branch's
`Incomplete` early and discarded every sibling branch's
expected set, dropping `is` and the comparison operators from
completion after a column. Fixed by splitting it into
explicit `NOT negatable` and bare `negatable` branches — no
`predicate_tail` branch starts with an `Optional` now. The
matched terminal sequence is unchanged, so `build_expr` is
untouched.
Docs: ADR-0026 gains an "As-built notes" section recording
the option-1 builder realization, its two deviations from the
§3 sketch, and the deferral of §7 diagnostic flagging to
ADR-0027. requirements.md C5a -> [x] (steps 1-4) with the
test baseline refreshed to 1079; CLAUDE.md's deferred list
reconciled (C5a implemented; the QA1/QA2 note now points at
ADR-0028).
Wires the stratified WHERE-expression fragment into the three
filter commands and compiles the resulting Expr to SQL.
Grammar (data.rs): the `update` / `delete` `where` clause is
now the expression fragment (`Subgrammar(&expr::OR_EXPR)`) in
place of the single `col = val` slot; `show data` gains an
optional `where <expr>` and an optional `limit <n>` (a
non-negative integer, validated at parse time). The
expression's right-hand operands are a schema-aware
`DynamicSubgrammar` so the hint panel still narrows to the
left column's type (ADR-0026 §8) — but the inner grammar is
permissive: a type-mismatched literal still parses (§7).
AST: `RowFilter::Where{column,value}` -> `RowFilter::Where(Expr)`;
`ShowData` gains `filter: Option<Expr>` and `limit: Option<u64>`.
A `RowFilter::eq` convenience constructor keeps simple-equality
call sites and tests readable.
SQL (db.rs): `compile_expr` lowers an `Expr` to a
parameterised WHERE — every literal a `?` placeholder,
identifiers `quote_ident`-quoted, `<>` for inequality. A
literal compared against a column binds through that column's
type where compatible and falls back to its syntactic shape on
a mismatch (§7 — permissive). `show data ... limit n` emits
`LIMIT ?` with an implicit primary-key `ORDER BY`, so it is a
stable "first n by primary key".
completion.rs: `invalid_ident_at_cursor` no longer mis-flags a
digit-led literal (`1`) as an unknown column now that the
WHERE operand slot also accepts a column reference; a
`ProseOnly` slot suppresses keyword candidates even when the
expected set also carries a column ident.
11 db integration tests cover AND / OR / NOT, BETWEEN, IN,
LIKE, filtered `show data`, and limit ordering; walker and
expr unit tests cover the parse surface. Type-mismatch /
`= NULL` diagnostic flagging (§7 highlight + hint) is the
remaining ADR-0026 piece.
The stratified WHERE-expression grammar — or / and / not /
bool_primary / predicate tiers as named `static` Node
fragments, recursing through `Subgrammar`. Covers the six
comparison operators (`<>` and `!=` both NotEq), AND / OR /
NOT, parentheses, LIKE / IN / BETWEEN with optional infix NOT,
and IS [NOT] NULL. `predicate_tail` factors the shared operand
prefix and the infix NOT so the Choice branches discriminate
on a cleanly-failing first token.
New recursive Expr / Predicate / Operand / CompareOp AST in
dsl::command. `build_expr` folds the flat matched-terminal
slice into an Expr — a deterministic recursive descent
mirroring the grammar tiers, with single-child tiers
collapsing. Per ADR-0026 §3 option 1: the walker stays a pure
structural matcher; Expr is assembled only in this
submit-time fold.
Fragment + builder are unit-tested standalone (walk against
&OR_EXPR, then build_expr); not yet wired into any command.
New `Node::Subgrammar(&'static Node)` variant lets a named
static grammar fragment recurse through a reference — `Seq` /
`Choice` embed children by value and cannot close a cycle, but
a `&'static Node` can point back at an enclosing fragment. This
is the mechanism the stratified WHERE-expression grammar
(ADR-0026 §2) recurses through.
The walker counts active Subgrammar frames in
`WalkContext::subgrammar_depth` and refuses past
`MAX_SUBGRAMMAR_DEPTH` (64), surfacing a friendly
`parse.custom.expression_too_deep` error instead of a stack
overflow. Depth is saved/restored per frame so a
speculatively-walked-then-rolled-back Choice branch leaves no
residue.
No grammar references the node yet; covered by walker unit
tests with a small recursive `( x )` test grammar.
`create table … with pk` parsed column types as `name:type`,
while `add column` uses `name(type)`. Unify on the parens
form so column-type syntax is consistent across the DSL:
create table T with pk id(serial), name(text)
Only `COL_SPEC` changes (`:` → `( … )`); `build_create_table`
reads columns by role, so it is unaffected. The `:` that
separates table from column in `add column` / `drop column`
is unchanged. Sweeps the test suite, the typing-surface
matrix (two `after_colon` cells renamed to `after_paren`,
4 snapshots regenerated), the friendly catalog's usage
templates, ADR-0009's example, and requirements.md.
1039 passing / 0 failing / 1 ignored; clippy clean.
`push_history` skipped its `history_cursor` / `history_draft`
reset on the consecutive-duplicate early-return path. Recalling
a command with Up and re-submitting it unchanged left the cursor
stranded at that entry, so the next Up stepped backwards from
there instead of restarting at the newest entry.
Move the reset ahead of the early-return guards. Adds a Tier-1
regression test driving the recall/resubmit keystroke sequence.
Implement ADR-0025 — indexes as a DSL DDL feature.
- Grammar: `add index [as <name>] on <T> (<cols>)`, `drop index
<name>` / `drop index on <T> (<cols>)`, plus a `--cascade`
flag on `drop column`.
- db.rs: index operations over the engine's native index
catalog (no metadata table). The rebuild-table primitive now
captures and recreates indexes, so `change column` and the
relationship operations no longer silently drop them.
- `drop column` refuses an indexed column unless `--cascade`,
which drops the covering indexes and reports each.
- Persistence: additive `indexes:` list in `project.yaml`
(version unchanged); round-trips through rebuild/export/import.
- Display: an `Indexes:` section in the structure view and a
nested tables/indexes items panel (S2).
Reconciles requirements.md (C3 index portion, S2 satisfied)
and CLAUDE.md. 1038 tests passing (+31), clippy clean.
HintMode became per-node (Node::Hinted) in the node-attached refactor;
the per-command hint_mode field was never the mechanism and is now
read by nothing. Removed the field and its 20 `None` initialisers.
Every CommandNode declared a help_id that nothing read; the in-app
`help` body was a single hand-kept catalog block that drifted from
the command set (handoff-12 §2.1).
note_help now iterates the command REGISTRY and translates each
CommandNode's help_id (`help.<id>`), framed by help.intro /
help.dsl_section / help.types_reference. A newly-registered command
appears in `help` automatically — no edit to note_help or a hand-kept
list. Added 20 per-command help entries plus the 3 framing entries;
removed help.in_app_body.
Per-command entries use block scalars: a libyml 0.0.5 scanner bug
panics on long internal space runs in double-quoted scalars, and the
entries are space-aligned.
run_replay parsed each line with the schemaless parse_command, so
Phase D typed-slot rejections (wrong-count value lists, wrong-type
column values) fired only at bind time during replay — inconsistent
with the interactive path (handoff-12 §2.1).
run_replay now re-snapshots the schema per line (the schema mutates
as replayed create-table / add-column commands run) and parses with
parse_command_with_schema. Extracted build_schema_cache, shared with
the interactive refresh_schema_cache.
Added a replay integration test asserting a typed-slot violation is
caught at parse time (through the replay.error_parse wrapper).
Form C (`insert into T (vals)`) shared the `(` opener with Form A,
so its paren was an untyped Repeated(Choice(literal, ident)) — values
weren't type- or count-checked at parse time (handoff-12 §2.2).
New Node::Lookahead variant: a factory that peeks the source. The
insert first-paren factory inspects the first token — a value literal
routes the contents through the typed column_value_list (Form B
dispatch contract: per-non-auto-column typed slots); an identifier or
empty paren routes to a Form A column-name list. So Form C now gets
the same per-column typed slots, hints, and parse-time type/count
checking Form B has.
The explicit-Choice-branch split is impossible here (committed-choice
semantics commit after `(` matches); lookahead is the only route, and
DynamicSubgrammar factories couldn't see the source. Node::Lookahead
is not memoized — its output depends on source — but it returns only
a small node (a Repeated, or a thin DynamicSubgrammar wrapper that
delegates to the memoized column_value_list).
`insert into T (` now cleanly shows Form A column candidates instead
of mixed Form-A/C suggestions. Form C matrix tests updated for the
type-aware behaviour.
Node::DynamicSubgrammar factories build a Node from the WalkContext and
must Box::leak it (the Node enum's combinator children are &'static).
Leaking per walk grew unbounded under per-keystroke completion
(handoff-12 §2.1).
resolve_dynamic now memoizes on the schema state a factory reads
(table columns, current column, user-listed columns) keyed by factory
fn-pointer. Each distinct value-list shape leaks exactly once — total
leak bounded by distinct (schema × form) combinations, not keystroke
count. TableColumn gains Hash for the cache key.
The handoff's original arena sketch needed a lifetime-generic Node
(major refactor); memoization gets the same bound without it.
Replaces the hint resolver's signature-matching (does the expected set
contain all five literal forms? an Ident{NewName}?) with a grammar-
declared annotation. New Node::Hinted { mode, inner } wrapper; the
walker records the mode in WalkContext::pending_hint_mode on entry and
clears it on any successful match (cursor moved past the slot — this
also undoes the leak where a failed Hinted branch of a Choice would
otherwise strand a stale mode). The resolver reads pending_hint_mode
directly.
Value-literal fallback slots carry ProseOnly; NewName ident slots carry
ForceProse. hint_mode_at_input_inner now delegates to
hint_resolution_at_input — one resolution path, no duplicated logic.
No behaviour change; the typing-surface matrix guards it.
Handoff-12 §2.2: Form B `insert into T values (…)` silently skips
auto-generated columns from the value list, so a user who wants to
set a serial/shortid column explicitly could only discover Form A by
reading help. Now the hint at the first Form B value slot appends a
note naming the skipped column(s) and pointing at the explicit-column
form.
hint_resolution_at_input derives the skipped columns from the
post-walk WalkContext (Form B = no user_listed_columns + table has
serial/shortid columns) and reports them on HintResolution; the note
fires only at the first slot so it doesn't repeat at every comma.
ambient_hint composes it onto the per-column prose.
The 5 structural-class entries (identifier/number/string_literal/flag/
end_of_input) and 3 lex-error entries (bad_flag/unknown_char/
unterminated_string) were unreachable: ADR-0024 Phase F made the
walker render keyword wording verbatim and the lex errors never
surface through today's walker. Handoff-12 §2.1 kept them as a
conservative call; removed now per user request. Re-adding is cheap
if a future need arises.
Handoff-12 §2.2: the DDL TABLE_NAME_EXISTING slot and the
relationship-endpoint table idents didn't set writes_table, so
column-name slots downstream (drop/rename/change column; relationship
qualified columns) couldn't narrow to the active table — candidates
leaked from every table. Set writes_table: true on TABLE_NAME_EXISTING
and on DR_PARENT/DR_CHILD/AR_PARENT/AR_CHILD table idents. The
deliberately-documenting completion test now asserts per-table
narrowing.
classify_input was schemaless; wrong-count Form B value lists
(`insert into Customers values ('Alice')` against a 3-column table)
showed as Valid until submit. Add classify_input_with_schema that
threads the SchemaCache through parse_command_with_schema and wire
render_input_runs to use it. Schemaless classify_input is kept public
for handoff-11/12 regression tests that exercise schema-independent
positions.
The typing-surface matrix exposed two bugs the existing 859-test suite
missed:
walk_repeated: when the separator consumed but the inner item failed
at EOF, the old path rolled the separator back and reported a definite
error at the rollback position (`insert into T (a, ` flashed red on
the `,` after each comma). Now propagates Incomplete with the inner's
expected set so the input renderer treats it as mid-typing.
build_insert Form C path: `insert into T (col)` walked to a complete
match but produced `values: []` because Form C's value collector drops
ident-shaped items. The user almost certainly meant Form A and just
hasn't typed `values (...)` yet. Reject with a ValidationError naming
the Form-A continuation; classify_input now reports IncompleteAtEof.
completion_probe / expected_at_input: ValidationFailed used to return
an empty expected set, leaving Tab with nothing to offer at the new
Form-A flag point. Now surface result.tail_expected (skipped-Optional
expectations captured before validation fired) so `values` is still
offered as a candidate.
Two related fixes:
1. \`update MyTable set \` was offering columns from every
table in the project — completion fetched
\`cache.for_source(IdentSource::Columns)\` which returns the
flat \`cache.columns\` (union of every table's columns).
The walker's WalkContext had \`current_table_columns\`
populated (because the update-table-name slot is
\`writes_table: true\`) but the completion engine never
consulted it.
2. \`insert into MyTable (\` was offering nothing — the
value-literal suppression fired because the expected set at
this position contains both Form A column-list candidates
(\`Ident{Columns}\`) and Form C bare-value-list literals
(null/true/false/NumberLit/StringLit). \`is_value_literal_signature\`
matched and the engine returned \`None\` before the column
candidates were considered.
The fix threads the walker's \`current_table_columns\` through
to the completion engine and narrows the suppression rule:
**Walker:**
- New \`walker::CompletionProbe { expected, current_table_columns }\`
struct.
- New \`walker::completion_probe(source, schema) -> CompletionProbe\`
runs one schema-aware walk and reports both the expected
set (or tail_expected on Match) and the resolved table-column
snapshot.
**Completion engine:**
- \`candidates_at_cursor_with\` calls \`completion_probe\` and
reads \`current_table_columns\` for the \`Columns\` ident
source. Schemaless or unknown-table falls back to the flat
\`cache.columns\` (preserves pre-fix behavior).
- Value-literal suppression now gated on
\`!has_schema_ident\` — if the expected set also offers a
schema-listable Ident, the user has actionable candidates
beyond the misleading null/true/false trio and we shouldn't
hide them.
Tests:
- \`update_set_offers_only_current_table_columns\` confirms
Customers' columns appear while Orders' columns don't.
- \`update_where_offers_only_current_table_columns\` covers
the where path.
- \`insert_into_open_paren_offers_current_table_columns\` and
\`insert_into_open_paren_does_not_offer_unrelated_columns\`
cover the Form A column-list position.
- \`drop_column_from_offers_only_current_table_columns\`
documents the DDL fallback (drop-column's table-name slot
doesn't currently \`writes_table\` — falls back to the flat
list).
For the user: \`update MyTable set \` now offers only
MyTable's columns. \`insert into MyTable (\` offers all of
MyTable's columns so Form A is fully discoverable.
Tests: 859 passing, 0 failing, 1 ignored. Clippy clean.
Two related fixes from a user-reported snag:
1. After typing \`insert into Orders \`, the hint suggested only
\`values\` even though the user could also choose \`(\` to
open Form A (the explicit-column-list variant). The walker
reports both \`Expectation::Word("values")\` and
\`Expectation::Punct('(')\` at that position, but
\`candidates_at_cursor\` had a blanket "no punctuation as Tab
candidate" policy.
Loosened the policy to surface branching punct
(specifically \`(\` opening a sub-shape). Closing punct
(\`)\`), separators (\`,\`), and content-trailing punct (\`:\`,
\`=\`, \`.\`) stay out — the user types those naturally and
advertising them in the Tab menu is noise. New
\`CandidateKind::Punct\` so the renderer colors it as punct
rather than mis-classifying as a keyword.
2. While typing \`insert into Orders (id, CustId, Total) values
(42, 89, 17.59\` (no closing paren yet), the word \`values\`
was rendered in \`tok_error\` red. The walker's
\`Optional(Seq[values, '(', list, ')'])\` was rolling back on
the partial inner match — treating \`(id, CustId, Total)\` as
Form C (bare value list) followed by trailing junk starting
at \`values\`. The classify_input call thus returned
\`DefiniteErrorAt(<values byte>)\` and the renderer overlaid.
Tightened \`walk_optional\`: roll back only when the inner
reports NoMatch (or Incomplete / Mismatch without consuming
anything). Once the inner has committed to at least one
terminal (e.g. matched the \`values\` keyword), propagate
Incomplete / Mismatch up — the user is mid-typing the
optional's content and rolling back would lose their
intent.
The pre-existing chumsky-or_not-style aggressive rollback
covered cases like \`save Customers\` (Optional(\`as\`)
inner is a single Word that returns NoMatch without
consuming, so rollback still fires). Those keep working.
3. Side effect: with \`Optional\` no longer hiding the
in-progress Form A from the leading slice, the walker on
\`create table T with \` correctly reports the next-expected
keyword as \`pk\` — so cursor at the end of the complete
command \`create table T with pk\` would now re-offer \`pk\`
as a Tab candidate against the partial \"pk\". Added a final
filter: when the full input is a valid parse AND the
partial prefix is non-empty, drop candidates that equal the
partial exactly. Preserves schema narrowing
(\`show data Cu\` → \`Customers\` is not an exact match).
Tests:
- New \`in_progress_form_a_values_list_classifies_as_incomplete\`
asserts the input-state for the user's exact scenario.
- New \`open_paren_branching_punct_surfaces_after_insert_into_table\`
and \`open_paren_candidate_is_classified_as_punct_kind\` cover
the punct-as-candidate surface.
- Renamed and rewrote \`punctuation_expected_does_not_produce_candidates\`
to \`non_branching_punctuation_is_not_surfaced_as_candidate\`
to document the new finer-grained policy.
- Existing tests for \`save Tab → as\` and the schema-
narrowing case continue to pass.
Tests: 854 passing, 0 failing, 1 ignored. Clippy clean.
Bug: hint at \`insert into Customers values (\` for a Customers
table with id:serial PK suggested typing an integer for \`id\`,
but the dispatch path (\`db::do_insert\`) deliberately doesn't
accept user-supplied values for auto-generated columns in
Form B. The grammar prompted for a value the dispatch would
refuse.
The fix aligns Phase D's \`column_value_list\` dynamic sub-grammar
with do_insert's three forms (ADR-0014 + ADR-0018 §3):
- **Form A** \`insert into <T> (col1, col2, …) values (…)\` —
user explicitly lists columns. Slot list mirrors that
selection; serial / shortid columns CAN appear if the user
lists them.
- **Form B** \`insert into <T> values (…)\` — bare values. Slot
list = non-auto-generated columns of the table in
declaration order. Serial / shortid get auto-filled by the
dispatch; the grammar doesn't prompt for them.
- **Form C** \`insert into <T> (v1, v2, …)\` — bare value list.
Not affected by this change (column_value_list isn't on this
path; Form C's literals route through the schemaless
INSERT_PAREN_LIST).
Implementation:
\`WalkContext.user_listed_columns: Option<Vec<String>>\` — when
\`Some\`, signals Form A; \`None\` is Form B. Populated by walking
the first paren's column-list idents.
\`Node::Ident.writes_user_listed_column: bool\` — new field;
\`true\` on the INSERT_PAREN_ITEM's Ident child. When the
walker matches that ident in Form A, it appends the
schema-canonical column name (case-corrected against the
schema) to user_listed_columns.
\`column_value_list\` factory:
- If user_listed_columns is Some → resolve each name from the
schema; one typed slot per listed column.
- Else → filter current_table_columns to non-auto-generated;
one typed slot per remaining column.
- Empty result → fall back to the schemaless value-literal
list (a serial-only table in Form B has nothing for the
user to type).
Tests:
- New \`phase_d_insert_form_b_skips_serial_column\` confirms the
bug: \`insert into Customers values (1, 'Alice')\` against a
Customers with serial id rejects at parse time (Form B
expects 1 value for Name, not 2).
- New \`phase_d_insert_form_a_accepts_serial_when_listed\`
confirms \`insert into Customers (id, Name) values (1, 'Alice')\`
works.
- New \`phase_d_insert_form_a_filters_to_user_listed_columns\`
confirms partial Form A (\`(Name) values ('Alice')\`).
- Updated \`phase_d_insert_with_schema_accepts_typed_values_per_column\`
to match the new Form B contract (2 user-typed values, not 3).
- Updated typed-hint test matrix split into form-B (8 types)
and form-A (serial / shortid).
- New \`typed_hint_form_b_skips_serial_column_to_generic_or_text_neighbor\`
pins the fallback behavior for a serial-only table.
For the user: \`insert into Customers values (\` for a Customers
with \`(id:serial, Name:text, Email:text)\` now hints
\`for \`Name\`: Type a quoted string …\` (skipping id entirely)
and accepts exactly 2 values. To set the serial explicitly,
use Form A: \`insert into Customers (id, Name, Email) values
(1, 'Alice', 'a@b.c')\`.
Tests: 851 passing, 0 failing, 1 ignored. Clippy clean.
User-facing improvement: typing into a value slot now surfaces
the column name in the hint. The hint at `insert into Customers
values (` (first column id:int) reads "for `id`: Type an
integer (e.g. 42, -7) or null" instead of the generic
"Type an integer …" prose. After `1, ` the panel updates to
the second column ("for `Name`: Type a quoted string …"). The
same applies to `update T set Email=` and `delete from T where
ts=` — the catalog wrapper threads the column name through.
Implementation:
**`Node::TypedValueSlot.column_name: Option<&'static str>`**
(new field, `src/dsl/grammar/mod.rs`). When `Some`, walker
writes `WalkContext::pending_value_column` on entry; clears
along with `pending_value_type` on inner success.
**Walker driver writes both names** (`src/dsl/walker/driver.rs`):
- `Node::TypedValueSlot` dispatch reads `column_name` and
populates `pending_value_column`.
- `Ident { writes_column: true }` dispatch also writes
`pending_value_column` (using the schema-canonical name when
available, falling back to the user's spelling) so update
set / where positions surface the column name.
**Shared sub-grammars** (`src/dsl/grammar/shared.rs`):
- New `slot_for_column(ty, name)` builds a `TypedValueSlot`
with the embedded leaked column name. Used by
`column_value_list`.
- New `slot_inner_for_type(ty)` returns just the Choice
(without TypedValueSlot wrapper) for slot_for_column to
rebuild.
- `column_value_list` factory now constructs per-column slots
via `slot_for_column(col.user_type, &col.name)`. Each slot
leaks its column name string with the same per-walk Box::leak
pattern the rest of dynamic dispatch uses.
**`WalkContext::pending_value_column: Option<String>`** (new
field, `src/dsl/walker/context.rs`). Pairs with
`pending_value_type` to give the hint resolver both pieces.
**Single-walk hint resolver** (`src/dsl/walker/mod.rs`):
- New `HintResolution { mode: HintMode, column: Option<String> }`
struct.
- New `hint_resolution_at_input(source, schema) -> Option<
HintResolution>` runs one walk and reports both pieces. The
ambient_hint dispatch composes per-column prose from the
result.
- Existing `hint_mode_at_input` / `hint_mode_at_input_with_schema`
preserved as thinner wrappers for tests / future callers
that don't need the column name.
**Catalog wrapper** (`src/friendly/strings/en-US.yaml`,
`src/friendly/keys.rs`):
- New `hint.value_slot_for_column: "for `{column}`: {detail}"`
prefixes the per-type prose with the actual column name when
the walker has it bound. Schemaless fallback continues to use
the generic value-literal prose with no column prefix.
**ambient_hint composes** (`src/input_render.rs`): consults
`hint_resolution_at_input`; when `column` is `Some`, wraps the
type prose through `hint.value_slot_for_column`; otherwise
emits the bare type prose.
Tests (846 total, 0 failing):
- 4 new input_render tests assert column names appear in the
prose at insert/update/where positions plus the
second-insert-value position (proves column tracking advances
with comma).
- All existing tests pass unchanged — the column-name addition
is layered on top of the type-only prose path.
Clippy clean.
The Phase D commit landed parse-time validation but not the
user-facing payoff — per-column-type hints. Typing
`insert into Customers values (` rightfully expected a hint
like "Type an integer (e.g. 42, -7) or null" at an int column.
This commit closes that gap.
End-to-end:
**`Node::TypedValueSlot { ty, inner }`** (new variant in
`src/dsl/grammar/mod.rs`):
- Walker walks `inner` to consume the literal but tags
`WalkContext::pending_value_type = Some(ty)` on entry, then
clears it on a successful inner match. Positions BETWEEN
slots (`insert into T values (1` mid-input) thus don't carry
a stale hint type.
**Typed slot factories wrapped in `TypedValueSlot`**
(`src/dsl/grammar/shared.rs`):
- `INT_SLOT`, `REAL_SLOT`, `DECIMAL_SLOT`, `BOOL_SLOT`,
`TEXT_SLOT`, `DATE_SLOT`, `DATETIME_SLOT`, `BLOB_SLOT`,
`SERIAL_SLOT`, `SHORTID_SLOT` — each pairs an inner literal
Choice with its `Type` so the walker can tag context.
- `slot_for_type(ty)` dispatches to the appropriate constant.
- Bug fix: `ShortId` previously dispatched to `INT_SLOT` (a
pre-Phase-D holdover from the chumsky-side generic
fallback). `shortid` columns store base58 text (ADR-0011
fk_target_type shortid → text); the corrected slot accepts
`StringLit` or `null`.
**Schema-aware hint resolver** (`src/dsl/walker/mod.rs`):
- `hint_mode_at_input_with_schema(source, &SchemaCache) ->
Option<HintMode>` is the new public entry point. Reads
`pending_value_type` from the walker's WalkContext and
emits `HintMode::ProseOnly("hint.value_slot_<type>")` —
one per Type.
- The schemaless `hint_mode_at_input(source)` falls back to
the generic `hint.value_literal_slot` at value-literal slots
(no per-type narrowing without a schema).
- `catalog_key_for_value_type(ty)` is the type → key
dispatcher.
**Catalog entries** (`src/friendly/strings/en-US.yaml`,
`src/friendly/keys.rs`):
- 10 new `hint.value_slot_<type>` keys with per-type prose:
- int/serial → "Type an integer (e.g. 42, -7) or null"
- real/decimal → "Type a number (e.g. 3.14, -0.5) or null"
- bool → "Type true, false, or null"
- text → "Type a quoted string (e.g. 'Alice') or null"
- date → "Type a quoted date as 'YYYY-MM-DD' or null"
- datetime → "Type a quoted datetime as 'YYYY-MM-DD
HH:MM:SS' or null"
- blob → "Type a quoted blob literal or null"
- shortid → "Type a quoted shortid (or omit to auto-generate)
or null"
**Ambient-hint dispatch** (`src/input_render.rs::ambient_hint`):
- Passes the SchemaCache through to
`hint_mode_at_input_with_schema`, so the live hint panel
surfaces per-column-type prose as the user types into a
value slot.
Tests:
- 8 walker-side tests cover insert / update / where typed-slot
hint dispatch, mid-value no-stale-hint behaviour, and a
full-coverage routing matrix for every `Type` variant.
- 4 input_render integration tests cover the end-to-end
ambient_hint path: insert first/second value, update set
value, and the schemaless fallback to generic prose.
Tests: 842 passing, 0 failing, 1 ignored. Clippy clean.
For the user: typing `insert into Customers values (` against
a Customers table whose first column is `id:int` now shows
"Type an integer (e.g. 42, -7) or null" in the hint panel,
replacing the previous generic value-literal prose. After
typing `1, `, the panel updates to whatever the second column
requires — "Type a quoted string (e.g. 'Alice') or null"
for text, "Type a quoted date as 'YYYY-MM-DD'" for date, etc.
Pre-Phase-D, `save ` parsed as a complete `save` command, so
the completion engine had nothing to mine: `as` never surfaced
as a Tab candidate. This is the round-5 gap the handoffs have
been tracking.
`WalkResult` gains a `tail_expected: Vec<Expectation>` field.
The walker's top-level `Matched` branch copies the outer
shape's skipped-Optional expectations into it. `expected_at_input`
returns `tail_expected` on `Match` so the completion engine
sees the optional-suffix continuations and offers them as Tab
candidates.
`hint_mode_at_input` deliberately does NOT consume
`tail_expected` — surfacing prose like "Type a name" at the
end of a valid command would be misleading. A new private
`expected_for_hint` returns empty on `Match` to preserve this.
The split distinguishes "valid + could continue" (completion
helps) from "invalid + must continue" (hint resolver helps).
Tests:
- `save ` Tab → `as` (new test, the original round-5 gap).
- `messages ` Tab → `short` and `verbose` (same shape).
- Existing `hint_mode_none_for_complete_command` stays green
because hint resolver ignores tail_expected.
- 830 total passing, 0 failing, 1 ignored. Clippy clean.
Schema-aware typed value slots — the central design claim of
ADR-0024 §Phase D. Insert / update / delete value slots now
dispatch on the user-facing column type at parse time, rejecting
mis-shaped input with localised wording instead of waiting for
the bind-time error.
What changed:
**SchemaCache extension** (`src/completion.rs`):
- New `TableColumn { name, user_type }` for per-table column
metadata.
- `SchemaCache.table_columns: HashMap<String, Vec<TableColumn>>`.
- `SchemaCache::columns_for_table(name)` — case-insensitive
lookup, mirrors the walker's case-insensitive entry-word
resolution.
**WalkContext schema plumbing** (`src/dsl/walker/context.rs`):
- `WalkContext<'a>` gains a lifetime and a `schema: Option<&'a
SchemaCache>`. `WalkContext::new()` keeps the schemaless
default; `with_schema(s)` is the new schema-aware constructor.
**Parser entry point** (`src/dsl/parser.rs`):
- `parse_command_with_schema(input, schema)` is the new public
schema-aware variant. `parse_command(input)` becomes a thin
wrapper that delegates with `None` for back-compat.
- Internal `try_walker_route` accepts an `Option<&SchemaCache>`
and threads it into the WalkContext.
**Node::Ident writes_table/writes_column** (`src/dsl/grammar/mod.rs`):
- Two new fields on `Node::Ident`. When `writes_table: true` and
`source: Tables`, the walker writes the matched ident's name
into `current_table` and resolves `current_table_columns`
against the schema cache. When `writes_column: true` and
`source: Columns`, the walker writes the resolved
`TableColumn` into `current_column`.
**Walker driver DynamicSubgrammar dispatch** (`src/dsl/walker/driver.rs`):
- The `Node::DynamicSubgrammar(factory)` branch now resolves the
factory at walk time and `Box::leak`s the result so its inner
static-slice fields (Choice/Seq) have the lifetime the walker
expects (per ADR-0024 §sub-grammars). The leak is bounded by
command-shape complexity per walk; per-walk arena is a future
optimisation.
- `walk_ident` extends to perform the schema writes when the
flags are set.
**Typed value slot factories + dynamic sub-grammars** (`src/dsl/grammar/shared.rs`):
- `int_slot` / `real_slot` / `decimal_slot` / `bool_slot` /
`text_slot` / `date_slot` / `datetime_slot` / `blob_slot` —
one per `Type`. Each accepts the appropriate literal kind plus
`null`; integer-only validator rejects `3.14` at int columns;
decimal validator pins numeric shape.
- `slot_for_type(ty) -> Node` is the dispatcher.
- `current_column_value(ctx) -> Node` is the dynamic sub-grammar
for `set col = …` and `where col = …` values; reads
`current_column` and dispatches via `slot_for_type`.
- `column_value_list(ctx) -> Node` is the dynamic sub-grammar
for `insert into T values (…)`; reads `current_table_columns`
and unfolds a Seq of typed slots separated by commas.
- Both fall back to the schemaless `VALUE_LITERAL` choice when
the context lacks the schema-resolved entries — keeps
schemaless `parse_command` callers (tests, replay path)
working.
**Data-command grammar wires the new types** (`src/dsl/grammar/data.rs`):
- `TABLE_NAME_INSERT` / `TABLE_NAME_WRITES` (new): table-name
slots that set `writes_table: true`. Used by insert / update /
delete to populate `current_table_columns`.
- `SET_COLUMN` / `FILTER_COLUMN` (new): column-name slots in
`set col=…` / `where col=…` set `writes_column: true`.
- `INSERT_VALUES_LIST` becomes `DynamicSubgrammar(column_value_list)`.
- `UPDATE_ASSIGNMENT` and `WHERE_CLAUSE` use
`PER_COLUMN_VALUE = DynamicSubgrammar(current_column_value)`.
**Runtime plumbs schema-with-types** (`src/runtime.rs`):
- `refresh_schema_cache` calls `describe_table` for each table
and populates `SchemaCache::table_columns` with
`TableColumn { name, user_type }` entries. Best-effort: a
`describe_table` miss leaves that table unpopulated and the
walker falls back to schemaless dispatch.
**App dispatches with schema** (`src/app.rs`):
- `dispatch_dsl` routes through `parse_command_with_schema(&self
.schema_cache, …)` so live typing/dispatch sees the typed
slots. The replay path stays schemaless (deferred — replay
bind-time errors still catch type mismatches).
**Catalog** (`src/friendly/strings/en-US.yaml`, `src/friendly/keys.rs`):
- New `parse.custom.bind_type_mismatch` entry with `{found}` and
`{expected}` placeholders. Surfaced by the int_slot /
decimal_slot validators.
Tests:
- 11 new walker-side Phase D tests cover insert / update /
delete with schemas — typed acceptance per column, decimal
rejection at int columns, null acceptance at any slot,
multi-assignment per-column dispatch, schemaless fallback.
- The pre-existing `parse_command(input)` test suite (no
schema) still passes — the fallback path is behaviour-
preserving.
- 828 passing total, 0 failing, 1 ignored. Clippy clean.
Adds the `HintMode` dispatch layer the ADR specified: the
ambient-hint resolver now consults a single
`walker::hint_mode_at_input(source) -> Option<HintMode>` to
decide between the prose / candidates ladder, rather than
discovering each slot kind through three separate post-hoc
helpers (`value_literal_hint_at_cursor`,
`typing_name_at_cursor`, and so on).
Behaviour at slot positions today:
- **Value-literal slot** (`null`/`true`/`false`/number/string
all in the expected set) → `HintMode::ProseOnly
("hint.value_literal_slot")`. The ambient-hint ladder
emits the catalog prose at empty prefix; once the user types
a partial (`n`, `tr`, `fa`) the partial check declines and
normal candidate completion takes over.
- **NewName ident slot** → `HintMode::ForceProse
("hint.ambient_typing_name")`. The ladder still consults
`typing_name_at_cursor` to learn what comes after the name
(the post-name probe is unchanged); `ForceProse` is the
declarative tag telling the resolver *that* we're in this
mode.
`HintMode` itself gains `PartialEq + Eq` for tests, and
its docstring is rewritten to describe the live semantics.
This is the structural shape ADR-0024 §HintMode-per-node
describes: one slot → one hint mode → one dispatch arm. The
detection inside `hint_mode_at_input` is transitional — it
pattern-matches the walker's expected-set today, which is
exactly what the previous ad-hoc detectors did. Phase D will
replace the signature match with node-attached `HintMode`
annotations on the typed value slots (so `date_slot`,
`int_slot`, etc. each carry a type-specific catalog key).
Two helpers move into `input_render.rs`:
- `hint_leading_slice(input, cursor)` mirrors the look-back
used by `candidates_at_cursor` so the hint resolver sees the
same token-boundary view of the world.
- `cursor_partial_is_empty(input, cursor)` distinguishes
empty-prefix from in-progress identifier shapes.
8 new walker tests pin the hint-mode resolver across
value-literal-after-paren, value-literal-after-set-assign,
value-literal-in-where, two NewName-slot cases, the
entry-keyword position, the complete-command position, and
the schema-ident position.
Tests: 817 passing, 0 failing, 1 ignored. Clippy clean.
Adds the `Ranker` plug-in point ADR-0024 §ranker-layer
specified. The grammar tree declares *what's valid*; the
ranker decides *what's likely useful first*. Default
`identity_ranker` preserves declaration order from the
grammar.
API:
- `pub type Ranker = fn(Vec<Candidate>) -> Vec<Candidate>;`
- `pub const fn identity_ranker(c) -> Vec<Candidate>` returns
its input unchanged.
- `candidates_at_cursor_with(input, cursor, cache, ranker)`
applies a custom ranker; the default `candidates_at_cursor`
delegates with `identity_ranker`.
Three new tests cover identity preservation, custom reordering,
and the empty-list-collapses-to-None edge.
This is a future-work hook — no production caller passes a
non-identity ranker yet. Hooks for frequency-based ranking,
content-aware priors, or recency plug in here without touching
the grammar declarations.
Tests: 809 passing, 0 failing, 1 ignored. Clippy clean.
Replaces the `ParseError::Invalid::expected: Vec<String>`
round-trip with structured `Expectation`s direct from the walker
(ADR-0024 §architecture). The completion engine no longer parses
formatted strings back into types — `Expectation::Ident { source,
role }`, `Expectation::Word`, `Expectation::Literal`,
`Expectation::Flag`, `Expectation::NumberLit`, and
`Expectation::StringLit` are consumed as enum variants.
New helper:
- `walker::expected_at_input(source) -> Vec<Expectation>`
consolidates the empty-input case (returns every CommandNode
entry word), unknown-command-word case (also entry words), and
walker-engaged case (Incomplete / Mismatch expectations) in one
place. ValidationFailed and Match resolve to empty.
`completion.rs` refactor:
- `expected_at(leading)` wraps the walker helper; replaces the
legacy string-based `expected_set`.
- Keyword candidates: filter `Expectation::Word(w)` /
`Expectation::Literal(s)` to alphabetic-only literals (no
more string-parsing / `strip_backticks`).
- Type names: detect `Expectation::Ident { source:
IdentSource::Types }` directly (replaces the `TYPE_SLOT_LABEL`
magic string).
- Flag candidates: read `Expectation::Flag(body)` and format
as `--{body}` (replaces backticked-string matching).
- Composite-literal candidates: match against
`Expectation::Literal("1")` (replaces the backticked-string
`` `1` ``).
- Schema identifiers: `Expectation::Ident { source, .. }`
filtered by `source.completes_from_schema()`.
- `is_value_literal_signature` checks for `Expectation::Word`
values "null"/"true"/"false" and `Expectation::NumberLit` +
`Expectation::StringLit` variants (replaces backticked-string
matching).
- `invalid_ident_at_cursor` and `typing_name_at_cursor` adopt
the same path.
The `typing_name_at_cursor` probe (substitute placeholder and
re-parse) still goes through `parse_command` because the probe
specifically wants the *post-name* expected set — `parse_command`
+ the string `expected` field carries that today. A future
follow-up could thread the structured probe through `walker`,
but the value-add is marginal.
`COMPOSITE_CANDIDATES` opener key changes from `` `1` `` (the
backticked-string the chumsky parser produced) to bare `"1"`
(the Expectation::Literal payload).
Touched modules: `dsl/walker/mod.rs` (new export),
`src/completion.rs` (refactor).
Tests: 806 passing, 0 failing, 1 ignored — every existing
completion test passes unchanged, proving the structured path
is behaviour-preserving. Clippy clean.
Drops the 47 `parse.token.keyword.*` and 6 `parse.token.punct.*`
catalog entries (and their `KEYS_AND_PLACEHOLDERS` declarations).
Nothing consumes them: the walker renders keyword wording in
`format!(\"`{word}`\")` directly, sourced from grammar-tree Word
literals; punct wording surfaces the same way via
`Expectation::Punct(ch)`.
Structural-class labels (`parse.token.identifier`,
`parse.token.number`, `parse.token.string_literal`,
`parse.token.flag`, `parse.token.end_of_input`) and the lex-error
wordings (`parse.token.error.{bad_flag,unknown_char,
unterminated_string}`) stay. These are not derivable from the
grammar tree and the walker's expected-set / validator paths still
read them.
`friendly::keys::tests::keys_validate_against_catalog` continues to
assert catalog ↔ `KEYS_AND_PLACEHOLDERS` bidirectional coverage,
so the trimmed declaration is pinned against the trimmed catalog.
Tests: 806 passing, 0 failing, 1 ignored. Clippy clean.
Removes the last consumers of `dsl::lexer`, `dsl::keyword`, and
`dsl::ident_slot`, then deletes the modules.
- `Theme::token_color(&TokenKind)` deleted along with its test;
`Theme::highlight_class_color(HighlightClass)` is the sole
highlight-colour mapper (the walker's `per_byte_class` feeds
it directly).
- `IdentSource` (`dsl::grammar`) absorbs the schema-list /
expected-label / round-trip semantics that previously lived
on `IdentSlot`. Adds `completes_from_schema`, `expected_label`,
and `from_expected_label` methods. The walker's
`Expectation::Ident { source }` and the schema-lookup request
on the database worker now share one enum.
- `SchemaCache::for_slot(IdentSlot)` → `for_source(IdentSource)`.
- `Database::list_names_for` and the `Request::ListNamesFor`
worker variant take `IdentSource`. Internal tables and column
/ relationship lookups dispatch on the same enum.
- `InvalidIdent.slot: IdentSlot` → `InvalidIdent.source: IdentSource`.
The `invalid_ident_at_cursor` rendering branch in
`input_render.rs::ambient_hint` updates accordingly.
- Completion's keyword filter (`Keyword::from_word`) becomes
"backticked items whose payload is all ASCII alphabetic" —
punct and digit literals still surface through their own
candidate sources (composite-literal, flag, schema-ident);
the alphabetic filter excludes them from the keyword bucket.
- `friendly::keys::tests::keyword_and_punct_have_complete_token_vocabulary`
is dropped. It cross-checked `Keyword::ALL` / `Punct::ALL`
against catalog entries; both enums are gone. The
`parse.token.keyword.*` / `parse.token.punct.*` catalog
entries themselves survive for one more commit (catalog
cleanup, ADR-0024 §cleanup-pass); the
`keys_validate_against_catalog` test still pins them.
- Modules deleted: `src/dsl/lexer.rs`, `src/dsl/keyword.rs`,
`src/dsl/ident_slot.rs`.
Tests: 806 passing, 0 failing, 1 ignored. The drop from 852
reflects the removed module-internal tests (~32 lexer, 7
keyword, 4 ident_slot, 1 theme token_color, 1 friendly keys
keyword/punct), and is the expected outcome.
Clippy clean with `nursery` lints + `-D warnings`.
Migrates parse-error usage-block rendering from the legacy
`dsl::usage::matched_entry` (which scanned a `Vec<Token>` for the
first matched Keyword) to walker-side lookup driven by each
`CommandNode`'s `usage_ids` slice.
`CommandNode.usage_id: Option<&'static str>` becomes
`usage_ids: &'static [&'static str]`. Multi-form families
(`drop`, `add`, `show`) carry every variant — `drop` lists
table/column/relationship templates; `add` lists column /
relationship; `show` lists data / table. The single-shape
commands carry their single catalog key.
App-lifecycle CommandNodes had pointed at non-existent
`parse.usage.app.*` keys (never noticed because the field was
unused); they now point at the real catalog entries
(`parse.usage.quit`, `parse.usage.help`, …).
New helpers in `dsl::grammar`:
- `usage_keys_for_input(source) -> Option<(entry_word, usage_ids)>`
resolves the first identifier-shape token to a CommandNode and
returns its usage_ids list. Used by `app::render_usage_block`
and `input_render::ambient_hint`.
- `entry_words_alphabetised() -> Vec<&'static str>` replaces
`dsl::usage::entry_keywords_alphabetised`.
`dsl::usage` is deleted. The "available commands:" fallback in
`render_usage_block` now formats entry words as `` `<word>` ``
directly (matching the `parse.token.keyword.*` catalog renders);
the per-keyword catalog wrappers will collapse in the next step
(ADR-0024 §cleanup-pass §F).
`parse_command` and `parse_tokens` slim down:
- `parse_command(input)` no longer pre-lexes — the walker scans
source bytes directly.
- `parse_tokens` (internal-only `pub` for "future I3/I4 work")
is removed; its body folded into `parse_command`.
- `unknown_command_error` reads the walker registry directly.
Touched modules also drop their `crate::dsl::lexer::lex` and
`crate::dsl::usage` imports: `app.rs`, `input_render.rs`,
`completion.rs`.
Tests: 852 passing, 0 failing, 1 ignored (down from 860 because
the 8 `dsl::usage::tests::*` tests are gone with the module).
Replaces the lex()-driven `base_runs` span builder in
`input_render.rs` with `walker::highlight_runs`. The new
walker-side `dsl::walker::highlight` module returns per-byte
`HighlightClass` assignments for every token shape in the source:
- For commands the walker engages on, `WalkResult::per_byte_class`
is the authoritative source (keyword / identifier / number /
string / punct / flag).
- Trailing junk past a partial match — and inputs the walker
doesn't engage on at all (no registered entry word) — fall
through to a byte-shape scanner over `lex_helpers` so unknown
command words, stray punctuation, and unterminated strings
still highlight sensibly.
`Theme::highlight_class_color` is the walker-side analogue of
`token_color(&TokenKind)`; the renderer reads `walker::highlight_runs`
output and looks up colours through it. `token_color` and the
`lex()` pre-pass remain in place for now — the lexer module is
still consumed by usage rendering and completion until the
remaining Phase F steps land.
`HighlightClass`'s and `WalkResult::per_byte_class`'s
`#[allow(dead_code)]` annotations come off — they're now part of
the production highlight path.
Tests:
- 16 new tests under `dsl::walker::highlight` cover end-to-end
walks, byte-shape fallbacks (unknown commands, bare flags,
numbers, punctuation), UTF-8 codepoint advance, and trailing-
token handling after partial walks.
- Existing `input_render` tests pass unchanged.
- 860 total tests passing (727 lib + 133 integration), 1 ignored.
Clippy clean with `nursery` lints + `-D warnings`.
Delete the chumsky-side command_parser and its per-command
sub-parsers, error humanisation helpers, and keyword/punct/
ident/value-literal combinators. The unified-grammar walker
in `crate::dsl::walker` is now the sole parse path.
parse_tokens flow (post-Phase F):
1. lex(input) — still produces the Token stream that
completion / highlighting / echo-line consumers depend on.
2. try_walker_route(source) — walker handles every entry
keyword in REGISTRY (20 commands across 14 entry words).
3. unknown_command_error(source) — synthetic ParseError for
inputs whose first identifier-shape token isn't a
registered entry word. Wording mirrors the chumsky-side
"expected one of `add`, `change`, …, found `<word>`"
structural error the legacy top-level Choice produced.
Cargo.toml: chumsky dependency dropped (no remaining uses).
Cargo.lock regenerated; ~58 lines net reduction in the
dependency graph.
Scope intentionally deferred (separate follow-up):
- dsl/lexer.rs, dsl/keyword.rs, dsl/ident_slot.rs,
dsl/usage.rs::REGISTRY: still consumed by completion.rs,
input_render.rs, app.rs, theme.rs, db.rs, runtime.rs,
friendly/keys.rs. Removing these requires migrating each
consumer to the walker's per-byte-class output / grammar
REGISTRY / IdentSource enum. Substantial blast radius;
worth a dedicated session.
- parse.token.keyword.* catalog entries (40+): used by
usage.rs and parse-error rendering for the unmigrated
consumers above. Collapse follows after the consumer
migration.
Tests:
- All existing parser tests (`dsl::parser::tests`) ported in
place; they call `parse_command` which now flows through
the walker. 844 passed, 0 failed, 1 ignored — same count
as Phase E (no test additions, no regressions).
- cargo clippy --all-targets -- -D warnings clean.
- cargo build (release-like dev profile) succeeds.
Migrate `replay <path>` to the walker. Shape is
Choice(StringLit, BarePath); the StringLit branch handles the
quoted form (with the existing `''` escape), and BarePath
handles the unquoted form.
Per ADR-0024's path-bearing UX change (already shipped for
import / export in Phase A), bare `replay` paths terminate at
the first whitespace byte. Paths with spaces require the
quoted form. The legacy `try_parse_replay_with_bare_path`
source-slice helper in dsl/parser.rs is removed; the
chumsky-side replay branch in command_parser stays declared
but unreachable until Phase F sweeps the chumsky path.
Tests:
- 7 new walker-specific tests for replay: bare relative path,
bare absolute path, quoted with whitespace, quoted with
escaped quote, case-insensitive keyword, missing-path
error, empty-quoted-path parses to empty (runtime layer
rejects).
- Total: 844 passed, 0 failed, 1 ignored (was 838 / 1).
- cargo clippy --all-targets -- -D warnings clean.
Migrate the four data commands at four entry words: show
(show data / show table), insert, update, delete. Walker now
owns the entire command set introduced through ADR-0014.
Scope deviation from ADR-0024: full schema-aware value typing
via DynamicSubgrammar(column_value_list) is deferred. The
walker accepts any value at any position — matching the
existing chumsky parser's behaviour, where per-column type
checks happen at bind time. The DynamicSubgrammar Node
variant and WalkContext schema fields stay declared so the
infrastructure is in place when the schema cache plumbs
through parse_command (a future refinement). All existing
tests pass on the new shape.
Walker extensions:
- StringLit terminal — wired to the consume_string_literal
helper that mirrors the legacy lexer's `''` escape handling.
MatchedItem text carries the unescaped payload; span covers
the surrounding quotes.
- Bridge: Incomplete error wording now appends `, found end
of input` (matching the chumsky-side structural error
contract that `structural_error_for_show_data_without_arg`
asserts on).
Grammar:
- src/dsl/grammar/data.rs: SHOW (Choice of show_data /
show_table), INSERT (three forms folded into a single shape
via a Choice ordered to disambiguate Form B's `values`
keyword from Forms A/C's `(`-prefixed content; the inner
paren list is a Choice(VALUE_LITERAL, Ident{Columns}) with
VALUE_LITERAL ordered first so `true`/`false`/`null` match
their Word branch rather than the broader identifier catch-
all), UPDATE (assignments + filter), DELETE (filter).
- VALUE_LITERAL = Choice(Word("null"), Word("true"),
Word("false"), NumberLit, StringLit) — matches the chumsky
`value_literal()`.
- WHERE_CLAUSE / FILTER_CLAUSE shared between update and
delete.
- AST builders walk MatchedPath items in order, using role
tags (`update_set_column`, `filter_column`,
`insert_first_item`) to discriminate column references
belonging to different shapes within the same command.
Tests:
- 13 new walker-specific tests covering all data forms:
show data / show table, insert with each of three forms,
insert with negative numbers, update with single + multiple
assignments + where, update with --all-rows, delete with
where, delete with --all-rows, update/delete without filter
errors, replay still routes via chumsky.
- Total: 838 passed, 0 failed, 1 ignored (was 825 / 1).
- cargo clippy --all-targets -- -D warnings clean.
Migrate `create table <Name> [with pk [<col>:<type>[, ...]]]`
to the walker. Exercises Repeated{separator: Some(Punct(','))}
for the first time — the with-pk column-spec list.
Walker behaviour changes:
- Optional now backtracks on partial-match failure (Incomplete
or Failed-Mismatch from a Seq mid-shape). Path / per-byte
state rolls back to before the partial attempt; the inner's
expected-set propagates as `skipped` so callers see "what
would have completed it". Matches chumsky's `or_not`
semantics. ValidationFailed (content errors) does NOT
backtrack — the user means to fix those.
- Bridge: ValidationFailed errors now classify as
`at_eof = true`, mirroring the chumsky-side custom-error
convention. This is what lets `create table Customers`
classify as IncompleteAtEof rather than DefiniteErrorAt
(the user can still continue typing `with pk …`).
Grammar:
- src/dsl/grammar/ddl.rs gains CREATE: shape is
Seq(Word("table"), Ident{NewName,table_name}, Optional(WITH_PK))
where WITH_PK = Seq(Word("with"), Word("pk"),
Optional(Repeated{COL_SPEC, separator: Punct(','), min:1})).
AST builder enforces `with pk needs at least one column`
with the existing parse.custom.create_table_needs_pk catalog
wording; `with pk` alone defaults to id:serial.
Tests:
- 6 new walker-specific tests for create_table: with-pk
default, named typed PK, compound PK, whitespace tolerance
around `:` and `,`, bare-create-table-errors-with-with-pk-
hint, case-insensitive keywords.
- Total: 825 passed, 0 failed, 1 ignored (was 819 / 1).
- cargo clippy --all-targets -- -D warnings clean.
Migrate the five DDL commands at four entry words: drop (drop
table / drop column / drop relationship), add (add column /
add 1:n relationship), rename (rename column), change (change
column). The walker route now owns these end-to-end; chumsky
declarations remain unreachable for these inputs but stay
until Phase F.
Walker extensions:
- New node kinds: NumberLit (with optional content validator)
and Literal(&str) (verbatim byte sequence with word-boundary
lookahead — used for the `1` in `add 1:n …` so it surfaces
as `\`1\`` in the expected-set, matching the existing
parse_error_pedagogy contract).
- Flag (--name) terminal — Phase A stubbed; now wired to the
walker driver with consume_flag() in lex_helpers.
- Repeated combinator with optional separator and `min` floor.
Used by referential clauses (0..2 `on <delete|update>` runs)
and change-column flags (0..N --force-conversion /
--dont-convert; AST builder enforces mutual exclusion).
- Optional now propagates its inner's expectations as a
`skipped` field on the Matched result. Seq accumulates these
across children so the next failure's expected-set surfaces
the full union — closes the keyword-completion regression
(`add column ` must offer `to`, `table`, plus the table-name
identifier slot).
- Expectation::Ident gained a `source: IdentSource` field; the
parser-side bridge maps Tables/Columns/Relationships/Types
to the IdentSlot::expected_label strings ("table name",
"column name", …) so the existing completion engine's
schema-cache lookup still resolves.
- Walker error wording now includes "after `<consumed>`,
expected …" framing — matches the chumsky-side test
contract for structural errors mid-shape.
- AST-builder validation errors now propagate as
WalkOutcome::ValidationFailed (not the generic "AST builder
failed" fallback), so `change column … --force-conversion
--dont-convert` and repeated `on delete` clauses surface
their friendly catalog wording verbatim.
Grammar additions:
- src/dsl/grammar/shared.rs: type-name validator (TYPE_VALIDATOR
uses Type::from_str via parse.custom.unknown_type catalog),
qualified_column sub-grammar, referential action keyword
(`cascade`/`restrict`/`set null`/`no action`), repeated
on-clauses.
- src/dsl/grammar/ddl.rs: drop/add/rename/change CommandNodes
with inline shapes (per-use-site `role` annotations let the
AST builder discriminate parent vs child columns, etc.).
The four entry words each have one CommandNode whose `shape`
is a Choice across sub-forms.
Tests:
- 14 new walker-specific tests covering all DDL forms (bare
drop table, drop column with optional connectives, drop
relationship by name and by endpoints, add column with type
validator, rename column, change column with each flag form
+ mutual-exclusion check, add 1:n relationship minimal /
full, repeated-clause-twice rejection).
- Total: 819 passed, 0 failed, 1 ignored (was 805 / 1).
- cargo clippy --all-targets -- -D warnings clean.
claude@clouddev1