rdbms-playground/docs/adr/0022-ambient-typing-assistance.md

ADR-0022: Ambient typing assistance — colour, hint panel, completion (I3 + I4)

Status

Accepted.

Subsumes the originally-planned I4 (syntax highlighting) and I3 (tab completion) into a single coherent feature: ambient typing assistance. Builds on ADR-0020 (the lexer + parser-over-tokens that this ADR consumes) and ADR-0021 (per-command usage templates, which this ADR promotes from on-submit-only to live ambient feedback). Cross-references ADR-0001 (theme conventions), ADR-0010 (worker-thread access for schema queries), ADR-0019 (catalog conventions for any new strings), and ADR-0009 (the closed grammar surface that defines what each token class means).

Context

ADR-0019, 0020, and 0021 closed the on-submit feedback gap: a parse error now renders with a caret, a structural / custom message, and a per-command usage template. Once the user hits Enter, they get a thorough explanation.

What's missing is during typing. Today the input field renders as plain foreground colour with one inverted character at the cursor. There is no signal for any of:

• "is crete a keyword or a typo?";
• "what comes after create table Customers?";
• "is Customers actually a table that exists?";
• "did I open this string and forget to close it?";
• "does Tab do anything here?".

Three separate mechanisms are commonly built to fix this: syntax highlighting (colour), tab completion (Tab key), and a hint / status surface (in our case the already-present hint panel, which has been empty most of the time). Each is correct on its own but they answer the same underlying question — what does the user need to know mid-typing? — and planning them separately produces three loose pieces that drift apart in voice, mechanism, and what they cover.

The framing this ADR adopts:

• Colour is the silent always-on layer.
• The hint panel is the verbose always-visible layer.
• Tab is the accept-this-suggestion action.

These layer cleanly, with each one taking the load the other two cannot bear:

• Colour catches lexer-level garbage and signals "this token is in error" instantly, but cannot explain why.
• The hint panel always says what comes next in prose, but cannot tell the user "the third character you typed is wrong" — colour does that.
• Tab does nothing on its own; the hint panel tells the user when Tab would help.

The hint panel is the central design move: it converts the "Tab opens a popup over your text" anti-pattern into "the already-visible panel becomes the chooser when Tab activates it", which keeps screen real estate stable and removes any modal floating UI.

Decision

1. Three mid-typing states

Every keystroke produces one of three classifications, derived from lex(input) followed by parse_tokens(...):

1. Valid-so-far. Lex produces zero Error tokens AND parse succeeds OR parse fails at end-of-input (more input would make it valid).
2. Definite error. Lex produces an Error token, OR parse fails at a position before end-of-input (a token appears where no production accepts it).
3. Incomplete-but-plausible. A subset of valid-so-far: parse fails at end-of-input. The user's input is on track but not done.

The three states drive what colour, hint, and tab all do.

2. Layered feedback channels

Four channels operate independently, layered:

Channel	Layer	Always on	Driver
Token-class colour	Silent	Yes	Lexer
Error overlay	Silent	When applies	Parse position
Hint panel ambient	Verbose	Yes	Parse expected-set + state
Hint panel completion	Verbose	Tab-triggered	User action

Each channel knows its layer's job and stays in it.

3. Token-class colour (the original I4 scope)

Theme gains seven token-class colour fields:

pub struct Theme {
    // ... existing fields ...
    pub tok_keyword: Color,
    pub tok_identifier: Color,
    pub tok_number: Color,
    pub tok_string: Color,
    pub tok_punct: Color,
    pub tok_flag: Color,
    pub tok_error: Color,
}

Both Theme::dark() and Theme::light() populate all seven with WCAG-AA-contrasting values. Where two classes can plausibly share theme.fg, the field still exists so a future palette refresh can distinguish them without code churn.

ui::render_input_panel runs lex(&app.input) per render. Each token contributes a styled Span using its class colour; whitespace gaps are preserved as theme.fg spans. The cursor reverse-styling is injected by splitting the span that contains the cursor's byte position into (before, under, after) sub-spans, marking under REVERSED while preserving its colour.

Per-render lex is microsecond-cheap. No caching.

4. Error overlay (parse-error highlighting on the failing token)

Render-time parse, in addition to lex. Then:

• If parse succeeds, no overlay.
• If parse fails at end-of-input (incomplete-but-plausible), no overlay.
• If parse fails at a token position before end-of-input, the token at that position renders with theme.tok_error overlay (foreground only, no underline / reverse — see reasoning in §6 of the previous draft).
• Lex Error tokens always render with theme.tok_error regardless of parse outcome.

Subsequent tokens after the error position render in their normal lex-class colours. Rationale: the user fixes one thing at a time; cascading red across the rest of the input is visually overwhelming and rarely informative.

5. Echo lines: same colour treatment for simple-mode echoes

render_output_line for OutputKind::Echo + Mode::Simple peels the catalog-bound running: prefix, lexes the rest, and renders the tokens identically to the input panel. Advanced-mode echoes stay plain. Lift the dsl::ECHO_PREFIX = "running: " constant + a unit test asserting t!("dsl.running", input = "") matches it (so a translator changing the prefix breaks the test, not the render).

6. Hint panel ambient mode (always visible)

The hint panel currently shows panel.hint_empty when app.hint is None. This ADR repurposes it as the verbose typing-assistance surface. Three sub-states, one per mid-typing state:

• Valid-so-far + complete (parse succeeds, all tokens consumed): "submit with Enter" — short and unobtrusive.
• Valid-so-far + incomplete-but-plausible (parse fails at EOF): "expected: ". Pulls from chumsky's expected-set at the failure point. For multi-entry families ("add family expects column or 1") this is one line of natural prose.
• Definite error: " — usage: ". The usage template is the same parse.usage.* catalog content rendered on submit (ADR-0021), now also surfaced live. Multi-entry families render the matching family member only, picked by the same usage::matched_entry logic. Keeps the panel one-or-two-lines-tall.

  Empty input keeps the existing panel.hint_empty content — the hint panel only takes over when the user starts typing.

  The catalog gains a small set of templates for these states:

  hint:
    ambient_complete: "submit with Enter"
    ambient_expected: "expected: {expected}"
    ambient_error_with_usage: "{message} — usage: {usage}"
  

  7. Hint panel completion mode (Tab-triggered)

  When the user presses Tab on a slot with multiple candidates, the hint panel switches into completion mode. Visually the panel shows the candidate list with one highlighted; the input field is unchanged.

  Mechanics:

  • Single candidate → Tab inserts it immediately (with a trailing space if the next token would expect one). No mode change.
  • Multiple candidates → Tab opens completion mode. The hint panel shows the candidates, one per line (or comma-joined if they fit), with the first highlighted.
  • Zero candidates → Tab is a no-op.

  While in completion mode, the input event flow shifts:

  Key	Behaviour
  Tab / Down	Move highlight to next candidate
  Shift+Tab / Up	Move highlight to previous candidate
  Enter	Accept highlighted candidate
  Esc	Close completion mode without accepting
  Letter / digit / _	Append to input AND narrow candidate list
  Backspace	Delete from input AND widen candidate list (or close mode if empty)
  Other (cursor moves, paste, etc.)	Close completion mode, then process key normally

  Completion mode is a sticky state on App (completion: Option<CompletionState>). It survives until accepted, cancelled, or the input changes in a way that no candidate matches anymore (close + return to ambient).

  8. Identifier slot taxonomy

  Identifier completion needs to know what kind of identifier fits at the cursor. The parser is the only thing that knows this — every ident() call has a semantic role. We make that role explicit by replacing ident() with a tagged variant at each call site:

  pub enum IdentSlot {
      /// A name the user is inventing — no completion candidates.
      /// Examples: new table name, new column name, new relationship
      /// alias.
      NewName,
      /// An existing table.
      TableName,
      /// An existing column in a specific table. The table is bound
      /// by an earlier-consumed token in the same command.
      ColumnIn(TableRef),
      /// An existing relationship.
      RelationshipName,
  }
  
  pub enum TableRef {
      /// The table identifier is the Nth `ident()` call in this
      /// command's parser combinator chain.
      Earlier(usize),
  }
  

  ident_ctx(slot) is a small wrapper around the existing ident() combinator that records the slot type alongside the parsed identifier in the parser's extra data. The AST itself is unchanged — slot tags are render-time concerns, not AST concerns.

  Concretely, every ident() call in dsl/parser.rs is audited and becomes one of:

  ident_ctx(IdentSlot::NewName)            // create table <Name>
  ident_ctx(IdentSlot::TableName)          // drop table <T>
  ident_ctx(IdentSlot::ColumnIn(Earlier(0)))  // drop column <T>: <Col>
  // etc.
  

  A unit test asserts every parser combinator references ident_ctx, not bare ident(), so future commands can't forget to tag.

  The taxonomy is intentionally minimal for v1. Future extensions (e.g. relationship endpoints needing both parent-table and child-table awareness) are amendments to this enum.

  9. Schema query plumbing

  The completion engine needs current schema data to enumerate candidates for TableName, ColumnIn, and RelationshipName slots. Per ADR-0010, all database access goes through the worker thread.

  Decision: a single new request:

  Request::ListNamesFor(IdentSlot) → NameList
  pub struct NameList { pub names: Vec<String> }
  

  The worker computes from cached metadata; reply is small and cheap. The completion engine fires this on Tab when the slot is identifier-typed; until the reply arrives the hint panel shows "(loading…)". For a teaching tool with a small schema, the reply is effectively instantaneous; we do not pre-cache or invalidate-on-write.

  If schema mutates between Tab and Enter (the user pressed Tab, then a background event changed the schema, then Enter), the worst case is the user accepts a stale name — caught by the next parse on submit. Acceptable.

  10. Tab on keyword slots

  Keyword candidates come from the parser's expected-token-set at the cursor. The expected-set is the same data that already drives the structural error wording (ADR-0020 §9 hook). For a slot with one keyword candidate, Tab inserts it directly; for multiple, Tab opens completion mode with the keyword list.

  Punctuation that the parser expects (:, (, ,) is not offered by Tab — punctuation is too short to benefit from completion, and inserting it without a following identifier is an awkward halfway state. Tab on a slot whose expected-set is purely punctuation is a no-op; the hint panel's ambient prose still names what's expected.

  11. Cursor position interpretation

  The completion engine needs to know what slot the cursor is at. Two cases:

  • Cursor at end of input: the slot is whatever comes next after the last consumed token. Easiest case.
  • Cursor inside the input: split the input at the cursor, lex the prefix, parse the prefix, and treat the failure-at-EOF state as the cursor slot. The suffix (the text after the cursor) is ignored for completion purposes.

  This means tab completion in the middle of a typed-out command works the same way as at the end — the completion engine pretends the input ends at the cursor.

  12. Mode interaction (simple vs. advanced; persistent vs. one-shot)

  Ambient typing assistance is a simple-mode feature. Advanced mode (persistent or :-one-shot) renders input plain, has no hint panel ambient content beyond panel.hint_empty, and Tab is a no-op. Justification: the DSL lexer + parser don't speak SQL; using them for SQL input would mark almost everything as Identifier/Error and mislead. Future SQL-subset ADR (Q4) decides whether to extend this.

  The mode-banner colour (red for advanced, blue for simple) already gives the user a strong "you are in a different mode" signal; the absence of typing assistance reinforces it.

  13. Performance posture

  Per render: lex + parse on the current input. For typical input sizes (~80 chars, ~10 tokens), this is well under 100 µs total — orders of magnitude under ratatui's frame budget. Schema queries on Tab incur one worker round-trip (~1 ms in practice); the hint panel shows (loading…) between Tab and reply. No caching, no debouncing in v1.

  If profiling later shows render-loop pressure, options are (a) cache the parse result keyed on input-string identity, (b) lex+parse only on input change, not every render. Both are local optimisations; not part of this ADR.

  Amendment 1 — Advanced-mode ambient assistance re-enabled (2026-05-21)

  This amendment supersedes §12's carve-out that ambient typing assistance is a simple-mode-only feature.

  The obsolete premise

  §12 disabled ambient assistance in advanced mode because, at the time, "the DSL lexer + parser don't speak SQL; using them for SQL input would mark almost everything as Identifier/Error and mislead." That premise no longer holds. ADR-0030 / ADR-0031 / ADR-0032 moved the SQL surface into the same unified, mode-aware walker grammar (ADR-0023 / ADR-0024). The walker now speaks SQL: in Mode::Advanced it parses SELECT (and, from ADR-0033, DML), highlights SQL keywords, resolves slot hints, and produces completion candidates — exactly the inputs ambient assistance needs.

  The bug this fixes

  Despite the walker being mode-aware, the UI never surfaced advanced-mode ambient assistance:

  • render_hint_panel hard-returned None for advanced mode (the stale §12 gate), so the hint panel showed only panel.hint_empty — no prose hints and no candidate preview for SQL.
  • The hint resolver (hint_resolution_at_input → expected_for_hint_snapshot) and ambient_hint never threaded the mode, so even the engine-level calls defaulted to Mode::Simple and gated a SQL statement as "this is SQL".

  The result: in advanced mode, hinting and completion-preview for SQL were completely dead, even for a bare SELECT.

  This gap survived Phase 2 because ADR-0032's cross-cut matrix rows for "Tab completion works for SQL keywords" / "Hint-panel prose appears at every SQL slot" were validated by engine-level tests (completion_probe_in_mode(…, Advanced), hint_mode_* called directly) — which prove the walker can produce SQL hints/candidates but never exercise the UI that suppressed them. This is the "free-for-free claim shipping without a real-app test" failure mode the project's process pins call out.

  What changed

  • ambient_hint_in_mode(input, cursor, memo, cache, mode) — the mode-aware ambient entry point. ambient_hint is now a thin wrapper that forwards Mode::Simple. Its sub-calls (input_diagnostics_in_mode, hint_resolution_at_input_in_mode, candidates_at_cursor_in_mode, the fallback parse_command_in_mode) all run in the supplied mode.
  • hint_resolution_at_input_in_mode + expected_for_hint_snapshot now set ctx.mode, so the hint walk respects the active mode.
  • render_hint_panel calls ambient_hint_in_mode with the effective mode for all modes (no more advanced-mode None).
  • One-shot : handling. In one-shot advanced mode the raw input carries the : sigil, which is not part of the grammar. The panel strips it (mirroring App::submit) before the ambient walk, so : sel hints select rather than the sigil.

  What still holds

  • Simple mode is unchanged. The simple-mode entry point keeps gating SQL as "this is SQL"; advanced assistance is opt-in via mode, never leaked into simple mode (regression-locked).
  • Syntax highlighting already ran with Mode::Advanced and is unaffected.
  • The validity indicator was already mode-aware (ADR-0032 §10.6); this amendment aligns the ambient hint panel with it.
  • §13 performance posture is unchanged — one walk per render, now in the active mode.

  Coverage

  App-level regression at the layer Phase 2 missed: src/ui.rs::advanced_mode_hint_panel_surfaces_sql_candidates (renders the panel in advanced mode and asserts the FROM-slot table candidate appears). Ambient-layer locks: src/input_render.rs::advanced_mode_ambient_offers_sql_from_slot_candidate and simple_mode_ambient_does_not_surface_sql_candidates.

  14. Catalog additions

  hint:
    ambient_complete: "submit with Enter"
    ambient_expected: "expected: {expected}"
    ambient_error_with_usage: "{message} — usage: {usage}"
    completion_loading: "(loading suggestions…)"
    completion_none: "no completions for this position"
  panel:
    hint_completion_title: "Completions ({count})"
    # The existing `panel.hint_empty` keeps its current text.
  

  15. Snapshot test churn

  Existing UI snapshots in src/snapshots/ cover the input and output panels at fg-only colour. The render changes re-baseline those snapshots. New snapshots cover:

  • Input field with one token of each lex class.
  • Input field with a definite-error overlay.
  • Input field with cursor mid-token.
  • Hint panel in each ambient sub-state.
  • Hint panel in completion mode with multiple candidates.
  • Hint panel after Tab on a single candidate (visual confirm of inserted text + closed completion mode).

  The snapshots are the regression net for "did we change the visual output unexpectedly".

  Amendment 2 — Candidate ordering: schema identifiers before keywords (2026-05-21)

  This amendment reverses the candidate-ordering call made in the handoff-14 ranker discussion (keywords before schema identifiers). That call was never recorded in an ADR — it lived only in tests/typing_surface/candidate_ordering.rs — so this amendment also gives the ordering a decision record.

  The obsolete premise

  Handoff-14 ordered command-part keywords before schema identifiers on the rationale that "grammar parts are read before the content that fills them," so add column to table T reads in order. That held while candidate lists were short. The SQL surface (ADR-0030/0031/0032/0033) made lists long — an expression position such as where Name or order by legitimately offers the column names plus the full expression-continuation keyword run (is not like between in and or, plus asc/desc in ORDER BY).

  The hint panel's candidate line is single-row and window-scrolled (render_candidate_line): when it overflows it centres on the selected item, or on item 0 when nothing is selected (the ambient, just-typed state). With keywords first, the schema identifiers sat at the tail and scrolled off behind the > marker — invisible until the user Tab-cycled to them.

  The decision

  Schema identifiers (table / column / relationship names) now sort before keywords in the candidate list. A name the user would otherwise have to look up is the highest-value completion — valuable even to experts, who come to know the keywords over time — so it must stay visible by default. Within each section the prior rules are unchanged: identifiers alphabetised; keywords in grammar-declaration order (to before table); then type names, composite literals, branching punct, flags. The existing colour split (tok_identifier teal vs tok_keyword purple, §colour) makes the section boundary legible once both are on screen.

  Related fix — Repeated trailing optionals

  This amendment shipped alongside a walk_repeated fix: a comma-separated list (Repeated) was discarding the last matched item's trailing-optional expectations at a clean item boundary, so order by Name offered no asc/desc, select Name no as, and create table … Code(text) no not/unique/default/ check. Those now surface (the separator , itself is deliberately not surfaced). This is what made identifier visibility pressing — the lists these positions produce are now both correct and long.

  Deferred — two-line hint box

  As hint lists grow, a two-line candidate box (more candidates visible without scrolling) is worth considering. Deferred for now on screen-space grounds; recorded so it is not lost.

  Coverage

  tests/typing_surface/candidate_ordering.rs rewritten to assert identifiers precede keywords (header invariant #2 inverted; the to-before-table keyword-order invariant #1 retained). completion::tests::identifiers_come_before_keywords_in_grammar_order and identifiers_precede_keywords_at_expression_position lock the ordering; order_by_after_sort_item_offers_direction, projection_after_item_offers_alias_keyword, and create_table_after_column_spec_offers_constraints lock the trailing-optional fix. ~20 typing-surface snapshots re-baselined.

  Amendment 3 — Ambient-hint fallback parses with the schema (2026-05-29)

  Amendment 1's "What changed" listed the fallback parse_command_in_mode among the sub-calls threaded through the active mode. That fallback (the bottom rung of ambient_hint's candidate-or-prose ladder, which produces the ambient_complete / ambient_expected / ambient_error_with_usage prose) was schemaless — it took the input and mode but not the SchemaCache, even though every earlier rung (input_diagnostics_in_mode, hint_resolution_at_input_in_mode, candidates_at_cursor_in_mode) already ran schema-aware. That was an oversight, not a decision: the ADR's whole intent is a schema- and mode-consistent hint surface.

  The gap surfaced as a user-reported bug (issue #2). Between two values of an insert … values (…) tuple, the type-blind (schemaless) grammar closes the tuple after one value, so the expected-token prose pointed at ) when the schema-aware grammar — knowing the remaining columns — expects ,. The same schemaless parse also accepted wrong-arity closed tuples as complete, so the prose read "submit with Enter" for an input the schema-aware parse (and the on-submit path) rejects.

  Change: the fallback rung now calls parse_command_with_schema_in_mode(input, cache, mode). The whole ladder is now schema-consistent. No new walk — the call site already held the cache; this swaps which parse function consumes it.

  What still holds: the friendly arity diagnostic (ADR-0033 §8.1) is checked at a higher rung, so where it fires (advanced-mode wrong-arity tuples) it still wins over this fallback — locked by advanced_mode_wrong_arity_insert_keeps_friendly_diagnostic_over_fallback. The §13 one-walk-per-render posture is unchanged.

  Coverage: ambient_hint_between_values_points_to_comma_not_close_paren, ambient_hint_after_last_value_points_to_close_paren, the no-masking guard above, and three re-baselined typing_surface snapshots (form_a_in_progress_one_value, form_b_too_few_values, form_c_wrong_count).

  Amendment 4 — Column types get a dedicated highlight class (2026-05-29)

  §3 introduced seven token-class colour fields and the matching HighlightClass enum. Column data-type keywords (int, serial, text, …) had no class of their own: the walker driver hardcoded HighlightClass::Identifier for every matched Node::Ident, so a type rendered identical to a table/column name (clause keywords were already distinct, in tok_keyword). A user-reported bug (issue #8) noted that in create table Orders (count int, id serial PRIMARY KEY) the identifiers and the type keywords were indistinguishably teal.

  Both terminal kinds already carried a highlight_override: Option<HighlightClass> field — Node::Ident and the Word struct alike — but both were dead: the driver destructured the Ident's to _, and walk_word hardcoded Keyword, neither ever consulting the field. This amendment wires both through.

  Change:

  1. New class. HighlightClass::Type (the eighth variant) and a matching eighth Theme field tok_type, populated in both dark() and light() with a tone deliberately distinct from both tok_keyword and tok_identifier (a pink / deep-magenta in the red-purple range — the only free slot in the existing palette). highlight_class_color maps the new variant.
  2. Overrides are now live. walk_ident emits override.unwrap_or(Identifier) and walk_word emits override.unwrap_or(Keyword) for the matched byte range. Every slot that leaves the field None is unchanged.
  3. Type slots opt in. The three IdentSource::Types slots — shared::TYPE_SLOT (add column / change column), the inline create-table column-type Ident (ddl.rs), and sql_create_table:: SQL_TYPE_NAME (advanced) — set highlight_override: Some(Type). In advanced mode, every single-word SQL type-name alias (float, varchar, integer, … — ADR-0035 §3) flows through SQL_TYPE_NAME and so is type-coloured for free.
  4. double precision too. The lone two-word alias (ADR-0035 §3) is matched as keyword tokens, not an IdentSource::Types Ident, so it cannot ride rule 3. Its double / precision grammar nodes use the new Word::type_keyword constructor (highlight_override: Some(Type)) so the spelling renders type-coloured like its single-word synonyms float / real.

  Pedagogy: a dedicated colour (over the lighter option of reusing tok_keyword) lets a learner tell "this is a type" from a clause keyword and from a name they invented — three distinct roles, three distinct colours.

  Coverage: dsl_type_keyword_classified_as_type, advanced_type_keywords_classified_as_type (the ticket's exact example), advanced_double_precision_classified_as_type, type_colour_is_distinct_from_keyword_and_identifier, and the extended theme mapping/contrast tests. Text snapshots are colour-blind (render_to_string strips style), so none churned.

  Out of scope

  Deliberately deferred to keep this ADR shippable as a single feature:

  1. "Did you mean?" near-keyword detection (typed crete, suggest create). Useful but a separate feature with its own UX (highlight the typo? offer replace?). Future ADR.
  2. Fuzzy matching in completion mode (typed Cmrs, match Customers). Today completion mode narrows by prefix only.
  3. Inline ghost text (the rest of the unique completion shown ahead of the cursor in dim colour). The hint panel already covers the same role; ghost text would be redundant and risks visual clutter.
  4. User-customisable keybindings. Tab / arrows / Enter / Esc are hard-coded.
  5. Schema completion across projects. Completion is scoped to the currently loaded project's schema.
  6. Live-error highlighting between tokens (e.g. a wavy underline spanning two tokens that together form an invalid sequence). Single-token error overlay only.
  7. Multi-line input. Out of this ADR; tracked separately as I1.
  8. Highlighting in past parse-error renderings in output history. Echo lines are highlighted; the error-block content is not retroactively re-styled.
  9. SQL highlighting / completion in advanced mode. Waits on Q4.
  10. Identifier completion that crosses table boundaries (e.g. completing column names across all tables). v1 requires a ColumnIn(table) binding from earlier in the command.

  Consequences

  Positive

  • Single coherent feature for typing assistance instead of three loose pieces. Voice, mechanism, and scope agree across colour / hint / completion.
  • The hint panel is finally useful. It earns its screen real estate with always-visible content.
  • Tab does the right thing in every position. Single candidate: insert. Multiple: open chooser. Zero: no-op (and the hint panel says why).
  • No floating UI. Completion lives in the existing hint panel; no popup that overlaps text.
  • Schema-aware from day one. IdentSlot taxonomy is in the parser; the completion engine knows what to ask.
  • Live error feedback. Definite errors light up the failing token mid-typing; the hint panel surfaces the usage template on the spot.
  • Error overlay does not require a Tab. Passive feedback you cannot miss.

  Costs

  • Largest single ADR-driven change so far: theme + ui
    • app event flow + parser combinator audit + worker request type + new App state for completion mode + catalog additions + snapshot rebaselining. Estimated 1500-2500 lines across changes plus tests.
  • Render-time parse is new work. Cheap absolutely but it is per-keystroke. Risk of perf regression on very large input strings (none today; the DSL is one-line commands).
  • Identifier-slot taxonomy is a new concept the parser combinators must thread through every ident() call. One-time refactor.
  • Schema-query worker request adds one new entry to the worker's request enum.
  • App state grows for completion mode (Option), with input-event routing changes when the state is active.
  • Snapshot tests churn. ~6 new snapshots, several existing ones rebaselined.
  • Catalog grows by a handful of hint.* entries.

  Neutral

  • Public parser API: gains the IdentSlot enum and the per-slot tagging via ident_ctx. The parse_command signature is unchanged.
  • Theme dependency direction unchanged.
  • Tab key was previously unused in the input panel; no conflict.

  Implementation notes

  Order of operations

  Implementation lands as a sequence of green-after-each commits:

  1. Theme colours (small, ~150 lines). Add seven tok_* fields with light + dark values. Add Theme::token_color(&TokenKind) -> Color helper.
  2. ui::lex_to_spans + input panel rewrite (medium, ~200 lines + snapshot rebaseline). Live token colouring on the input field; cursor splitting logic.
  3. Echo-line rewrite (small, ~80 lines). Simple-mode echo lines highlighted; dsl::ECHO_PREFIX constant + test.
  4. Render-time parse + error overlay (medium, ~150 lines). Classify input into one of three states; overlay error styling on the failing token.
  5. Hint panel ambient (medium, ~250 lines). Three sub-states; catalog additions; rendering. The hint panel begins to earn its space.
  6. Identifier-slot taxonomy + parser tagging (medium, ~300 lines + parser audit). IdentSlot enum, ident_ctx wrapper, audit every ident() call, per-command unit test asserting tagging coverage.
  7. Schema query plumbing (medium, ~200 lines). New worker request, App-side dispatch, async handling model.
  8. Completion mode + Tab/arrow/Enter/Esc bindings (medium-large, ~400 lines). App state, event routing, hint-panel render variant, completion-list filtering as user types.
  9. Snapshot test pass + manual smoke (small).

  Each stage is a checkpoint commit. The commit messages name the stage and the cumulative state.

  Things that interact subtly

  • Cursor splitting at multi-byte UTF-8 boundaries. The current input renderer handles this; the new span-based renderer must keep it. Walk to next char boundary when splitting a token's span at the cursor.
  • Empty input. Lex returns vec![], parse returns Empty, hint panel falls back to panel.hint_empty.
  • Input mid-token at cursor. The completion engine treats the token-prefix-up-to-cursor as the typed prefix; candidate filtering matches against that prefix.
  • Mode transitions during completion mode. If the user toggles persistent advanced mode (or types :) while completion mode is active, completion mode closes immediately.
  • Project switch / load while completion mode is active. Same: close completion mode before any project state change is applied.
  • Schema cache after a DDL command. The first ListNamesFor request after a schema-mutating command fetches fresh data; no explicit cache-invalidation mechanism in v1 because the worker re-reads metadata on each call.
  • Completion of a partial token. Typing Cust, then Tab, where one candidate matches: the engine replaces Cust with Customers (not appends). The replace span is the partial-identifier token's span.
  • Tab at end of complete-and-valid input. The expected-set may be empty or contain only end of input; Tab is a no-op (the hint panel says "submit with Enter").
  • messages setting. Verbosity governs engine-error rendering (ADR-0019); ambient hint content is unaffected by it. Hint always shows the same level of detail.
  • Render performance. Lex + parse + classify per render for typical input is microseconds; profile if bigger inputs land later (Query DSL, multi-line).