ADR-0022: ambient typing assistance (unifies I3 + I4)

Replaces the originally-planned separate ADRs for syntax highlighting (I4) and tab completion (I3) with a single unified design. The framing: colour, hint panel, and Tab are three answers to the same question — what does the user need to know mid-typing? — and planning them separately produces three loose pieces that drift apart. Three mid-typing states (valid-so-far / definite-error / incomplete-but-plausible) drive four layered channels: token-class colour and parse-error overlay (silent, always on), hint panel ambient and Tab-triggered completion mode (verbose, in the existing hint panel — no floating popups). Schema-aware from day one via an IdentSlot taxonomy in the parser (NewName / TableName / ColumnIn(TableRef::Earlier(N)) / RelationshipName); every existing ident() call gets audited and tagged. Completion candidates come from chumsky's expected-token-set for keyword slots and from a new worker request (ListNamesFor) for identifier slots. Implementation lands in 8 green-after-each commits: theme colours; input panel highlighting; echo line highlighting; render-time parse + error overlay; hint panel ambient; identifier-slot taxonomy + parser audit; schema query plumbing; completion mode + key bindings. Estimated 1500-2500 lines across the eight stages. Out of scope (deliberately): inline ghost text (could return as a "most-likely" affordance later — fish-shell style), fuzzy matching, punctuation completion, user-customisable keybindings, SQL highlighting in advanced mode (waits on Q4).
2026-05-10 15:51:22 +00:00
parent 11071ae164
commit f0632af8af
2 changed files with 566 additions and 0 deletions
@@ -0,0 +1,565 @@
 # ADR-0022: Ambient typing assistance — colour, hint panel, completion (I3 + I4)
 ## Status
 Accepted.
 Subsumes the originally-planned I4 (syntax highlighting) and
 I3 (tab completion) into a single coherent feature: ambient
 typing assistance. Builds on ADR-0020 (the lexer + parser-over-tokens
 that this ADR consumes) and ADR-0021 (per-command usage
 templates, which this ADR promotes from on-submit-only to live
 ambient feedback). Cross-references ADR-0001 (theme conventions),
 ADR-0010 (worker-thread access for schema queries), ADR-0019
 (catalog conventions for any new strings), and ADR-0009 (the
 closed grammar surface that defines what each token class means).
 ## Context
 ADR-0019, 0020, and 0021 closed the on-submit feedback gap: a
 parse error now renders with a caret, a structural / custom
 message, and a per-command usage template. Once the user
 hits Enter, they get a thorough explanation.
 What's missing is *during* typing. Today the input field
 renders as plain foreground colour with one inverted character
 at the cursor. There is no signal for any of:
 - "is `crete` a keyword or a typo?";
 - "what comes after `create table Customers`?";
 - "is `Customers` actually a table that exists?";
 - "did I open this string and forget to close it?";
 - "does Tab do anything here?".
 Three separate mechanisms are commonly built to fix this:
 **syntax highlighting** (colour), **tab completion** (Tab
 key), and a **hint / status surface** (in our case the
 already-present hint panel, which has been empty most of the
 time). Each is correct on its own but they answer the same
 underlying question — *what does the user need to know mid-typing?*
 — and planning them separately produces three loose pieces
 that drift apart in voice, mechanism, and what they cover.
 The framing this ADR adopts:
 - Colour is the **silent always-on** layer.
 - The hint panel is the **verbose always-visible** layer.
 - Tab is the **accept-this-suggestion action**.
 These layer cleanly, with each one taking the load the other
 two cannot bear:
 - Colour catches lexer-level garbage and signals "this
  token is in error" instantly, but cannot explain why.
 - The hint panel always says what comes next in prose, but
  cannot tell the user "the third character you typed is
  wrong" — colour does that.
 - Tab does nothing on its own; the hint panel tells the
  user when Tab would help.
 The hint panel is the central design move: it converts the
 "Tab opens a popup over your text" anti-pattern into "the
 already-visible panel becomes the chooser when Tab activates
 it", which keeps screen real estate stable and removes any
 modal floating UI.
 ## Decision
 ### 1. Three mid-typing states
 Every keystroke produces one of three classifications, derived
 from `lex(input)` followed by `parse_tokens(...)`:
 1. **Valid-so-far.** Lex produces zero `Error` tokens AND
   parse succeeds OR parse fails at end-of-input (more input
   would make it valid).
 2. **Definite error.** Lex produces an `Error` token, OR
   parse fails at a position before end-of-input (a token
   appears where no production accepts it).
 3. **Incomplete-but-plausible.** A subset of valid-so-far:
   parse fails at end-of-input. The user's input is on
   track but not done.
 The three states drive what colour, hint, and tab all do.
 ### 2. Layered feedback channels
 Four channels operate independently, layered:
 | Channel              | Layer        | Always on | Driver              |
 |----------------------|--------------|-----------|---------------------|
 | Token-class colour   | Silent       | Yes       | Lexer               |
 | Error overlay        | Silent       | When applies | Parse position    |
 | Hint panel ambient   | Verbose      | Yes       | Parse expected-set + state |
 | Hint panel completion| Verbose      | Tab-triggered | User action       |
 Each channel knows its layer's job and stays in it.
 ### 3. Token-class colour (the original I4 scope)
 `Theme` gains seven token-class colour fields:
 ```rust
 pub struct Theme {
    // ... existing fields ...
    pub tok_keyword: Color,
    pub tok_identifier: Color,
    pub tok_number: Color,
    pub tok_string: Color,
    pub tok_punct: Color,
    pub tok_flag: Color,
    pub tok_error: Color,
 }
 ```
 Both `Theme::dark()` and `Theme::light()` populate all seven
 with WCAG-AA-contrasting values. Where two classes can
 plausibly share `theme.fg`, the field still exists so a
 future palette refresh can distinguish them without code
 churn.
 `ui::render_input_panel` runs `lex(&app.input)` per render.
 Each token contributes a styled `Span` using its class
 colour; whitespace gaps are preserved as `theme.fg` spans.
 The cursor reverse-styling is injected by splitting the span
 that contains the cursor's byte position into
 (before, under, after) sub-spans, marking `under` REVERSED
 while preserving its colour.
 Per-render lex is microsecond-cheap. No caching.
 ### 4. Error overlay (parse-error highlighting on the failing token)
 Render-time parse, in addition to lex. Then:
 - If parse succeeds, no overlay.
 - If parse fails at end-of-input (incomplete-but-plausible),
  no overlay.
 - If parse fails at a token position before end-of-input,
  the token at that position renders with `theme.tok_error`
  overlay (foreground only, no underline / reverse — see
  reasoning in §6 of the previous draft).
 - Lex `Error` tokens always render with `theme.tok_error`
  regardless of parse outcome.
 Subsequent tokens after the error position render in their
 normal lex-class colours. Rationale: the user fixes one
 thing at a time; cascading red across the rest of the input
 is visually overwhelming and rarely informative.
 ### 5. Echo lines: same colour treatment for simple-mode echoes
 `render_output_line` for `OutputKind::Echo + Mode::Simple`
 peels the catalog-bound `running: ` prefix, lexes the rest,
 and renders the tokens identically to the input panel.
 Advanced-mode echoes stay plain. Lift the
 `dsl::ECHO_PREFIX = "running: "` constant + a unit test
 asserting `t!("dsl.running", input = "")` matches it (so a
 translator changing the prefix breaks the test, not the
 render).
 ### 6. Hint panel ambient mode (always visible)
 The hint panel currently shows `panel.hint_empty` when
 `app.hint` is `None`. This ADR repurposes it as the verbose
 typing-assistance surface. Three sub-states, one per
 mid-typing state:
 - **Valid-so-far + complete** (parse succeeds, all tokens
  consumed): "submit with Enter" — short and unobtrusive.
 - **Valid-so-far + incomplete-but-plausible** (parse fails at
  EOF): "expected: <oxford-joined expected set>". Pulls
  from chumsky's expected-set at the failure point. For
  multi-entry families ("`add` family expects `column` or
  `1`") this is one line of natural prose.
 - **Definite error**: "<short description of error> — usage:
  <template>". The usage template is the same `parse.usage.*`
  catalog content rendered on submit (ADR-0021), now also
  surfaced live. Multi-entry families render the matching
  family member only, picked by the same `usage::matched_entry`
  logic. Keeps the panel one-or-two-lines-tall.
 Empty input keeps the existing `panel.hint_empty` content —
 the hint panel only takes over when the user starts typing.
 The catalog gains a small set of templates for these states:
 ```yaml
 hint:
  ambient_complete: "submit with Enter"
  ambient_expected: "expected: {expected}"
  ambient_error_with_usage: "{message} — usage: {usage}"
 ```
 ### 7. Hint panel completion mode (Tab-triggered)
 When the user presses Tab on a slot with multiple candidates,
 the hint panel switches into completion mode. Visually the
 panel shows the candidate list with one highlighted; the
 input field is unchanged.
 Mechanics:
 - **Single candidate** → Tab inserts it immediately (with a
  trailing space if the next token would expect one). No
  mode change.
 - **Multiple candidates** → Tab opens completion mode. The
  hint panel shows the candidates, one per line (or
  comma-joined if they fit), with the first highlighted.
 - **Zero candidates** → Tab is a no-op.
 While in completion mode, the input event flow shifts:
 | Key                 | Behaviour                                  |
 |---------------------|--------------------------------------------|
 | Tab / Down          | Move highlight to next candidate           |
 | Shift+Tab / Up      | Move highlight to previous candidate       |
 | Enter               | Accept highlighted candidate               |
 | Esc                 | Close completion mode without accepting   |
 | Letter / digit / `_`| Append to input AND narrow candidate list  |
 | Backspace           | Delete from input AND widen candidate list (or close mode if empty) |
 | Other (cursor moves, paste, etc.) | Close completion mode, then process key normally |
 Completion mode is a sticky state on `App` (`completion: Option<CompletionState>`).
 It survives until accepted, cancelled, or the input changes
 in a way that no candidate matches anymore (close + return
 to ambient).
 ### 8. Identifier slot taxonomy
 Identifier completion needs to know *what kind of identifier*
 fits at the cursor. The parser is the only thing that knows
 this — every `ident()` call has a semantic role. We make
 that role explicit by replacing `ident()` with a tagged
 variant at each call site:
 ```rust
 pub enum IdentSlot {
    /// A name the user is inventing — no completion candidates.
    /// Examples: new table name, new column name, new relationship
    /// alias.
    NewName,
    /// An existing table.
    TableName,
    /// An existing column in a specific table. The table is bound
    /// by an earlier-consumed token in the same command.
    ColumnIn(TableRef),
    /// An existing relationship.
    RelationshipName,
 }
 pub enum TableRef {
    /// The table identifier is the Nth `ident()` call in this
    /// command's parser combinator chain.
    Earlier(usize),
 }
 ```
 `ident_ctx(slot)` is a small wrapper around the existing
 `ident()` combinator that records the slot type alongside
 the parsed identifier in the parser's `extra` data. The AST
 itself is unchanged — slot tags are render-time concerns,
 not AST concerns.
 Concretely, every `ident()` call in `dsl/parser.rs` is
 audited and becomes one of:
 ```rust
 ident_ctx(IdentSlot::NewName)            // create table <Name>
 ident_ctx(IdentSlot::TableName)          // drop table <T>
 ident_ctx(IdentSlot::ColumnIn(Earlier(0)))  // drop column <T>: <Col>
 // etc.
 ```
 A unit test asserts every parser combinator references
 `ident_ctx`, not bare `ident()`, so future commands can't
 forget to tag.
 The taxonomy is intentionally minimal for v1. Future
 extensions (e.g. relationship endpoints needing both
 parent-table and child-table awareness) are amendments to
 this enum.
 ### 9. Schema query plumbing
 The completion engine needs current schema data to enumerate
 candidates for `TableName`, `ColumnIn`, and `RelationshipName`
 slots. Per ADR-0010, all database access goes through the
 worker thread.
 Decision: a single new request:
 ```rust
 Request::ListNamesFor(IdentSlot) → NameList
 pub struct NameList { pub names: Vec<String> }
 ```
 The worker computes from cached metadata; reply is small
 and cheap. The completion engine fires this on Tab when
 the slot is identifier-typed; until the reply arrives the
 hint panel shows "(loading…)". For a teaching tool with a
 small schema, the reply is effectively instantaneous; we
 do not pre-cache or invalidate-on-write.
 If schema mutates between Tab and Enter (the user pressed
 Tab, then a background event changed the schema, then
 Enter), the worst case is the user accepts a stale name —
 caught by the next parse on submit. Acceptable.
 ### 10. Tab on keyword slots
 Keyword candidates come from the parser's expected-token-set
 at the cursor. The expected-set is the same data that
 already drives the structural error wording (ADR-0020 §9
 hook). For a slot with one keyword candidate, Tab inserts
 it directly; for multiple, Tab opens completion mode with
 the keyword list.
 Punctuation that the parser expects (`:`, `(`, `,`) is
 *not* offered by Tab — punctuation is too short to benefit
 from completion, and inserting it without a following
 identifier is an awkward halfway state. Tab on a slot whose
 expected-set is purely punctuation is a no-op; the hint
 panel's ambient prose still names what's expected.
 ### 11. Cursor position interpretation
 The completion engine needs to know what slot the cursor is
 at. Two cases:
 - **Cursor at end of input**: the slot is whatever comes
  next after the last consumed token. Easiest case.
 - **Cursor inside the input**: split the input at the
  cursor, lex the prefix, parse the prefix, and treat the
  failure-at-EOF state as the cursor slot. The suffix (the
  text after the cursor) is ignored for completion purposes.
 This means tab completion in the middle of a typed-out
 command works the same way as at the end — the completion
 engine pretends the input ends at the cursor.
 ### 12. Mode interaction (simple vs. advanced; persistent vs. one-shot)
 Ambient typing assistance is a **simple-mode** feature.
 Advanced mode (persistent or `:`-one-shot) renders input
 plain, has no hint panel ambient content beyond
 `panel.hint_empty`, and Tab is a no-op. Justification: the
 DSL lexer + parser don't speak SQL; using them for SQL
 input would mark almost everything as Identifier/Error and
 mislead. Future SQL-subset ADR (Q4) decides whether to
 extend this.
 The mode-banner colour (red for advanced, blue for simple)
 already gives the user a strong "you are in a different
 mode" signal; the absence of typing assistance reinforces
 it.
 ### 13. Performance posture
 Per render: lex + parse on the current input. For typical
 input sizes (~80 chars, ~10 tokens), this is well under
 100 µs total — orders of magnitude under ratatui's frame
 budget. Schema queries on Tab incur one worker round-trip
 (~1 ms in practice); the hint panel shows `(loading…)`
 between Tab and reply. No caching, no debouncing in v1.
 If profiling later shows render-loop pressure, options are
 (a) cache the parse result keyed on input-string identity,
 (b) lex+parse only on input change, not every render. Both
 are local optimisations; not part of this ADR.
 ### 14. Catalog additions
 ```yaml
 hint:
  ambient_complete: "submit with Enter"
  ambient_expected: "expected: {expected}"
  ambient_error_with_usage: "{message} — usage: {usage}"
  completion_loading: "(loading suggestions…)"
  completion_none: "no completions for this position"
 panel:
  hint_completion_title: "Completions ({count})"
  # The existing `panel.hint_empty` keeps its current text.
 ```
 ### 15. Snapshot test churn
 Existing UI snapshots in `src/snapshots/` cover the input
 and output panels at fg-only colour. The render changes
 re-baseline those snapshots. New snapshots cover:
 - Input field with one token of each lex class.
 - Input field with a definite-error overlay.
 - Input field with cursor mid-token.
 - Hint panel in each ambient sub-state.
 - Hint panel in completion mode with multiple candidates.
 - Hint panel after Tab on a single candidate (visual confirm
  of inserted text + closed completion mode).
 The snapshots are the regression net for "did we change the
 visual output unexpectedly".
 ## Out of scope
 Deliberately deferred to keep this ADR shippable as a single
 feature:
 1. **"Did you mean?" near-keyword detection** (typed
   `crete`, suggest `create`). Useful but a separate
   feature with its own UX (highlight the typo? offer
   replace?). Future ADR.
 2. **Fuzzy matching in completion mode** (typed `Cmrs`,
   match `Customers`). Today completion mode narrows by
   prefix only.
 3. **Inline ghost text** (the rest of the unique completion
   shown ahead of the cursor in dim colour). The hint panel
   already covers the same role; ghost text would be
   redundant and risks visual clutter.
 4. **User-customisable keybindings.** Tab / arrows / Enter
   / Esc are hard-coded.
 5. **Schema completion across projects.** Completion is
   scoped to the currently loaded project's schema.
 6. **Live-error highlighting *between* tokens** (e.g. a
   wavy underline spanning two tokens that together form
   an invalid sequence). Single-token error overlay only.
 7. **Multi-line input.** Out of this ADR; tracked
   separately as I1.
 8. **Highlighting in past parse-error renderings** in
   output history. Echo lines are highlighted; the
   error-block content is not retroactively re-styled.
 9. **SQL highlighting / completion in advanced mode.**
   Waits on Q4.
 10. **Identifier completion that crosses table boundaries**
    (e.g. completing column names across all tables). v1
    requires a `ColumnIn(table)` binding from earlier in
    the command.
 ## Consequences
 ### Positive
 - **Single coherent feature** for typing assistance
  instead of three loose pieces. Voice, mechanism, and
  scope agree across colour / hint / completion.
 - **The hint panel is finally useful.** It earns its
  screen real estate with always-visible content.
 - **Tab does the right thing in every position.** Single
  candidate: insert. Multiple: open chooser. Zero: no-op
  (and the hint panel says why).
 - **No floating UI.** Completion lives in the existing
  hint panel; no popup that overlaps text.
 - **Schema-aware from day one.** `IdentSlot` taxonomy is
  in the parser; the completion engine knows what to ask.
 - **Live error feedback.** Definite errors light up the
  failing token mid-typing; the hint panel surfaces the
  usage template on the spot.
 - **Error overlay does not require a Tab.** Passive
  feedback you cannot miss.
 ### Costs
 - **Largest single ADR-driven change so far**: theme + ui
  + app event flow + parser combinator audit + worker
  request type + new App state for completion mode +
  catalog additions + snapshot rebaselining. Estimated
  1500-2500 lines across changes plus tests.
 - **Render-time parse** is new work. Cheap absolutely but
  it is per-keystroke. Risk of perf regression on very
  large input strings (none today; the DSL is one-line
  commands).
 - **Identifier-slot taxonomy** is a new concept the parser
  combinators must thread through every `ident()` call.
  One-time refactor.
 - **Schema-query worker request** adds one new entry to
  the worker's request enum.
 - **App state grows** for completion mode (Option<CompletionState>),
  with input-event routing changes when the state is
  active.
 - **Snapshot tests churn.** ~6 new snapshots, several
  existing ones rebaselined.
 - **Catalog grows** by a handful of `hint.*` entries.
 ### Neutral
 - **Public parser API**: gains the `IdentSlot` enum and
  the per-slot tagging via `ident_ctx`. The
  `parse_command` signature is unchanged.
 - **Theme dependency direction unchanged.**
 - **Tab key was previously unused** in the input panel; no
  conflict.
 ## Implementation notes
 ### Order of operations
 Implementation lands as a sequence of green-after-each commits:
 1. **Theme colours (small, ~150 lines).** Add seven
   `tok_*` fields with light + dark values. Add
   `Theme::token_color(&TokenKind) -> Color` helper.
 2. **`ui::lex_to_spans` + input panel rewrite (medium,
   ~200 lines + snapshot rebaseline).** Live token
   colouring on the input field; cursor splitting logic.
 3. **Echo-line rewrite (small, ~80 lines).**
   Simple-mode echo lines highlighted; `dsl::ECHO_PREFIX`
   constant + test.
 4. **Render-time parse + error overlay (medium, ~150
   lines).** Classify input into one of three states;
   overlay error styling on the failing token.
 5. **Hint panel ambient (medium, ~250 lines).** Three
   sub-states; catalog additions; rendering. The hint
   panel begins to earn its space.
 6. **Identifier-slot taxonomy + parser tagging (medium,
   ~300 lines + parser audit).** `IdentSlot` enum,
   `ident_ctx` wrapper, audit every `ident()` call,
   per-command unit test asserting tagging coverage.
 7. **Schema query plumbing (medium, ~200 lines).** New
   worker request, App-side dispatch, async handling
   model.
 8. **Completion mode + Tab/arrow/Enter/Esc bindings
   (medium-large, ~400 lines).** App state, event
   routing, hint-panel render variant, completion-list
   filtering as user types.
 9. **Snapshot test pass + manual smoke (small).**
 Each stage is a checkpoint commit. The commit messages name
 the stage and the cumulative state.
 ### Things that interact subtly
 - **Cursor splitting at multi-byte UTF-8 boundaries.**
  The current input renderer handles this; the new
  span-based renderer must keep it. Walk to next char
  boundary when splitting a token's span at the cursor.
 - **Empty input.** Lex returns `vec![]`, parse returns
  Empty, hint panel falls back to `panel.hint_empty`.
 - **Input mid-token at cursor.** The completion engine
  treats the token-prefix-up-to-cursor as the typed
  prefix; candidate filtering matches against that
  prefix.
 - **Mode transitions during completion mode.** If the
  user toggles persistent advanced mode (or types `:`)
  while completion mode is active, completion mode closes
  immediately.
 - **Project switch / load while completion mode is
  active.** Same: close completion mode before any project
  state change is applied.
 - **Schema cache after a DDL command.** The first
  ListNamesFor request after a schema-mutating command
  fetches fresh data; no explicit cache-invalidation
  mechanism in v1 because the worker re-reads metadata on
  each call.
 - **Completion of a partial token.** Typing `Cust`, then
  Tab, where one candidate matches: the engine replaces
  `Cust` with `Customers` (not appends). The replace span
  is the partial-identifier token's span.
 - **Tab at end of complete-and-valid input.** The
  expected-set may be empty or contain only `end of
  input`; Tab is a no-op (the hint panel says "submit
  with Enter").
 - **`messages` setting.** Verbosity governs engine-error
  rendering (ADR-0019); ambient hint content is unaffected
  by it. Hint always shows the same level of detail.
 - **Render performance.** Lex + parse + classify per render
  for typical input is microseconds; profile if bigger
  inputs land later (Query DSL, multi-line).
@@ -27,3 +27,4 @@ This directory contains the project's ADRs, recorded per
 - [ADR-0019 — Friendly error layer (H1) and i18n message catalog](0019-friendly-error-layer-and-i18n.md)
 - [ADR-0020 — Tokenization layer for the DSL parser](0020-tokenization-layer-for-the-dsl-parser.md)
 - [ADR-0021 — Parser-as-source-of-truth for H1a (per-command usage in parse errors)](0021-parser-as-source-of-truth-for-h1a.md)
 - [ADR-0022 — Ambient typing assistance: colour, hint panel, completion (I3 + I4)](0022-ambient-typing-assistance.md)