rdbms-playground/docs/adr/0022-ambient-typing-assistance.md

ADR-0022: Ambient typing assistance — colour, hint panel, completion (I3 + I4)

Status

Accepted.

Subsumes the originally-planned I4 (syntax highlighting) and I3 (tab completion) into a single coherent feature: ambient typing assistance. Builds on ADR-0020 (the lexer + parser-over-tokens that this ADR consumes) and ADR-0021 (per-command usage templates, which this ADR promotes from on-submit-only to live ambient feedback). Cross-references ADR-0001 (theme conventions), ADR-0010 (worker-thread access for schema queries), ADR-0019 (catalog conventions for any new strings), and ADR-0009 (the closed grammar surface that defines what each token class means).

Context

ADR-0019, 0020, and 0021 closed the on-submit feedback gap: a parse error now renders with a caret, a structural / custom message, and a per-command usage template. Once the user hits Enter, they get a thorough explanation.

What's missing is during typing. Today the input field renders as plain foreground colour with one inverted character at the cursor. There is no signal for any of:

• "is crete a keyword or a typo?";
• "what comes after create table Customers?";
• "is Customers actually a table that exists?";
• "did I open this string and forget to close it?";
• "does Tab do anything here?".

Three separate mechanisms are commonly built to fix this: syntax highlighting (colour), tab completion (Tab key), and a hint / status surface (in our case the already-present hint panel, which has been empty most of the time). Each is correct on its own but they answer the same underlying question — what does the user need to know mid-typing? — and planning them separately produces three loose pieces that drift apart in voice, mechanism, and what they cover.

The framing this ADR adopts:

• Colour is the silent always-on layer.
• The hint panel is the verbose always-visible layer.
• Tab is the accept-this-suggestion action.

These layer cleanly, with each one taking the load the other two cannot bear:

• Colour catches lexer-level garbage and signals "this token is in error" instantly, but cannot explain why.
• The hint panel always says what comes next in prose, but cannot tell the user "the third character you typed is wrong" — colour does that.
• Tab does nothing on its own; the hint panel tells the user when Tab would help.

The hint panel is the central design move: it converts the "Tab opens a popup over your text" anti-pattern into "the already-visible panel becomes the chooser when Tab activates it", which keeps screen real estate stable and removes any modal floating UI.

Decision

1. Three mid-typing states

Every keystroke produces one of three classifications, derived from lex(input) followed by parse_tokens(...):

1. Valid-so-far. Lex produces zero Error tokens AND parse succeeds OR parse fails at end-of-input (more input would make it valid).
2. Definite error. Lex produces an Error token, OR parse fails at a position before end-of-input (a token appears where no production accepts it).
3. Incomplete-but-plausible. A subset of valid-so-far: parse fails at end-of-input. The user's input is on track but not done.

The three states drive what colour, hint, and tab all do.

2. Layered feedback channels

Four channels operate independently, layered:

Channel	Layer	Always on	Driver
Token-class colour	Silent	Yes	Lexer
Error overlay	Silent	When applies	Parse position
Hint panel ambient	Verbose	Yes	Parse expected-set + state
Hint panel completion	Verbose	Tab-triggered	User action

Each channel knows its layer's job and stays in it.

3. Token-class colour (the original I4 scope)

Theme gains seven token-class colour fields:

pub struct Theme {
    // ... existing fields ...
    pub tok_keyword: Color,
    pub tok_identifier: Color,
    pub tok_number: Color,
    pub tok_string: Color,
    pub tok_punct: Color,
    pub tok_flag: Color,
    pub tok_error: Color,
}

Both Theme::dark() and Theme::light() populate all seven with WCAG-AA-contrasting values. Where two classes can plausibly share theme.fg, the field still exists so a future palette refresh can distinguish them without code churn.

ui::render_input_panel runs lex(&app.input) per render. Each token contributes a styled Span using its class colour; whitespace gaps are preserved as theme.fg spans. The cursor reverse-styling is injected by splitting the span that contains the cursor's byte position into (before, under, after) sub-spans, marking under REVERSED while preserving its colour.

Per-render lex is microsecond-cheap. No caching.

4. Error overlay (parse-error highlighting on the failing token)

Render-time parse, in addition to lex. Then:

• If parse succeeds, no overlay.
• If parse fails at end-of-input (incomplete-but-plausible), no overlay.
• If parse fails at a token position before end-of-input, the token at that position renders with theme.tok_error overlay (foreground only, no underline / reverse — see reasoning in §6 of the previous draft).
• Lex Error tokens always render with theme.tok_error regardless of parse outcome.

Subsequent tokens after the error position render in their normal lex-class colours. Rationale: the user fixes one thing at a time; cascading red across the rest of the input is visually overwhelming and rarely informative.

5. Echo lines: same colour treatment for simple-mode echoes

render_output_line for OutputKind::Echo + Mode::Simple peels the catalog-bound running: prefix, lexes the rest, and renders the tokens identically to the input panel. Advanced-mode echoes stay plain. Lift the dsl::ECHO_PREFIX = "running: " constant + a unit test asserting t!("dsl.running", input = "") matches it (so a translator changing the prefix breaks the test, not the render).

6. Hint panel ambient mode (always visible)

The hint panel currently shows panel.hint_empty when app.hint is None. This ADR repurposes it as the verbose typing-assistance surface. Three sub-states, one per mid-typing state:

• Valid-so-far + complete (parse succeeds, all tokens consumed): "submit with Enter" — short and unobtrusive.
• Valid-so-far + incomplete-but-plausible (parse fails at EOF): "expected: ". Pulls from chumsky's expected-set at the failure point. For multi-entry families ("add family expects column or 1") this is one line of natural prose.
• Definite error: " — usage: ". The usage template is the same parse.usage.* catalog content rendered on submit (ADR-0021), now also surfaced live. Multi-entry families render the matching family member only, picked by the same usage::matched_entry logic. Keeps the panel one-or-two-lines-tall.

  Empty input keeps the existing panel.hint_empty content — the hint panel only takes over when the user starts typing.

  The catalog gains a small set of templates for these states:

  hint:
    ambient_complete: "submit with Enter"
    ambient_expected: "expected: {expected}"
    ambient_error_with_usage: "{message} — usage: {usage}"
  

  7. Hint panel completion mode (Tab-triggered)

  When the user presses Tab on a slot with multiple candidates, the hint panel switches into completion mode. Visually the panel shows the candidate list with one highlighted; the input field is unchanged.

  Mechanics:

  • Single candidate → Tab inserts it immediately (with a trailing space if the next token would expect one). No mode change.
  • Multiple candidates → Tab opens completion mode. The hint panel shows the candidates, one per line (or comma-joined if they fit), with the first highlighted.
  • Zero candidates → Tab is a no-op.

  While in completion mode, the input event flow shifts:

  Key	Behaviour
  Tab / Down	Move highlight to next candidate
  Shift+Tab / Up	Move highlight to previous candidate
  Enter	Accept highlighted candidate
  Esc	Close completion mode without accepting
  Letter / digit / _	Append to input AND narrow candidate list
  Backspace	Delete from input AND widen candidate list (or close mode if empty)
  Other (cursor moves, paste, etc.)	Close completion mode, then process key normally

  Completion mode is a sticky state on App (completion: Option<CompletionState>). It survives until accepted, cancelled, or the input changes in a way that no candidate matches anymore (close + return to ambient).

  8. Identifier slot taxonomy

  Identifier completion needs to know what kind of identifier fits at the cursor. The parser is the only thing that knows this — every ident() call has a semantic role. We make that role explicit by replacing ident() with a tagged variant at each call site:

  pub enum IdentSlot {
      /// A name the user is inventing — no completion candidates.
      /// Examples: new table name, new column name, new relationship
      /// alias.
      NewName,
      /// An existing table.
      TableName,
      /// An existing column in a specific table. The table is bound
      /// by an earlier-consumed token in the same command.
      ColumnIn(TableRef),
      /// An existing relationship.
      RelationshipName,
  }
  
  pub enum TableRef {
      /// The table identifier is the Nth `ident()` call in this
      /// command's parser combinator chain.
      Earlier(usize),
  }
  

  ident_ctx(slot) is a small wrapper around the existing ident() combinator that records the slot type alongside the parsed identifier in the parser's extra data. The AST itself is unchanged — slot tags are render-time concerns, not AST concerns.

  Concretely, every ident() call in dsl/parser.rs is audited and becomes one of:

  ident_ctx(IdentSlot::NewName)            // create table <Name>
  ident_ctx(IdentSlot::TableName)          // drop table <T>
  ident_ctx(IdentSlot::ColumnIn(Earlier(0)))  // drop column <T>: <Col>
  // etc.
  

  A unit test asserts every parser combinator references ident_ctx, not bare ident(), so future commands can't forget to tag.

  The taxonomy is intentionally minimal for v1. Future extensions (e.g. relationship endpoints needing both parent-table and child-table awareness) are amendments to this enum.

  9. Schema query plumbing

  The completion engine needs current schema data to enumerate candidates for TableName, ColumnIn, and RelationshipName slots. Per ADR-0010, all database access goes through the worker thread.

  Decision: a single new request:

  Request::ListNamesFor(IdentSlot) → NameList
  pub struct NameList { pub names: Vec<String> }
  

  The worker computes from cached metadata; reply is small and cheap. The completion engine fires this on Tab when the slot is identifier-typed; until the reply arrives the hint panel shows "(loading…)". For a teaching tool with a small schema, the reply is effectively instantaneous; we do not pre-cache or invalidate-on-write.

  If schema mutates between Tab and Enter (the user pressed Tab, then a background event changed the schema, then Enter), the worst case is the user accepts a stale name — caught by the next parse on submit. Acceptable.

  10. Tab on keyword slots

  Keyword candidates come from the parser's expected-token-set at the cursor. The expected-set is the same data that already drives the structural error wording (ADR-0020 §9 hook). For a slot with one keyword candidate, Tab inserts it directly; for multiple, Tab opens completion mode with the keyword list.

  Punctuation that the parser expects (:, (, ,) is not offered by Tab — punctuation is too short to benefit from completion, and inserting it without a following identifier is an awkward halfway state. Tab on a slot whose expected-set is purely punctuation is a no-op; the hint panel's ambient prose still names what's expected.

  11. Cursor position interpretation

  The completion engine needs to know what slot the cursor is at. Two cases:

  • Cursor at end of input: the slot is whatever comes next after the last consumed token. Easiest case.
  • Cursor inside the input: split the input at the cursor, lex the prefix, parse the prefix, and treat the failure-at-EOF state as the cursor slot. The suffix (the text after the cursor) is ignored for completion purposes.

  This means tab completion in the middle of a typed-out command works the same way as at the end — the completion engine pretends the input ends at the cursor.

  12. Mode interaction (simple vs. advanced; persistent vs. one-shot)

  Ambient typing assistance is a simple-mode feature. Advanced mode (persistent or :-one-shot) renders input plain, has no hint panel ambient content beyond panel.hint_empty, and Tab is a no-op. Justification: the DSL lexer + parser don't speak SQL; using them for SQL input would mark almost everything as Identifier/Error and mislead. Future SQL-subset ADR (Q4) decides whether to extend this.

  The mode-banner colour (red for advanced, blue for simple) already gives the user a strong "you are in a different mode" signal; the absence of typing assistance reinforces it.

  13. Performance posture

  Per render: lex + parse on the current input. For typical input sizes (~80 chars, ~10 tokens), this is well under 100 µs total — orders of magnitude under ratatui's frame budget. Schema queries on Tab incur one worker round-trip (~1 ms in practice); the hint panel shows (loading…) between Tab and reply. No caching, no debouncing in v1.

  If profiling later shows render-loop pressure, options are (a) cache the parse result keyed on input-string identity, (b) lex+parse only on input change, not every render. Both are local optimisations; not part of this ADR.

  14. Catalog additions

  hint:
    ambient_complete: "submit with Enter"
    ambient_expected: "expected: {expected}"
    ambient_error_with_usage: "{message} — usage: {usage}"
    completion_loading: "(loading suggestions…)"
    completion_none: "no completions for this position"
  panel:
    hint_completion_title: "Completions ({count})"
    # The existing `panel.hint_empty` keeps its current text.
  

  15. Snapshot test churn

  Existing UI snapshots in src/snapshots/ cover the input and output panels at fg-only colour. The render changes re-baseline those snapshots. New snapshots cover:

  • Input field with one token of each lex class.
  • Input field with a definite-error overlay.
  • Input field with cursor mid-token.
  • Hint panel in each ambient sub-state.
  • Hint panel in completion mode with multiple candidates.
  • Hint panel after Tab on a single candidate (visual confirm of inserted text + closed completion mode).

  The snapshots are the regression net for "did we change the visual output unexpectedly".

  Out of scope

  Deliberately deferred to keep this ADR shippable as a single feature:

  1. "Did you mean?" near-keyword detection (typed crete, suggest create). Useful but a separate feature with its own UX (highlight the typo? offer replace?). Future ADR.
  2. Fuzzy matching in completion mode (typed Cmrs, match Customers). Today completion mode narrows by prefix only.
  3. Inline ghost text (the rest of the unique completion shown ahead of the cursor in dim colour). The hint panel already covers the same role; ghost text would be redundant and risks visual clutter.
  4. User-customisable keybindings. Tab / arrows / Enter / Esc are hard-coded.
  5. Schema completion across projects. Completion is scoped to the currently loaded project's schema.
  6. Live-error highlighting between tokens (e.g. a wavy underline spanning two tokens that together form an invalid sequence). Single-token error overlay only.
  7. Multi-line input. Out of this ADR; tracked separately as I1.
  8. Highlighting in past parse-error renderings in output history. Echo lines are highlighted; the error-block content is not retroactively re-styled.
  9. SQL highlighting / completion in advanced mode. Waits on Q4.
  10. Identifier completion that crosses table boundaries (e.g. completing column names across all tables). v1 requires a ColumnIn(table) binding from earlier in the command.

  Consequences

  Positive

  • Single coherent feature for typing assistance instead of three loose pieces. Voice, mechanism, and scope agree across colour / hint / completion.
  • The hint panel is finally useful. It earns its screen real estate with always-visible content.
  • Tab does the right thing in every position. Single candidate: insert. Multiple: open chooser. Zero: no-op (and the hint panel says why).
  • No floating UI. Completion lives in the existing hint panel; no popup that overlaps text.
  • Schema-aware from day one. IdentSlot taxonomy is in the parser; the completion engine knows what to ask.
  • Live error feedback. Definite errors light up the failing token mid-typing; the hint panel surfaces the usage template on the spot.
  • Error overlay does not require a Tab. Passive feedback you cannot miss.

  Costs

  • Largest single ADR-driven change so far: theme + ui
    • app event flow + parser combinator audit + worker request type + new App state for completion mode + catalog additions + snapshot rebaselining. Estimated 1500-2500 lines across changes plus tests.
  • Render-time parse is new work. Cheap absolutely but it is per-keystroke. Risk of perf regression on very large input strings (none today; the DSL is one-line commands).
  • Identifier-slot taxonomy is a new concept the parser combinators must thread through every ident() call. One-time refactor.
  • Schema-query worker request adds one new entry to the worker's request enum.
  • App state grows for completion mode (Option), with input-event routing changes when the state is active.
  • Snapshot tests churn. ~6 new snapshots, several existing ones rebaselined.
  • Catalog grows by a handful of hint.* entries.

  Neutral

  • Public parser API: gains the IdentSlot enum and the per-slot tagging via ident_ctx. The parse_command signature is unchanged.
  • Theme dependency direction unchanged.
  • Tab key was previously unused in the input panel; no conflict.

  Implementation notes

  Order of operations

  Implementation lands as a sequence of green-after-each commits:

  1. Theme colours (small, ~150 lines). Add seven tok_* fields with light + dark values. Add Theme::token_color(&TokenKind) -> Color helper.
  2. ui::lex_to_spans + input panel rewrite (medium, ~200 lines + snapshot rebaseline). Live token colouring on the input field; cursor splitting logic.
  3. Echo-line rewrite (small, ~80 lines). Simple-mode echo lines highlighted; dsl::ECHO_PREFIX constant + test.
  4. Render-time parse + error overlay (medium, ~150 lines). Classify input into one of three states; overlay error styling on the failing token.
  5. Hint panel ambient (medium, ~250 lines). Three sub-states; catalog additions; rendering. The hint panel begins to earn its space.
  6. Identifier-slot taxonomy + parser tagging (medium, ~300 lines + parser audit). IdentSlot enum, ident_ctx wrapper, audit every ident() call, per-command unit test asserting tagging coverage.
  7. Schema query plumbing (medium, ~200 lines). New worker request, App-side dispatch, async handling model.
  8. Completion mode + Tab/arrow/Enter/Esc bindings (medium-large, ~400 lines). App state, event routing, hint-panel render variant, completion-list filtering as user types.
  9. Snapshot test pass + manual smoke (small).

  Each stage is a checkpoint commit. The commit messages name the stage and the cumulative state.

  Things that interact subtly

  • Cursor splitting at multi-byte UTF-8 boundaries. The current input renderer handles this; the new span-based renderer must keep it. Walk to next char boundary when splitting a token's span at the cursor.
  • Empty input. Lex returns vec![], parse returns Empty, hint panel falls back to panel.hint_empty.
  • Input mid-token at cursor. The completion engine treats the token-prefix-up-to-cursor as the typed prefix; candidate filtering matches against that prefix.
  • Mode transitions during completion mode. If the user toggles persistent advanced mode (or types :) while completion mode is active, completion mode closes immediately.
  • Project switch / load while completion mode is active. Same: close completion mode before any project state change is applied.
  • Schema cache after a DDL command. The first ListNamesFor request after a schema-mutating command fetches fresh data; no explicit cache-invalidation mechanism in v1 because the worker re-reads metadata on each call.
  • Completion of a partial token. Typing Cust, then Tab, where one candidate matches: the engine replaces Cust with Customers (not appends). The replace span is the partial-identifier token's span.
  • Tab at end of complete-and-valid input. The expected-set may be empty or contain only end of input; Tab is a no-op (the hint panel says "submit with Enter").
  • messages setting. Verbosity governs engine-error rendering (ADR-0019); ambient hint content is unaffected by it. Hint always shows the same level of detail.
  • Render performance. Lex + parse + classify per render for typical input is microseconds; profile if bigger inputs land later (Query DSL, multi-line).