# ADR-0022: Ambient typing assistance — colour, hint panel, completion (I3 + I4)

## Status

Accepted.

Subsumes the originally-planned I4 (syntax highlighting) and
I3 (tab completion) into a single coherent feature: ambient
typing assistance. Builds on ADR-0020 (the lexer + parser-over-tokens
that this ADR consumes) and ADR-0021 (per-command usage
templates, which this ADR promotes from on-submit-only to live
ambient feedback). Cross-references ADR-0001 (theme conventions),
ADR-0010 (worker-thread access for schema queries), ADR-0019
(catalog conventions for any new strings), and ADR-0009 (the
closed grammar surface that defines what each token class means).

## Context

ADR-0019, 0020, and 0021 closed the on-submit feedback gap: a
parse error now renders with a caret, a structural / custom
message, and a per-command usage template. Once the user
hits Enter, they get a thorough explanation.

What's missing is *during* typing. Today the input field
renders as plain foreground colour with one inverted character
at the cursor. There is no signal for any of:

- "is `crete` a keyword or a typo?";
- "what comes after `create table Customers`?";
- "is `Customers` actually a table that exists?";
- "did I open this string and forget to close it?";
- "does Tab do anything here?".

Three separate mechanisms are commonly built to fix this:
**syntax highlighting** (colour), **tab completion** (Tab
key), and a **hint / status surface** (in our case the
already-present hint panel, which has been empty most of the
time). Each is correct on its own but they answer the same
underlying question — *what does the user need to know mid-typing?*
— and planning them separately produces three loose pieces
that drift apart in voice, mechanism, and what they cover.

The framing this ADR adopts:

- Colour is the **silent always-on** layer.
- The hint panel is the **verbose always-visible** layer.
- Tab is the **accept-this-suggestion action**.

These layer cleanly, with each one taking the load the other
two cannot bear:

- Colour catches lexer-level garbage and signals "this
  token is in error" instantly, but cannot explain why.
- The hint panel always says what comes next in prose, but
  cannot tell the user "the third character you typed is
  wrong" — colour does that.
- Tab does nothing on its own; the hint panel tells the
  user when Tab would help.

The hint panel is the central design move: it converts the
"Tab opens a popup over your text" anti-pattern into "the
already-visible panel becomes the chooser when Tab activates
it", which keeps screen real estate stable and removes any
modal floating UI.

## Decision

### 1. Three mid-typing states

Every keystroke produces one of three classifications, derived
from `lex(input)` followed by `parse_tokens(...)`:

1. **Valid-so-far.** Lex produces zero `Error` tokens AND
   parse succeeds OR parse fails at end-of-input (more input
   would make it valid).
2. **Definite error.** Lex produces an `Error` token, OR
   parse fails at a position before end-of-input (a token
   appears where no production accepts it).
3. **Incomplete-but-plausible.** A subset of valid-so-far:
   parse fails at end-of-input. The user's input is on
   track but not done.

The three states drive what colour, hint, and tab all do.

### 2. Layered feedback channels

Four channels operate independently, layered:

| Channel              | Layer        | Always on | Driver              |
|----------------------|--------------|-----------|---------------------|
| Token-class colour   | Silent       | Yes       | Lexer               |
| Error overlay        | Silent       | When applies | Parse position    |
| Hint panel ambient   | Verbose      | Yes       | Parse expected-set + state |
| Hint panel completion| Verbose      | Tab-triggered | User action       |

Each channel knows its layer's job and stays in it.

### 3. Token-class colour (the original I4 scope)

`Theme` gains seven token-class colour fields:

```rust
pub struct Theme {
    // ... existing fields ...
    pub tok_keyword: Color,
    pub tok_identifier: Color,
    pub tok_number: Color,
    pub tok_string: Color,
    pub tok_punct: Color,
    pub tok_flag: Color,
    pub tok_error: Color,
}
```

Both `Theme::dark()` and `Theme::light()` populate all seven
with WCAG-AA-contrasting values. Where two classes can
plausibly share `theme.fg`, the field still exists so a
future palette refresh can distinguish them without code
churn.

`ui::render_input_panel` runs `lex(&app.input)` per render.
Each token contributes a styled `Span` using its class
colour; whitespace gaps are preserved as `theme.fg` spans.
The cursor reverse-styling is injected by splitting the span
that contains the cursor's byte position into
(before, under, after) sub-spans, marking `under` REVERSED
while preserving its colour.

Per-render lex is microsecond-cheap. No caching.

### 4. Error overlay (parse-error highlighting on the failing token)

Render-time parse, in addition to lex. Then:

- If parse succeeds, no overlay.
- If parse fails at end-of-input (incomplete-but-plausible),
  no overlay.
- If parse fails at a token position before end-of-input,
  the token at that position renders with `theme.tok_error`
  overlay (foreground only, no underline / reverse — see
  reasoning in §6 of the previous draft).
- Lex `Error` tokens always render with `theme.tok_error`
  regardless of parse outcome.

Subsequent tokens after the error position render in their
normal lex-class colours. Rationale: the user fixes one
thing at a time; cascading red across the rest of the input
is visually overwhelming and rarely informative.

### 5. Echo lines: same colour treatment for simple-mode echoes

`render_output_line` for `OutputKind::Echo + Mode::Simple`
peels the catalog-bound `running: ` prefix, lexes the rest,
and renders the tokens identically to the input panel.
Advanced-mode echoes stay plain. Lift the
`dsl::ECHO_PREFIX = "running: "` constant + a unit test
asserting `t!("dsl.running", input = "")` matches it (so a
translator changing the prefix breaks the test, not the
render).

### 6. Hint panel ambient mode (always visible)

The hint panel currently shows `panel.hint_empty` when
`app.hint` is `None`. This ADR repurposes it as the verbose
typing-assistance surface. Three sub-states, one per
mid-typing state:

- **Valid-so-far + complete** (parse succeeds, all tokens
  consumed): "submit with Enter" — short and unobtrusive.
- **Valid-so-far + incomplete-but-plausible** (parse fails at
  EOF): "expected: <oxford-joined expected set>". Pulls
  from chumsky's expected-set at the failure point. For
  multi-entry families ("`add` family expects `column` or
  `1`") this is one line of natural prose.
- **Definite error**: "<short description of error> — usage:
  <template>". The usage template is the same `parse.usage.*`
  catalog content rendered on submit (ADR-0021), now also
  surfaced live. Multi-entry families render the matching
  family member only, picked by the same `usage::matched_entry`
  logic. Keeps the panel one-or-two-lines-tall.

Empty input keeps the existing `panel.hint_empty` content —
the hint panel only takes over when the user starts typing.

The catalog gains a small set of templates for these states:

```yaml
hint:
  ambient_complete: "submit with Enter"
  ambient_expected: "expected: {expected}"
  ambient_error_with_usage: "{message} — usage: {usage}"
```

### 7. Hint panel completion mode (Tab-triggered)

When the user presses Tab on a slot with multiple candidates,
the hint panel switches into completion mode. Visually the
panel shows the candidate list with one highlighted; the
input field is unchanged.

Mechanics:

- **Single candidate** → Tab inserts it immediately (with a
  trailing space if the next token would expect one). No
  mode change.
- **Multiple candidates** → Tab opens completion mode. The
  hint panel shows the candidates, one per line (or
  comma-joined if they fit), with the first highlighted.
- **Zero candidates** → Tab is a no-op.

While in completion mode, the input event flow shifts:

| Key                 | Behaviour                                  |
|---------------------|--------------------------------------------|
| Tab / Down          | Move highlight to next candidate           |
| Shift+Tab / Up      | Move highlight to previous candidate       |
| Enter               | Accept highlighted candidate               |
| Esc                 | Close completion mode without accepting   |
| Letter / digit / `_`| Append to input AND narrow candidate list  |
| Backspace           | Delete from input AND widen candidate list (or close mode if empty) |
| Other (cursor moves, paste, etc.) | Close completion mode, then process key normally |

Completion mode is a sticky state on `App` (`completion: Option<CompletionState>`).
It survives until accepted, cancelled, or the input changes
in a way that no candidate matches anymore (close + return
to ambient).

### 8. Identifier slot taxonomy

Identifier completion needs to know *what kind of identifier*
fits at the cursor. The parser is the only thing that knows
this — every `ident()` call has a semantic role. We make
that role explicit by replacing `ident()` with a tagged
variant at each call site:

```rust
pub enum IdentSlot {
    /// A name the user is inventing — no completion candidates.
    /// Examples: new table name, new column name, new relationship
    /// alias.
    NewName,
    /// An existing table.
    TableName,
    /// An existing column in a specific table. The table is bound
    /// by an earlier-consumed token in the same command.
    ColumnIn(TableRef),
    /// An existing relationship.
    RelationshipName,
}

pub enum TableRef {
    /// The table identifier is the Nth `ident()` call in this
    /// command's parser combinator chain.
    Earlier(usize),
}
```

`ident_ctx(slot)` is a small wrapper around the existing
`ident()` combinator that records the slot type alongside
the parsed identifier in the parser's `extra` data. The AST
itself is unchanged — slot tags are render-time concerns,
not AST concerns.

Concretely, every `ident()` call in `dsl/parser.rs` is
audited and becomes one of:

```rust
ident_ctx(IdentSlot::NewName)            // create table <Name>
ident_ctx(IdentSlot::TableName)          // drop table <T>
ident_ctx(IdentSlot::ColumnIn(Earlier(0)))  // drop column <T>: <Col>
// etc.
```

A unit test asserts every parser combinator references
`ident_ctx`, not bare `ident()`, so future commands can't
forget to tag.

The taxonomy is intentionally minimal for v1. Future
extensions (e.g. relationship endpoints needing both
parent-table and child-table awareness) are amendments to
this enum.

### 9. Schema query plumbing

The completion engine needs current schema data to enumerate
candidates for `TableName`, `ColumnIn`, and `RelationshipName`
slots. Per ADR-0010, all database access goes through the
worker thread.

Decision: a single new request:

```rust
Request::ListNamesFor(IdentSlot) → NameList
pub struct NameList { pub names: Vec<String> }
```

The worker computes from cached metadata; reply is small
and cheap. The completion engine fires this on Tab when
the slot is identifier-typed; until the reply arrives the
hint panel shows "(loading…)". For a teaching tool with a
small schema, the reply is effectively instantaneous; we
do not pre-cache or invalidate-on-write.

If schema mutates between Tab and Enter (the user pressed
Tab, then a background event changed the schema, then
Enter), the worst case is the user accepts a stale name —
caught by the next parse on submit. Acceptable.

### 10. Tab on keyword slots

Keyword candidates come from the parser's expected-token-set
at the cursor. The expected-set is the same data that
already drives the structural error wording (ADR-0020 §9
hook). For a slot with one keyword candidate, Tab inserts
it directly; for multiple, Tab opens completion mode with
the keyword list.

Punctuation that the parser expects (`:`, `(`, `,`) is
*not* offered by Tab — punctuation is too short to benefit
from completion, and inserting it without a following
identifier is an awkward halfway state. Tab on a slot whose
expected-set is purely punctuation is a no-op; the hint
panel's ambient prose still names what's expected.

### 11. Cursor position interpretation

The completion engine needs to know what slot the cursor is
at. Two cases:

- **Cursor at end of input**: the slot is whatever comes
  next after the last consumed token. Easiest case.
- **Cursor inside the input**: split the input at the
  cursor, lex the prefix, parse the prefix, and treat the
  failure-at-EOF state as the cursor slot. The suffix (the
  text after the cursor) is ignored for completion purposes.

This means tab completion in the middle of a typed-out
command works the same way as at the end — the completion
engine pretends the input ends at the cursor.

### 12. Mode interaction (simple vs. advanced; persistent vs. one-shot)

Ambient typing assistance is a **simple-mode** feature.
Advanced mode (persistent or `:`-one-shot) renders input
plain, has no hint panel ambient content beyond
`panel.hint_empty`, and Tab is a no-op. Justification: the
DSL lexer + parser don't speak SQL; using them for SQL
input would mark almost everything as Identifier/Error and
mislead. Future SQL-subset ADR (Q4) decides whether to
extend this.

The mode-banner colour (red for advanced, blue for simple)
already gives the user a strong "you are in a different
mode" signal; the absence of typing assistance reinforces
it.

### 13. Performance posture

Per render: lex + parse on the current input. For typical
input sizes (~80 chars, ~10 tokens), this is well under
100 µs total — orders of magnitude under ratatui's frame
budget. Schema queries on Tab incur one worker round-trip
(~1 ms in practice); the hint panel shows `(loading…)`
between Tab and reply. No caching, no debouncing in v1.

If profiling later shows render-loop pressure, options are
(a) cache the parse result keyed on input-string identity,
(b) lex+parse only on input change, not every render. Both
are local optimisations; not part of this ADR.

### 14. Catalog additions

```yaml
hint:
  ambient_complete: "submit with Enter"
  ambient_expected: "expected: {expected}"
  ambient_error_with_usage: "{message} — usage: {usage}"
  completion_loading: "(loading suggestions…)"
  completion_none: "no completions for this position"
panel:
  hint_completion_title: "Completions ({count})"
  # The existing `panel.hint_empty` keeps its current text.
```

### 15. Snapshot test churn

Existing UI snapshots in `src/snapshots/` cover the input
and output panels at fg-only colour. The render changes
re-baseline those snapshots. New snapshots cover:

- Input field with one token of each lex class.
- Input field with a definite-error overlay.
- Input field with cursor mid-token.
- Hint panel in each ambient sub-state.
- Hint panel in completion mode with multiple candidates.
- Hint panel after Tab on a single candidate (visual confirm
  of inserted text + closed completion mode).

The snapshots are the regression net for "did we change the
visual output unexpectedly".

## Out of scope

Deliberately deferred to keep this ADR shippable as a single
feature:

1. **"Did you mean?" near-keyword detection** (typed
   `crete`, suggest `create`). Useful but a separate
   feature with its own UX (highlight the typo? offer
   replace?). Future ADR.
2. **Fuzzy matching in completion mode** (typed `Cmrs`,
   match `Customers`). Today completion mode narrows by
   prefix only.
3. **Inline ghost text** (the rest of the unique completion
   shown ahead of the cursor in dim colour). The hint panel
   already covers the same role; ghost text would be
   redundant and risks visual clutter.
4. **User-customisable keybindings.** Tab / arrows / Enter
   / Esc are hard-coded.
5. **Schema completion across projects.** Completion is
   scoped to the currently loaded project's schema.
6. **Live-error highlighting *between* tokens** (e.g. a
   wavy underline spanning two tokens that together form
   an invalid sequence). Single-token error overlay only.
7. **Multi-line input.** Out of this ADR; tracked
   separately as I1.
8. **Highlighting in past parse-error renderings** in
   output history. Echo lines are highlighted; the
   error-block content is not retroactively re-styled.
9. **SQL highlighting / completion in advanced mode.**
   Waits on Q4.
10. **Identifier completion that crosses table boundaries**
    (e.g. completing column names across all tables). v1
    requires a `ColumnIn(table)` binding from earlier in
    the command.

## Consequences

### Positive

- **Single coherent feature** for typing assistance
  instead of three loose pieces. Voice, mechanism, and
  scope agree across colour / hint / completion.
- **The hint panel is finally useful.** It earns its
  screen real estate with always-visible content.
- **Tab does the right thing in every position.** Single
  candidate: insert. Multiple: open chooser. Zero: no-op
  (and the hint panel says why).
- **No floating UI.** Completion lives in the existing
  hint panel; no popup that overlaps text.
- **Schema-aware from day one.** `IdentSlot` taxonomy is
  in the parser; the completion engine knows what to ask.
- **Live error feedback.** Definite errors light up the
  failing token mid-typing; the hint panel surfaces the
  usage template on the spot.
- **Error overlay does not require a Tab.** Passive
  feedback you cannot miss.

### Costs

- **Largest single ADR-driven change so far**: theme + ui
  + app event flow + parser combinator audit + worker
  request type + new App state for completion mode +
  catalog additions + snapshot rebaselining. Estimated
  1500-2500 lines across changes plus tests.
- **Render-time parse** is new work. Cheap absolutely but
  it is per-keystroke. Risk of perf regression on very
  large input strings (none today; the DSL is one-line
  commands).
- **Identifier-slot taxonomy** is a new concept the parser
  combinators must thread through every `ident()` call.
  One-time refactor.
- **Schema-query worker request** adds one new entry to
  the worker's request enum.
- **App state grows** for completion mode (Option<CompletionState>),
  with input-event routing changes when the state is
  active.
- **Snapshot tests churn.** ~6 new snapshots, several
  existing ones rebaselined.
- **Catalog grows** by a handful of `hint.*` entries.

### Neutral

- **Public parser API**: gains the `IdentSlot` enum and
  the per-slot tagging via `ident_ctx`. The
  `parse_command` signature is unchanged.
- **Theme dependency direction unchanged.**
- **Tab key was previously unused** in the input panel; no
  conflict.

## Implementation notes

### Order of operations

Implementation lands as a sequence of green-after-each commits:

1. **Theme colours (small, ~150 lines).** Add seven
   `tok_*` fields with light + dark values. Add
   `Theme::token_color(&TokenKind) -> Color` helper.
2. **`ui::lex_to_spans` + input panel rewrite (medium,
   ~200 lines + snapshot rebaseline).** Live token
   colouring on the input field; cursor splitting logic.
3. **Echo-line rewrite (small, ~80 lines).**
   Simple-mode echo lines highlighted; `dsl::ECHO_PREFIX`
   constant + test.
4. **Render-time parse + error overlay (medium, ~150
   lines).** Classify input into one of three states;
   overlay error styling on the failing token.
5. **Hint panel ambient (medium, ~250 lines).** Three
   sub-states; catalog additions; rendering. The hint
   panel begins to earn its space.
6. **Identifier-slot taxonomy + parser tagging (medium,
   ~300 lines + parser audit).** `IdentSlot` enum,
   `ident_ctx` wrapper, audit every `ident()` call,
   per-command unit test asserting tagging coverage.
7. **Schema query plumbing (medium, ~200 lines).** New
   worker request, App-side dispatch, async handling
   model.
8. **Completion mode + Tab/arrow/Enter/Esc bindings
   (medium-large, ~400 lines).** App state, event
   routing, hint-panel render variant, completion-list
   filtering as user types.
9. **Snapshot test pass + manual smoke (small).**

Each stage is a checkpoint commit. The commit messages name
the stage and the cumulative state.

### Things that interact subtly

- **Cursor splitting at multi-byte UTF-8 boundaries.**
  The current input renderer handles this; the new
  span-based renderer must keep it. Walk to next char
  boundary when splitting a token's span at the cursor.
- **Empty input.** Lex returns `vec![]`, parse returns
  Empty, hint panel falls back to `panel.hint_empty`.
- **Input mid-token at cursor.** The completion engine
  treats the token-prefix-up-to-cursor as the typed
  prefix; candidate filtering matches against that
  prefix.
- **Mode transitions during completion mode.** If the
  user toggles persistent advanced mode (or types `:`)
  while completion mode is active, completion mode closes
  immediately.
- **Project switch / load while completion mode is
  active.** Same: close completion mode before any project
  state change is applied.
- **Schema cache after a DDL command.** The first
  ListNamesFor request after a schema-mutating command
  fetches fresh data; no explicit cache-invalidation
  mechanism in v1 because the worker re-reads metadata on
  each call.
- **Completion of a partial token.** Typing `Cust`, then
  Tab, where one candidate matches: the engine replaces
  `Cust` with `Customers` (not appends). The replace span
  is the partial-identifier token's span.
- **Tab at end of complete-and-valid input.** The
  expected-set may be empty or contain only `end of
  input`; Tab is a no-op (the hint panel says "submit
  with Enter").
- **`messages` setting.** Verbosity governs engine-error
  rendering (ADR-0019); ambient hint content is unaffected
  by it. Hint always shows the same level of detail.
- **Render performance.** Lex + parse + classify per render
  for typical input is microseconds; profile if bigger
  inputs land later (Query DSL, multi-line).