From f0632af8af3579dc3dcb767a080064d37bc099a6 Mon Sep 17 00:00:00 2001
From: "claude@clouddev1" <claude-clouddev1@oliversturm.ch>
Date: Sun, 10 May 2026 15:51:22 +0000
Subject: [PATCH] ADR-0022: ambient typing assistance (unifies I3 + I4)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Replaces the originally-planned separate ADRs for syntax
highlighting (I4) and tab completion (I3) with a single
unified design. The framing: colour, hint panel, and Tab
are three answers to the same question — what does the user
need to know mid-typing? — and planning them separately
produces three loose pieces that drift apart.

Three mid-typing states (valid-so-far / definite-error /
incomplete-but-plausible) drive four layered channels:
token-class colour and parse-error overlay (silent, always
on), hint panel ambient and Tab-triggered completion mode
(verbose, in the existing hint panel — no floating popups).

Schema-aware from day one via an IdentSlot taxonomy in the
parser (NewName / TableName / ColumnIn(TableRef::Earlier(N))
/ RelationshipName); every existing ident() call gets
audited and tagged. Completion candidates come from chumsky's
expected-token-set for keyword slots and from a new worker
request (ListNamesFor) for identifier slots.

Implementation lands in 8 green-after-each commits: theme
colours; input panel highlighting; echo line highlighting;
render-time parse + error overlay; hint panel ambient;
identifier-slot taxonomy + parser audit; schema query
plumbing; completion mode + key bindings. Estimated
1500-2500 lines across the eight stages.

Out of scope (deliberately): inline ghost text (could return
as a "most-likely" affordance later — fish-shell style),
fuzzy matching, punctuation completion, user-customisable
keybindings, SQL highlighting in advanced mode (waits on Q4).
---
 docs/adr/0022-ambient-typing-assistance.md | 565 +++++++++++++++++++++
 docs/adr/README.md                         |   1 +
 2 files changed, 566 insertions(+)
 create mode 100644 docs/adr/0022-ambient-typing-assistance.md

diff --git a/docs/adr/0022-ambient-typing-assistance.md b/docs/adr/0022-ambient-typing-assistance.md
new file mode 100644
index 0000000..03fba23
--- /dev/null
+++ b/docs/adr/0022-ambient-typing-assistance.md
@@ -0,0 +1,565 @@
+# ADR-0022: Ambient typing assistance — colour, hint panel, completion (I3 + I4)
+
+## Status
+
+Accepted.
+
+Subsumes the originally-planned I4 (syntax highlighting) and
+I3 (tab completion) into a single coherent feature: ambient
+typing assistance. Builds on ADR-0020 (the lexer + parser-over-tokens
+that this ADR consumes) and ADR-0021 (per-command usage
+templates, which this ADR promotes from on-submit-only to live
+ambient feedback). Cross-references ADR-0001 (theme conventions),
+ADR-0010 (worker-thread access for schema queries), ADR-0019
+(catalog conventions for any new strings), and ADR-0009 (the
+closed grammar surface that defines what each token class means).
+
+## Context
+
+ADR-0019, 0020, and 0021 closed the on-submit feedback gap: a
+parse error now renders with a caret, a structural / custom
+message, and a per-command usage template. Once the user
+hits Enter, they get a thorough explanation.
+
+What's missing is *during* typing. Today the input field
+renders as plain foreground colour with one inverted character
+at the cursor. There is no signal for any of:
+
+- "is `crete` a keyword or a typo?";
+- "what comes after `create table Customers`?";
+- "is `Customers` actually a table that exists?";
+- "did I open this string and forget to close it?";
+- "does Tab do anything here?".
+
+Three separate mechanisms are commonly built to fix this:
+**syntax highlighting** (colour), **tab completion** (Tab
+key), and a **hint / status surface** (in our case the
+already-present hint panel, which has been empty most of the
+time). Each is correct on its own but they answer the same
+underlying question — *what does the user need to know mid-typing?*
+— and planning them separately produces three loose pieces
+that drift apart in voice, mechanism, and what they cover.
+
+The framing this ADR adopts:
+
+- Colour is the **silent always-on** layer.
+- The hint panel is the **verbose always-visible** layer.
+- Tab is the **accept-this-suggestion action**.
+
+These layer cleanly, with each one taking the load the other
+two cannot bear:
+
+- Colour catches lexer-level garbage and signals "this
+  token is in error" instantly, but cannot explain why.
+- The hint panel always says what comes next in prose, but
+  cannot tell the user "the third character you typed is
+  wrong" — colour does that.
+- Tab does nothing on its own; the hint panel tells the
+  user when Tab would help.
+
+The hint panel is the central design move: it converts the
+"Tab opens a popup over your text" anti-pattern into "the
+already-visible panel becomes the chooser when Tab activates
+it", which keeps screen real estate stable and removes any
+modal floating UI.
+
+## Decision
+
+### 1. Three mid-typing states
+
+Every keystroke produces one of three classifications, derived
+from `lex(input)` followed by `parse_tokens(...)`:
+
+1. **Valid-so-far.** Lex produces zero `Error` tokens AND
+   parse succeeds OR parse fails at end-of-input (more input
+   would make it valid).
+2. **Definite error.** Lex produces an `Error` token, OR
+   parse fails at a position before end-of-input (a token
+   appears where no production accepts it).
+3. **Incomplete-but-plausible.** A subset of valid-so-far:
+   parse fails at end-of-input. The user's input is on
+   track but not done.
+
+The three states drive what colour, hint, and tab all do.
+
+### 2. Layered feedback channels
+
+Four channels operate independently, layered:
+
+| Channel              | Layer        | Always on | Driver              |
+|----------------------|--------------|-----------|---------------------|
+| Token-class colour   | Silent       | Yes       | Lexer               |
+| Error overlay        | Silent       | When applies | Parse position    |
+| Hint panel ambient   | Verbose      | Yes       | Parse expected-set + state |
+| Hint panel completion| Verbose      | Tab-triggered | User action       |
+
+Each channel knows its layer's job and stays in it.
+
+### 3. Token-class colour (the original I4 scope)
+
+`Theme` gains seven token-class colour fields:
+
+```rust
+pub struct Theme {
+    // ... existing fields ...
+    pub tok_keyword: Color,
+    pub tok_identifier: Color,
+    pub tok_number: Color,
+    pub tok_string: Color,
+    pub tok_punct: Color,
+    pub tok_flag: Color,
+    pub tok_error: Color,
+}
+```
+
+Both `Theme::dark()` and `Theme::light()` populate all seven
+with WCAG-AA-contrasting values. Where two classes can
+plausibly share `theme.fg`, the field still exists so a
+future palette refresh can distinguish them without code
+churn.
+
+`ui::render_input_panel` runs `lex(&app.input)` per render.
+Each token contributes a styled `Span` using its class
+colour; whitespace gaps are preserved as `theme.fg` spans.
+The cursor reverse-styling is injected by splitting the span
+that contains the cursor's byte position into
+(before, under, after) sub-spans, marking `under` REVERSED
+while preserving its colour.
+
+Per-render lex is microsecond-cheap. No caching.
+
+### 4. Error overlay (parse-error highlighting on the failing token)
+
+Render-time parse, in addition to lex. Then:
+
+- If parse succeeds, no overlay.
+- If parse fails at end-of-input (incomplete-but-plausible),
+  no overlay.
+- If parse fails at a token position before end-of-input,
+  the token at that position renders with `theme.tok_error`
+  overlay (foreground only, no underline / reverse — see
+  reasoning in §6 of the previous draft).
+- Lex `Error` tokens always render with `theme.tok_error`
+  regardless of parse outcome.
+
+Subsequent tokens after the error position render in their
+normal lex-class colours. Rationale: the user fixes one
+thing at a time; cascading red across the rest of the input
+is visually overwhelming and rarely informative.
+
+### 5. Echo lines: same colour treatment for simple-mode echoes
+
+`render_output_line` for `OutputKind::Echo + Mode::Simple`
+peels the catalog-bound `running: ` prefix, lexes the rest,
+and renders the tokens identically to the input panel.
+Advanced-mode echoes stay plain. Lift the
+`dsl::ECHO_PREFIX = "running: "` constant + a unit test
+asserting `t!("dsl.running", input = "")` matches it (so a
+translator changing the prefix breaks the test, not the
+render).
+
+### 6. Hint panel ambient mode (always visible)
+
+The hint panel currently shows `panel.hint_empty` when
+`app.hint` is `None`. This ADR repurposes it as the verbose
+typing-assistance surface. Three sub-states, one per
+mid-typing state:
+
+- **Valid-so-far + complete** (parse succeeds, all tokens
+  consumed): "submit with Enter" — short and unobtrusive.
+- **Valid-so-far + incomplete-but-plausible** (parse fails at
+  EOF): "expected: <oxford-joined expected set>". Pulls
+  from chumsky's expected-set at the failure point. For
+  multi-entry families ("`add` family expects `column` or
+  `1`") this is one line of natural prose.
+- **Definite error**: "<short description of error> — usage:
+  <template>". The usage template is the same `parse.usage.*`
+  catalog content rendered on submit (ADR-0021), now also
+  surfaced live. Multi-entry families render the matching
+  family member only, picked by the same `usage::matched_entry`
+  logic. Keeps the panel one-or-two-lines-tall.
+
+Empty input keeps the existing `panel.hint_empty` content —
+the hint panel only takes over when the user starts typing.
+
+The catalog gains a small set of templates for these states:
+
+```yaml
+hint:
+  ambient_complete: "submit with Enter"
+  ambient_expected: "expected: {expected}"
+  ambient_error_with_usage: "{message} — usage: {usage}"
+```
+
+### 7. Hint panel completion mode (Tab-triggered)
+
+When the user presses Tab on a slot with multiple candidates,
+the hint panel switches into completion mode. Visually the
+panel shows the candidate list with one highlighted; the
+input field is unchanged.
+
+Mechanics:
+
+- **Single candidate** → Tab inserts it immediately (with a
+  trailing space if the next token would expect one). No
+  mode change.
+- **Multiple candidates** → Tab opens completion mode. The
+  hint panel shows the candidates, one per line (or
+  comma-joined if they fit), with the first highlighted.
+- **Zero candidates** → Tab is a no-op.
+
+While in completion mode, the input event flow shifts:
+
+| Key                 | Behaviour                                  |
+|---------------------|--------------------------------------------|
+| Tab / Down          | Move highlight to next candidate           |
+| Shift+Tab / Up      | Move highlight to previous candidate       |
+| Enter               | Accept highlighted candidate               |
+| Esc                 | Close completion mode without accepting   |
+| Letter / digit / `_`| Append to input AND narrow candidate list  |
+| Backspace           | Delete from input AND widen candidate list (or close mode if empty) |
+| Other (cursor moves, paste, etc.) | Close completion mode, then process key normally |
+
+Completion mode is a sticky state on `App` (`completion: Option<CompletionState>`).
+It survives until accepted, cancelled, or the input changes
+in a way that no candidate matches anymore (close + return
+to ambient).
+
+### 8. Identifier slot taxonomy
+
+Identifier completion needs to know *what kind of identifier*
+fits at the cursor. The parser is the only thing that knows
+this — every `ident()` call has a semantic role. We make
+that role explicit by replacing `ident()` with a tagged
+variant at each call site:
+
+```rust
+pub enum IdentSlot {
+    /// A name the user is inventing — no completion candidates.
+    /// Examples: new table name, new column name, new relationship
+    /// alias.
+    NewName,
+    /// An existing table.
+    TableName,
+    /// An existing column in a specific table. The table is bound
+    /// by an earlier-consumed token in the same command.
+    ColumnIn(TableRef),
+    /// An existing relationship.
+    RelationshipName,
+}
+
+pub enum TableRef {
+    /// The table identifier is the Nth `ident()` call in this
+    /// command's parser combinator chain.
+    Earlier(usize),
+}
+```
+
+`ident_ctx(slot)` is a small wrapper around the existing
+`ident()` combinator that records the slot type alongside
+the parsed identifier in the parser's `extra` data. The AST
+itself is unchanged — slot tags are render-time concerns,
+not AST concerns.
+
+Concretely, every `ident()` call in `dsl/parser.rs` is
+audited and becomes one of:
+
+```rust
+ident_ctx(IdentSlot::NewName)            // create table <Name>
+ident_ctx(IdentSlot::TableName)          // drop table <T>
+ident_ctx(IdentSlot::ColumnIn(Earlier(0)))  // drop column <T>: <Col>
+// etc.
+```
+
+A unit test asserts every parser combinator references
+`ident_ctx`, not bare `ident()`, so future commands can't
+forget to tag.
+
+The taxonomy is intentionally minimal for v1. Future
+extensions (e.g. relationship endpoints needing both
+parent-table and child-table awareness) are amendments to
+this enum.
+
+### 9. Schema query plumbing
+
+The completion engine needs current schema data to enumerate
+candidates for `TableName`, `ColumnIn`, and `RelationshipName`
+slots. Per ADR-0010, all database access goes through the
+worker thread.
+
+Decision: a single new request:
+
+```rust
+Request::ListNamesFor(IdentSlot) → NameList
+pub struct NameList { pub names: Vec<String> }
+```
+
+The worker computes from cached metadata; reply is small
+and cheap. The completion engine fires this on Tab when
+the slot is identifier-typed; until the reply arrives the
+hint panel shows "(loading…)". For a teaching tool with a
+small schema, the reply is effectively instantaneous; we
+do not pre-cache or invalidate-on-write.
+
+If schema mutates between Tab and Enter (the user pressed
+Tab, then a background event changed the schema, then
+Enter), the worst case is the user accepts a stale name —
+caught by the next parse on submit. Acceptable.
+
+### 10. Tab on keyword slots
+
+Keyword candidates come from the parser's expected-token-set
+at the cursor. The expected-set is the same data that
+already drives the structural error wording (ADR-0020 §9
+hook). For a slot with one keyword candidate, Tab inserts
+it directly; for multiple, Tab opens completion mode with
+the keyword list.
+
+Punctuation that the parser expects (`:`, `(`, `,`) is
+*not* offered by Tab — punctuation is too short to benefit
+from completion, and inserting it without a following
+identifier is an awkward halfway state. Tab on a slot whose
+expected-set is purely punctuation is a no-op; the hint
+panel's ambient prose still names what's expected.
+
+### 11. Cursor position interpretation
+
+The completion engine needs to know what slot the cursor is
+at. Two cases:
+
+- **Cursor at end of input**: the slot is whatever comes
+  next after the last consumed token. Easiest case.
+- **Cursor inside the input**: split the input at the
+  cursor, lex the prefix, parse the prefix, and treat the
+  failure-at-EOF state as the cursor slot. The suffix (the
+  text after the cursor) is ignored for completion purposes.
+
+This means tab completion in the middle of a typed-out
+command works the same way as at the end — the completion
+engine pretends the input ends at the cursor.
+
+### 12. Mode interaction (simple vs. advanced; persistent vs. one-shot)
+
+Ambient typing assistance is a **simple-mode** feature.
+Advanced mode (persistent or `:`-one-shot) renders input
+plain, has no hint panel ambient content beyond
+`panel.hint_empty`, and Tab is a no-op. Justification: the
+DSL lexer + parser don't speak SQL; using them for SQL
+input would mark almost everything as Identifier/Error and
+mislead. Future SQL-subset ADR (Q4) decides whether to
+extend this.
+
+The mode-banner colour (red for advanced, blue for simple)
+already gives the user a strong "you are in a different
+mode" signal; the absence of typing assistance reinforces
+it.
+
+### 13. Performance posture
+
+Per render: lex + parse on the current input. For typical
+input sizes (~80 chars, ~10 tokens), this is well under
+100 µs total — orders of magnitude under ratatui's frame
+budget. Schema queries on Tab incur one worker round-trip
+(~1 ms in practice); the hint panel shows `(loading…)`
+between Tab and reply. No caching, no debouncing in v1.
+
+If profiling later shows render-loop pressure, options are
+(a) cache the parse result keyed on input-string identity,
+(b) lex+parse only on input change, not every render. Both
+are local optimisations; not part of this ADR.
+
+### 14. Catalog additions
+
+```yaml
+hint:
+  ambient_complete: "submit with Enter"
+  ambient_expected: "expected: {expected}"
+  ambient_error_with_usage: "{message} — usage: {usage}"
+  completion_loading: "(loading suggestions…)"
+  completion_none: "no completions for this position"
+panel:
+  hint_completion_title: "Completions ({count})"
+  # The existing `panel.hint_empty` keeps its current text.
+```
+
+### 15. Snapshot test churn
+
+Existing UI snapshots in `src/snapshots/` cover the input
+and output panels at fg-only colour. The render changes
+re-baseline those snapshots. New snapshots cover:
+
+- Input field with one token of each lex class.
+- Input field with a definite-error overlay.
+- Input field with cursor mid-token.
+- Hint panel in each ambient sub-state.
+- Hint panel in completion mode with multiple candidates.
+- Hint panel after Tab on a single candidate (visual confirm
+  of inserted text + closed completion mode).
+
+The snapshots are the regression net for "did we change the
+visual output unexpectedly".
+
+## Out of scope
+
+Deliberately deferred to keep this ADR shippable as a single
+feature:
+
+1. **"Did you mean?" near-keyword detection** (typed
+   `crete`, suggest `create`). Useful but a separate
+   feature with its own UX (highlight the typo? offer
+   replace?). Future ADR.
+2. **Fuzzy matching in completion mode** (typed `Cmrs`,
+   match `Customers`). Today completion mode narrows by
+   prefix only.
+3. **Inline ghost text** (the rest of the unique completion
+   shown ahead of the cursor in dim colour). The hint panel
+   already covers the same role; ghost text would be
+   redundant and risks visual clutter.
+4. **User-customisable keybindings.** Tab / arrows / Enter
+   / Esc are hard-coded.
+5. **Schema completion across projects.** Completion is
+   scoped to the currently loaded project's schema.
+6. **Live-error highlighting *between* tokens** (e.g. a
+   wavy underline spanning two tokens that together form
+   an invalid sequence). Single-token error overlay only.
+7. **Multi-line input.** Out of this ADR; tracked
+   separately as I1.
+8. **Highlighting in past parse-error renderings** in
+   output history. Echo lines are highlighted; the
+   error-block content is not retroactively re-styled.
+9. **SQL highlighting / completion in advanced mode.**
+   Waits on Q4.
+10. **Identifier completion that crosses table boundaries**
+    (e.g. completing column names across all tables). v1
+    requires a `ColumnIn(table)` binding from earlier in
+    the command.
+
+## Consequences
+
+### Positive
+
+- **Single coherent feature** for typing assistance
+  instead of three loose pieces. Voice, mechanism, and
+  scope agree across colour / hint / completion.
+- **The hint panel is finally useful.** It earns its
+  screen real estate with always-visible content.
+- **Tab does the right thing in every position.** Single
+  candidate: insert. Multiple: open chooser. Zero: no-op
+  (and the hint panel says why).
+- **No floating UI.** Completion lives in the existing
+  hint panel; no popup that overlaps text.
+- **Schema-aware from day one.** `IdentSlot` taxonomy is
+  in the parser; the completion engine knows what to ask.
+- **Live error feedback.** Definite errors light up the
+  failing token mid-typing; the hint panel surfaces the
+  usage template on the spot.
+- **Error overlay does not require a Tab.** Passive
+  feedback you cannot miss.
+
+### Costs
+
+- **Largest single ADR-driven change so far**: theme + ui
+  + app event flow + parser combinator audit + worker
+  request type + new App state for completion mode +
+  catalog additions + snapshot rebaselining. Estimated
+  1500-2500 lines across changes plus tests.
+- **Render-time parse** is new work. Cheap absolutely but
+  it is per-keystroke. Risk of perf regression on very
+  large input strings (none today; the DSL is one-line
+  commands).
+- **Identifier-slot taxonomy** is a new concept the parser
+  combinators must thread through every `ident()` call.
+  One-time refactor.
+- **Schema-query worker request** adds one new entry to
+  the worker's request enum.
+- **App state grows** for completion mode (Option<CompletionState>),
+  with input-event routing changes when the state is
+  active.
+- **Snapshot tests churn.** ~6 new snapshots, several
+  existing ones rebaselined.
+- **Catalog grows** by a handful of `hint.*` entries.
+
+### Neutral
+
+- **Public parser API**: gains the `IdentSlot` enum and
+  the per-slot tagging via `ident_ctx`. The
+  `parse_command` signature is unchanged.
+- **Theme dependency direction unchanged.**
+- **Tab key was previously unused** in the input panel; no
+  conflict.
+
+## Implementation notes
+
+### Order of operations
+
+Implementation lands as a sequence of green-after-each commits:
+
+1. **Theme colours (small, ~150 lines).** Add seven
+   `tok_*` fields with light + dark values. Add
+   `Theme::token_color(&TokenKind) -> Color` helper.
+2. **`ui::lex_to_spans` + input panel rewrite (medium,
+   ~200 lines + snapshot rebaseline).** Live token
+   colouring on the input field; cursor splitting logic.
+3. **Echo-line rewrite (small, ~80 lines).**
+   Simple-mode echo lines highlighted; `dsl::ECHO_PREFIX`
+   constant + test.
+4. **Render-time parse + error overlay (medium, ~150
+   lines).** Classify input into one of three states;
+   overlay error styling on the failing token.
+5. **Hint panel ambient (medium, ~250 lines).** Three
+   sub-states; catalog additions; rendering. The hint
+   panel begins to earn its space.
+6. **Identifier-slot taxonomy + parser tagging (medium,
+   ~300 lines + parser audit).** `IdentSlot` enum,
+   `ident_ctx` wrapper, audit every `ident()` call,
+   per-command unit test asserting tagging coverage.
+7. **Schema query plumbing (medium, ~200 lines).** New
+   worker request, App-side dispatch, async handling
+   model.
+8. **Completion mode + Tab/arrow/Enter/Esc bindings
+   (medium-large, ~400 lines).** App state, event
+   routing, hint-panel render variant, completion-list
+   filtering as user types.
+9. **Snapshot test pass + manual smoke (small).**
+
+Each stage is a checkpoint commit. The commit messages name
+the stage and the cumulative state.
+
+### Things that interact subtly
+
+- **Cursor splitting at multi-byte UTF-8 boundaries.**
+  The current input renderer handles this; the new
+  span-based renderer must keep it. Walk to next char
+  boundary when splitting a token's span at the cursor.
+- **Empty input.** Lex returns `vec![]`, parse returns
+  Empty, hint panel falls back to `panel.hint_empty`.
+- **Input mid-token at cursor.** The completion engine
+  treats the token-prefix-up-to-cursor as the typed
+  prefix; candidate filtering matches against that
+  prefix.
+- **Mode transitions during completion mode.** If the
+  user toggles persistent advanced mode (or types `:`)
+  while completion mode is active, completion mode closes
+  immediately.
+- **Project switch / load while completion mode is
+  active.** Same: close completion mode before any project
+  state change is applied.
+- **Schema cache after a DDL command.** The first
+  ListNamesFor request after a schema-mutating command
+  fetches fresh data; no explicit cache-invalidation
+  mechanism in v1 because the worker re-reads metadata on
+  each call.
+- **Completion of a partial token.** Typing `Cust`, then
+  Tab, where one candidate matches: the engine replaces
+  `Cust` with `Customers` (not appends). The replace span
+  is the partial-identifier token's span.
+- **Tab at end of complete-and-valid input.** The
+  expected-set may be empty or contain only `end of
+  input`; Tab is a no-op (the hint panel says "submit
+  with Enter").
+- **`messages` setting.** Verbosity governs engine-error
+  rendering (ADR-0019); ambient hint content is unaffected
+  by it. Hint always shows the same level of detail.
+- **Render performance.** Lex + parse + classify per render
+  for typical input is microseconds; profile if bigger
+  inputs land later (Query DSL, multi-line).
diff --git a/docs/adr/README.md b/docs/adr/README.md
index 5b03a68..e75082d 100644
--- a/docs/adr/README.md
+++ b/docs/adr/README.md
@@ -27,3 +27,4 @@ This directory contains the project's ADRs, recorded per
 - [ADR-0019 — Friendly error layer (H1) and i18n message catalog](0019-friendly-error-layer-and-i18n.md)
 - [ADR-0020 — Tokenization layer for the DSL parser](0020-tokenization-layer-for-the-dsl-parser.md)
 - [ADR-0021 — Parser-as-source-of-truth for H1a (per-command usage in parse errors)](0021-parser-as-source-of-truth-for-h1a.md)
+- [ADR-0022 — Ambient typing assistance: colour, hint panel, completion (I3 + I4)](0022-ambient-typing-assistance.md)