266b4c2ef4ad441164b72e1b6f166aaf9b8f283a
13 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 claude@clouddev1
|
266b4c2ef4 |
ADR-0024 Phase F (full) step 3: delete legacy parser modules
Removes the last consumers of `dsl::lexer`, `dsl::keyword`, and
`dsl::ident_slot`, then deletes the modules.
- `Theme::token_color(&TokenKind)` deleted along with its test;
`Theme::highlight_class_color(HighlightClass)` is the sole
highlight-colour mapper (the walker's `per_byte_class` feeds
it directly).
- `IdentSource` (`dsl::grammar`) absorbs the schema-list /
expected-label / round-trip semantics that previously lived
on `IdentSlot`. Adds `completes_from_schema`, `expected_label`,
and `from_expected_label` methods. The walker's
`Expectation::Ident { source }` and the schema-lookup request
on the database worker now share one enum.
- `SchemaCache::for_slot(IdentSlot)` → `for_source(IdentSource)`.
- `Database::list_names_for` and the `Request::ListNamesFor`
worker variant take `IdentSource`. Internal tables and column
/ relationship lookups dispatch on the same enum.
- `InvalidIdent.slot: IdentSlot` → `InvalidIdent.source: IdentSource`.
The `invalid_ident_at_cursor` rendering branch in
`input_render.rs::ambient_hint` updates accordingly.
- Completion's keyword filter (`Keyword::from_word`) becomes
"backticked items whose payload is all ASCII alphabetic" —
punct and digit literals still surface through their own
candidate sources (composite-literal, flag, schema-ident);
the alphabetic filter excludes them from the keyword bucket.
- `friendly::keys::tests::keyword_and_punct_have_complete_token_vocabulary`
is dropped. It cross-checked `Keyword::ALL` / `Punct::ALL`
against catalog entries; both enums are gone. The
`parse.token.keyword.*` / `parse.token.punct.*` catalog
entries themselves survive for one more commit (catalog
cleanup, ADR-0024 §cleanup-pass); the
`keys_validate_against_catalog` test still pins them.
- Modules deleted: `src/dsl/lexer.rs`, `src/dsl/keyword.rs`,
`src/dsl/ident_slot.rs`.
Tests: 806 passing, 0 failing, 1 ignored. The drop from 852
reflects the removed module-internal tests (~32 lexer, 7
keyword, 4 ident_slot, 1 theme token_color, 1 friendly keys
keyword/punct), and is the expected outcome.
Clippy clean with `nursery` lints + `-D warnings`.
|
||
|
claude@clouddev1
|
a41400e532 |
ADR-0024 Phase F (full) step 2: usage via CommandNode.usage_ids
Migrates parse-error usage-block rendering from the legacy `dsl::usage::matched_entry` (which scanned a `Vec<Token>` for the first matched Keyword) to walker-side lookup driven by each `CommandNode`'s `usage_ids` slice. `CommandNode.usage_id: Option<&'static str>` becomes `usage_ids: &'static [&'static str]`. Multi-form families (`drop`, `add`, `show`) carry every variant — `drop` lists table/column/relationship templates; `add` lists column / relationship; `show` lists data / table. The single-shape commands carry their single catalog key. App-lifecycle CommandNodes had pointed at non-existent `parse.usage.app.*` keys (never noticed because the field was unused); they now point at the real catalog entries (`parse.usage.quit`, `parse.usage.help`, …). New helpers in `dsl::grammar`: - `usage_keys_for_input(source) -> Option<(entry_word, usage_ids)>` resolves the first identifier-shape token to a CommandNode and returns its usage_ids list. Used by `app::render_usage_block` and `input_render::ambient_hint`. - `entry_words_alphabetised() -> Vec<&'static str>` replaces `dsl::usage::entry_keywords_alphabetised`. `dsl::usage` is deleted. The "available commands:" fallback in `render_usage_block` now formats entry words as `` `<word>` `` directly (matching the `parse.token.keyword.*` catalog renders); the per-keyword catalog wrappers will collapse in the next step (ADR-0024 §cleanup-pass §F). `parse_command` and `parse_tokens` slim down: - `parse_command(input)` no longer pre-lexes — the walker scans source bytes directly. - `parse_tokens` (internal-only `pub` for "future I3/I4 work") is removed; its body folded into `parse_command`. - `unknown_command_error` reads the walker registry directly. Touched modules also drop their `crate::dsl::lexer::lex` and `crate::dsl::usage` imports: `app.rs`, `input_render.rs`, `completion.rs`. Tests: 852 passing, 0 failing, 1 ignored (down from 860 because the 8 `dsl::usage::tests::*` tests are gone with the module). |
||
|
claude@clouddev1
|
7bdd3987e1 |
ADR-0024 Phase F (full) step 1: walker-driven highlighting
Replaces the lex()-driven `base_runs` span builder in `input_render.rs` with `walker::highlight_runs`. The new walker-side `dsl::walker::highlight` module returns per-byte `HighlightClass` assignments for every token shape in the source: - For commands the walker engages on, `WalkResult::per_byte_class` is the authoritative source (keyword / identifier / number / string / punct / flag). - Trailing junk past a partial match — and inputs the walker doesn't engage on at all (no registered entry word) — fall through to a byte-shape scanner over `lex_helpers` so unknown command words, stray punctuation, and unterminated strings still highlight sensibly. `Theme::highlight_class_color` is the walker-side analogue of `token_color(&TokenKind)`; the renderer reads `walker::highlight_runs` output and looks up colours through it. `token_color` and the `lex()` pre-pass remain in place for now — the lexer module is still consumed by usage rendering and completion until the remaining Phase F steps land. `HighlightClass`'s and `WalkResult::per_byte_class`'s `#[allow(dead_code)]` annotations come off — they're now part of the production highlight path. Tests: - 16 new tests under `dsl::walker::highlight` cover end-to-end walks, byte-shape fallbacks (unknown commands, bare flags, numbers, punctuation), UTF-8 codepoint advance, and trailing- token handling after partial walks. - Existing `input_render` tests pass unchanged. - 860 total tests passing (727 lib + 133 integration), 1 ignored. Clippy clean with `nursery` lints + `-D warnings`. |
||
|
claude@clouddev1
|
3b36bbb4d6 |
hint: replace misleading "null true false" suggestions at value slots
At value-literal slots (`insert into T values (`, `update T set
col=`, `where col=`, comma positions) the expected-token set
contains null/true/false/number/string-literal. The completion
engine was surfacing the three keyword candidates as Tab options
— actively misleading because the user is usually about to enter
a number, quoted text, or date, and seeing "null true false"
implies those are *the* options. User report (round-6 testing):
"especially not when I'm trying to insert a datetime value and
don't know the correct format for the literal".
Fix: detect the value-literal slot by its expected-set
fingerprint. Suppress Tab candidates at empty prefix. Surface a
prose hint listing all literal forms with format examples
('YYYY-MM-DD' for dates, 'YYYY-MM-DDTHH:MM:SS' for datetimes).
Once the user starts typing a prefix (n / tr / fa), normal
keyword completion still applies.
Schema-aware narrowing (show ONLY the datetime format at a
datetime column) waits on ADR-0023.
Tests: 769 -> 777 passing (+8). Clippy clean.
|
||
|
claude@clouddev1
|
22119d6a4e |
ADR-0022 follow-up r3: identifier colour, NewName hint, "Next:" wording, "type" label
Three fixes from a third round of real testing. 1. **tok_identifier vivid (round-3 #1).** The cool grey-blue from r2 was still too close to theme.fg to register as distinct. Bumped to cyan-teal (#56B6C2 dark / #0F6B76 light) — identifiers are the user's most "special" content and now read that way against keywords (purple), numbers (orange), strings (green), and flags (amber). 2. **"Type a name" hint at NewName slots (round-3 #2).** New `completion::typing_name_at_cursor(input, cursor)` returns `Some(TypingName)` when the cursor sits at — or inside — an `IdentSlot::NewName` position. It probes by substituting a single-letter placeholder identifier and re-parsing to discover what the parser would expect AFTER the name; the hint then reads "Type a name, then `(`" instead of the technical "next: `(`" that surfaces once the partial identifier has been consumed by the live parser. When the probe yields nothing useful (custom errors with empty expected, or a complete-on-substitute case), falls back to "Type a name". New catalog keys hint.ambient_typing_name and hint.ambient_typing_name_then. Wired into ambient_hint between the candidate-list and invalid-ident checks. 3. **"Next:" instead of "expected:" wording.** "Expected" read as a leaked diagnostic; "Next:" is shorter, conversational, and consistent with the action-oriented voice of "Submit with Enter" and "Type a name". Hint sentences now also start capitalised (Submit/Next/Type/No-such), per the user's Capital-T-on- "type a name" preference. 4. **type_keyword labelled "type".** Without a label, the `select_ref!` over an Identifier token produced `RichPattern::SomethingElse`, which rendered as the meaningless "something else" in the hint after `(`. Labelled now: error reads "Next: type" — terse but honest. The label is applied BEFORE try_map (not after, not via as_context) so the existing custom-error wording for unknown types ("unknown type 'varchar' (expected one of: …)") still surfaces unchanged. Tests: 755 passing, 0 failing, 1 ignored (no net change — +5 typing_name cases, -0 net since one test was reworded for capitalisation rather than added). Clippy clean. Smoke probe verifies: "add column to table T: " → "Type a name, then `(`"; "add column to table T: Name (" → "Next: type"; "show data Custp" → "No such table: `Custp`"; valid input → "Submit with Enter". Note for next testing round: parser-side custom errors (e.g. the "tables need at least one column" message that fires for `create table Customers `) still read in lowercase — they're hand-written in parser.rs source rather than via the catalog. If the lowercase "tables need…" intrusion bothers you, easy follow-up. |
||
|
claude@clouddev1
|
bd1cce672d |
ADR-0022 stage 8 follow-up: fixes from real-app testing
Three fixes from the user's testing run, plus an investigation note on a fourth. #4 Sticky hint during cycling. The previous code recomputed candidates_at_cursor at the post-Tab cursor position, which made the panel whiplash through "what comes next at the new cursor" between cycles. ambient_hint now short-circuits to the memo's stored candidate list while the memo is alive — so Tab Tab Tab keeps showing the same list with the selection moving, then snaps to the post-Tab ambient state once any non-Tab key clears the memo. #2 Candidate ordering and kind-coloured rendering. New `Candidate { text, kind: Keyword|Identifier }` carries the classification through completion, last-completion memo, and ambient-hint payload. candidates_at_cursor now sorts keywords first (alphabetical), identifiers second (alphabetical), and the hint-panel renderer colours keywords in `tok_keyword` and identifiers in `tok_identifier`. Keyword-vs-identifier name collisions resolve in favour of the keyword (rare; the user can still address their table via different syntax). #3 tok_identifier no longer matches theme.fg. Identifiers in the input pane now render in a distinct cool grey-blue (dark) / dark steel-blue (light), so they stand out from prose-like default text without competing with keyword purple. Same colour drives the identifier candidates in the hint panel for visual consistency input ↔ hint. Limitation worth knowing: "keywords first, alphabetical" is not the same as grammatical order. For "add column " the hint shows `table to` not `to table` — chumsky's expected-set doesn't preserve combinator-source order, and encoding it in the registry adds maintenance overhead the fix doesn't cleanly justify. Marked for future revisit if it bites. #1 (Tab does nothing on "add column ") — not reproduced through App::update. The internal logic works correctly: "add column " + Tab inserts "Customers ", second Tab cycles to "Orders ", third to "Thing ". The most likely explanation is a stale binary or a terminal-level event intercept (tmux focus, kitty-keyboard protocol differences, etc.) — needs user verification with a fresh build. Tests: 747 passing, 0 failing, 1 ignored (744 baseline → +3: 2 new completion-ordering cases including the keyword-wins-on-name-collision edge, plus 1 hint-mid-cycle sticky test). Clippy clean. |
||
|
claude@clouddev1
|
8214e4136a |
ADR-0022 stage 8e: invalid-identifier detection + hint variant
Per the user's #5: "if our candidate selection works correctly, then entering a character that removes all matches is the same as entering an invalid token." Closes the loop between schema cache (8c/8d) and live error feedback (4). New `completion::invalid_ident_at_cursor(input, cursor, cache)` returns `Some(InvalidIdent { range, found, slot })` when: - the cursor is on a partial identifier-shaped token; - the parser's expected-set at the start of that token contains a known-set IdentSlot (TableName / Column / RelationshipName); - no schema entry across those slots prefix-matches the typed text. `render_input_runs` extended to take a `&SchemaCache` and overlay the invalid-identifier range with `tok_error` — same visual treatment as the parse-error overlay (4), unified red signal regardless of which detector fires. `ambient_hint` extended to surface `hint.ambient_invalid_ident` when invalid_ident_at_cursor returns Some — wording "no such {kind}: `{found}`" mirrors ADR-0019's engine-error voice for consistency. Catalog + KEYS_AND_PLACEHOLDERS declaration added; validator passes. Render priority: candidates win over invalid-ident (if any schema match exists for the partial prefix, the state is "in-progress completion" not "invalid"). Falls through to the existing parse-error/incomplete/Valid framings otherwise. NewName slots are filtered out at the source — typing into a "user invents this name" position is never invalid (per `IdentSlot::completes_from_schema`). Tests: 744 passing, 0 failing, 1 ignored (738 baseline → +6: 5 invalid_ident_at_cursor cases covering unknown-prefix-fires, prefix-match-doesn't-fire, NewName-immune, no-cursor-token, keyword-slot-immune; plus 1 ambient_hint integration test). Clippy clean. This closes ADR-0022. Stages 1-8e together deliver the ambient-typing-assistance feature: token highlighting, error overlay, hint panel ambient, hint panel multi- candidate display with scroll markers, Tab/Shift-Tab cycling with one-keystroke Esc/Backspace undo, schema-aware identifier completion, and invalid-identifier live feedback. Total stage-8 footprint: 5 commits, ~1600 lines. |
||
|
claude@clouddev1
|
51a8d9ac44 |
ADR-0022 stage 8c: IdentSlot propagation + SchemaCache API
`IdentSlot` gains `expected_label()` and the round-trip
`from_expected_label()`. The four slot kinds map to the
user-facing labels "identifier" (NewName), "table name",
"column name", "relationship name".
`ident_ctx(slot)` now actually applies `slot.expected_label()`
as the chumsky label (was documentation-only after stage 6).
Parser errors and the hint panel's "expected: …" prose now
read with the slot-specific name: "expected table name"
instead of the generic "expected identifier". One parser
test updated accordingly; the four catalog `parse.token.*`
keys are unaffected (the slot labels are a parallel surface).
New `completion::SchemaCache { tables, columns,
relationships }` struct + `for_slot(slot) -> &[String]`
accessor. Empty by default; runtime wiring lands in a
follow-on substage. NewName slots return `&[]`
unconditionally.
`candidates_at_cursor` extended to accept `&SchemaCache`:
when the parser's expected-set includes a slot label,
schema candidates from the cache are added alongside the
keyword candidates. Both sources are then prefix-filtered,
combined, sorted, deduplicated. App::schema_cache field
threaded into both the App-side completion paths and the
ambient_hint computation in ui.
Tests: 738 passing, 0 failing, 1 ignored (730 baseline →
+8: 2 IdentSlot label round-trip tests, 6 completion-with-cache
cases covering table/column/relationship slots, prefix
filtering, empty cache, and NewName-no-candidates).
Clippy clean.
User-visible: identifier completion infrastructure is in
place but the cache is always empty — runtime wiring (the
next substage) will populate it on project load and after
successful DDL, at which point Tab on identifier slots
starts offering schema names.
|
||
|
claude@clouddev1
|
faebeed588 |
ADR-0022 stage 8b: hint panel candidate list with scroll markers
Refactor `ambient_hint` to return a richer enum:
- `Prose(String)` — the existing single-line hint (Valid /
incomplete-with-no-keywords / definite-error states);
- `Candidates { items, selected }` — multi-candidate (or
single-candidate) keyword completion at the cursor.
When `candidates_at_cursor` returns Some, the new
`Candidates` variant wins over the prose framing — the
candidate list is more actionable than "expected: `data` or
`table`". `selected` tracks the live `LastCompletion` memo's
selection_idx for the renderer to highlight.
`render_candidate_line` (new helper in ui.rs):
- All items fit → render space-separated; selected item
rendered bold + theme.fg, others theme.muted.
- Overflow → window centred on the selected item (or
item 0 with no selection); `< ` / ` >` markers at the
edges (per the user's #2). Window expands right-first
then left-first to use available width.
- Returns `Line<'static>` (items cloned into spans) so the
caller doesn't fight lifetimes between the
AmbientHint::Candidates payload and the rendered Line.
Updated callers in ui.rs and input_render tests for the new
signature. Added `ambient_hint_with_memo_carries_selected_index`
test asserting the renderer-side `selected` plumbing.
Tests: 730 passing, 0 failing, 1 ignored (728 baseline →
+2 net: -3 reworked + 5 new candidate-related cases).
Clippy clean.
Stage 8c will plumb identifier completion (schema cache +
candidate fetch from worker on demand or pre-cache) and add
the invalid-identifier hint variant.
|
||
|
claude@clouddev1
|
9c4857eb50 |
ADR-0022 stage 5/8: hint panel ambient typing assistance
ParseError::Invalid gains an `expected: Vec<String>` field —
the human-rendered names of the patterns chumsky was looking
for at the failure point (`\`create\``, `identifier`, etc.).
Empty for custom errors, which have no expected-set framing.
Populated by a new `describe_expected()` helper in parser.rs
that humanise() also delegates to (eliminates duplication).
`input_render::ambient_hint(input) -> Option<String>` returns
the hint-panel content per ADR-0022 §6:
- empty input → None (caller falls back to panel.hint_empty);
- Valid → t!("hint.ambient_complete") ("submit with Enter");
- IncompleteAtEof → t!("hint.ambient_expected", expected = …)
listing the parser's expected next tokens, oxford-joined;
- DefiniteErrorAt → t!("hint.ambient_error_with_usage", …)
composing the parse-error message with the matching
parse.usage.* template if a known entry keyword was
consumed, else the bare message.
Catalog gains the three hint.ambient_* keys + validator
declarations.
ui::render_hint_panel resolution order:
1. explicit app.hint (modal contexts) wins;
2. simple-mode + non-empty input → ambient_hint;
3. fallback to panel.hint_empty.
Advanced mode (persistent + one-shot `:`) bypasses ambient
hinting per ADR-0022 §12.
Snapshot: highlighted_input_all_token_classes rebaselined
because the hint panel now displays an ambient hint instead
of the empty placeholder when input is non-empty.
Tests: 698 passing, 0 failing, 1 ignored (693 baseline →
+5 ambient_hint cases). Clippy clean.
Stage 6 introduces the IdentSlot taxonomy + parser audit so
identifier-typed slots can yield schema-aware completion
candidates in stage 8.
|
||
|
claude@clouddev1
|
313d4f8346 |
ADR-0022 stage 4/8: render-time parse + error overlay
Add `classify_input(&str) -> InputState` that returns one of
{Empty, Valid, IncompleteAtEof, DefiniteErrorAt(byte)}.
The renderer uses this to overlay tok_error on the failing
token of mid-typed input that can never be valid.
ParseError::Invalid gains an `at_eof: bool` field populated
by `into_parse_error`:
- structural failures: at_eof = found.is_none()
(chumsky's own "ran out of input" discriminator);
- custom errors from try_map: at_eof = true,
conservatively.
The conservative custom-error classification is a deliberate
under-highlighting bias. It means three classes of error
currently DO NOT get a live red overlay (only on submit):
- "tables need at least one column" (correct: this is
genuinely an incomplete state — adding `with pk ...` fixes it);
- "unknown type 'varchar'" (sub-optimal: should overlay);
- "--force-conversion and --dont-convert are mutually
exclusive" (sub-optimal: should overlay).
The trade-off is documented inline on the at_eof field. A
future refinement could carry an explicit definite/incomplete
tag through Custom errors (would change RichReason::Custom's
payload from String to a typed value).
render_input_runs now applies the overlay on the failing
token's run before injecting the cursor. Tokens after the
error keep their lex-class colour — fixes one thing at a
time per ADR-0022 §4. Lex errors continue to render in
tok_error from stage 2.
Pattern-matches on ParseError::Invalid throughout the
codebase use `..` and are unaffected; only the two
constructions in parser.rs needed updating.
Tests: 693 passing, 0 failing, 1 ignored (683 baseline →
+10: 7 classify + overlay tests, +1 adapted full-command
test, +2 valid-vs-incomplete coverage). Clippy clean.
Stage 5 lights up the hint panel as the verbose-feedback
surface — needs the InputState classifier from this stage.
|
||
|
claude@clouddev1
|
39da399add |
ADR-0022 stage 3/8: simple-mode echo lines highlighted
Lift `dsl::ECHO_PREFIX = "running: "` as a public const,
with a unit test asserting `t!("dsl.running", input = "")`
matches it. The catalog template is now contracted to equal
`format!("{ECHO_PREFIX}{input}")` — a translator changing
the prefix breaks the test.
Add `input_render::lex_to_runs(input, theme)` — a
cursor-less variant of `render_input_runs` for use cases
(echo lines, future hint panel) that need token-class
colouring without an inverted cursor.
ui::render_output_line: when the line is an Echo submitted
in Simple mode, peel the prefix and re-tokenise the rest
through lex_to_runs, rendering each token at its class
colour. Advanced-mode echoes and any echo whose body
unexpectedly lacks the prefix fall through to the plain
rendering.
Tests: 683 passing, 0 failing, 1 ignored (682 baseline →
+1 echo_prefix_matches_catalog_template). Clippy clean
(uses let-chain to keep the if condition flat).
Stage 4 adds render-time parse + error overlay so the
failing token in mid-typed input lights up in the error
colour.
|
||
|
claude@clouddev1
|
cafc455c8a |
ADR-0022 stage 2/8: input panel — token-class highlighting
New `input_render` module with `render_input_runs(input, cursor_byte, theme) -> Vec<StyledRun>`. Lexes the input, assigns each token its `theme.token_color`, preserves whitespace gaps as `theme.fg` runs, and injects the cursor by splitting the run that contains it into before/under/after sub-spans (under marked Modifier::REVERSED). End-of-input cursor is an empty-range sentinel rendered as an inverted space. ui::render_input_panel switches over EffectiveMode: simple mode goes through render_input_runs + a small runs_to_spans helper that borrows from the input string; advanced modes (persistent + one-shot `:`) keep the previous plain before/under/after rendering since the DSL lexer doesn't speak SQL (ADR-0022 §12). Multi-byte UTF-8 in string literals is handled by walking to the next char boundary when splitting the cursor run, mirroring the previous renderer. Tests: 682 passing, 0 failing, 1 ignored (672 baseline → +10: 9 input_render unit tests covering each token class, cursor placements, multi-byte, full-command shape; +1 new "all token classes" UI snapshot). Clippy clean. Caveat (noted inline in the new snapshot test): the TestBackend/render_to_string path records text symbols only, not ratatui style. The new snapshot is therefore a text-layout regression net; the unit tests in input_render::tests are the authoritative regression net for colour mappings. Stage 3 wires the same colouring into simple-mode echo lines in the output panel. |