# TypoSuggest — Complete Technical & Protocol Reference (`llms-full.txt`)

> This document is the deep-dive technical companion to https://typosuggest.pial.digital/llms.txt. While `/llms.txt` provides a concise product overview and quickstart index, `/llms-full.txt` specifies the complete internal data structures, Unicode script normalization rules, exact request/response schemas across every REST and MCP endpoint, deterministic ranking internals, and production batch-ingestion patterns.

- **Production Base URL**: `https://typosuggest.pial.digital`
- **Concise Reference**: https://typosuggest.pial.digital/llms.txt
- **OpenAPI 3.1 Spec**: https://typosuggest.pial.digital/openapi.yaml
- **Interactive Docs**: https://typosuggest.pial.digital/docs
- **MCP Endpoint**: `https://typosuggest.pial.digital/mcp` (JSON-RPC 2.0, protocol `2024-11-05`)

---

## 1. Internal Engine Data Structures (`internal/engine`)

Each tenant dataset is isolated in its own in-memory `DatasetIndex` instance managed by `IndexManager` and guarded by a reader-writer mutex (`sync.RWMutex`):

### 1.1 `DatasetConfig`
| Field | Go Type | JSON Field | Default | Description |
| :--- | :--- | :--- | :--- | :--- |
| `TypoTolerance` | `bool` | `typo_tolerance` | `true` | Enables Stage 3 SymSpell + Damerau-Levenshtein typo recovery. |
| `BoilerplateRegex` | `string` | `boilerplate_regex` | `(?i)^i\s+will\s+(be\s+(your\s+)?\|do\s+)?` | Compiled regular expression stripped from the start of ingested strings before phrase extraction. |
| `MaxNgramWords` | `int` | `max_ngram_words` | `4` (clamped `1..10`) | Maximum word count for leading n-gram sub-phrases extracted from each ingested item. |
| `GroupCaps` | `map[string]int` | `group_caps` | `{}` (optional) | Per-group result cap enforced at query time (e.g. `{"category": 2}`). |

### 1.2 `PhraseEntry` & Index Tables
- **`Phrases` (`[]PhraseEntry`)**: Contiguous slice indexed by `uint32` `PhraseID`.
  - `Normalized` (`string`): Lowercased, NFC-normalized, whitespace-collapsed lookup key.
  - `DisplayText` (`string`): Human-readable phrase returned in `Suggestion.text`.
  - `Group` (`string`): Category tag (defaults to `"item"` when omitted on ingest).
  - `RefCount` (`uint32`): Cumulative frequency count across ingested items (`Suggestion.frequency`). Deleted phrases have `RefCount == 0`.
- **`PhraseLookup` (`map[string]uint32`)**: Maps `Normalized` phrase text to its `PhraseID` in `O(1)`.
- **`TokenMap` (`map[string][]uint32`)**: Inverted index mapping each UAX #29 word token (plus Latin diacritic-folded and Arabic-normalized variants) to a deduplicated list of `PhraseID`s.
- **`TopTokenPostings` (`map[string][]uint32`)**: Pre-sorted top-64 `PhraseID`s (by `RefCount` descending) per token for instant single-token Stage 1/Stage 2 lookups without scanning full posting lists.
- **`SortedTokens` (`[]string`)**: Lexicographically sorted slice of all unique tokens, rebuilt lazily (`tokensDirty`) for `O(log V)` binary-search prefix expansion.
- **`TypoMap` (`map[string][]string`)**: SymSpell deletion hash table mapping 1-edit and 2-edit rune deletion strings to exact dictionary tokens. For tokens longer than 8 runes (`symSpellMaxPrefixLen = 8`), deletes are generated from the 8-rune prefix as well as the full token.
- **`DocHashes` (`map[uint64][]uint32`)**: Maps the 64-bit FNV-1a document hash to the `PhraseID`s incremented by that document, enabling idempotency detection (`skipped_duplicates`) and clean `RefCount` decrements on item deletion or update.

---

## 2. Unicode Normalization & Phrase Extraction Pipeline (`internal/normalizer`)

Every ingested item and search query passes through a deterministic Unicode pipeline:

1. **Length Guardrail (`MaxQueryLength = 100`)**: Search queries longer than 100 bytes are truncated cleanly on a valid UTF-8 rune boundary.
2. **Unicode NFC & Case Folding**: Input is normalized to Unicode Canonical Composition (`norm.NFC`) and lowercased via `strings.ToLower`.
3. **Bengali Script Normalization (`NormalizeBengali`)**:
   - Normalizes precomposed nukta consonants (`ড়` -> `ড`+`়`, `ঢ়` -> `ঢ`+`়`, `য়` -> `য`+`়`).
   - Normalizes Assamese `ৰ` (`U+09F0`) and `ৱ` (`U+09F1`) to standard Bengali `র` (`U+09B0`) and `ব` (`U+09AC`).
   - Strips zero-width non-joiner (`U+200C`) and zero-width joiner (`U+200D`) so `"র‍্যাব"` and `"র্যাব"` converge to the same token.
4. **Arabic / Persian / Urdu Normalization (`NormalizeArabic`)**:
   - Strips harakat/tashkeel diacritics (`U+064B..U+065F`, `U+0670`) and tatweel/kashida (`U+0640`).
   - Unifies Alef variants (`أ`, `إ`, `آ`, `ٱ` -> `ا`), Teh Marbuta (`ة` -> `ه`), Persian Yeh/Alef Maksura (`ى`, `ی` -> `ي`), and Persian Kaf (`ک` -> `ك`).
5. **Latin Diacritic Folding (`FoldLatinDiacritics`)**:
   - Decomposes via `norm.NFD`, strips nonspacing marks (`unicode.Mn`), and maps special ligatures/letters (`ß` -> `ss`, `ø` -> `o`, `ł` -> `l`, `đ` -> `d`, `æ` -> `ae`, `œ` -> `oe`).
6. **UAX #29 Word Segmentation (`TokenizeWords`)**:
   - Uses `github.com/rivo/uniseg` (`NewWordSegmenter`) to split text on Unicode Standard Annex #29 word boundaries, preserving grapheme clusters in Brahmic (Bengali, Devanagari), Arabic, and CJK scripts.
7. **Boilerplate Stripping & N-Gram Extraction (`ExtractPhrases`)**:
   - On ingestion, `BoilerplateRegex` is stripped from the start of the text.
   - Leading n-grams from `1` up to `MaxNgramWords` tokens are extracted (skipping single-word stop words or pure numbers), and the full cleaned phrase (up to 10 words) is appended.

---

## 3. Detailed 4-Stage Retrieval & Post-Processing Specification

### 3.1 Empty Query Behavior (`q == ""`)
When `q` (and `query`) is empty after normalization:
- The engine scans up to `10,000` active phrases (`RefCount > 0`), computes `score = 25.0 * log10(1 + RefCount)`, sorts by `score` descending (tie-broken by `RefCount` descending), enforces `GroupCaps`, and returns the top `limit` suggestions.

### 3.2 Stage Execution & Candidate Caps
| Stage | Name | Base Score | Max Candidates | Trigger Condition |
| :--- | :--- | :--- | :--- | :--- |
| **Stage 1** | Direct Prefix Lookup (Redis `ZRANGEBYLEX` + `SortedTokens` binary search) | `100.0` | `50` (Redis per variant) / `100` (in-memory) | Always runs on non-empty query. |
| **Stage 2** | Multi-Token Inverted Index (`TokenMap` intersection + prefix expansion) | `80.0` (exact) / `78.0` (single-token prefix fan-out) | `250` | Always runs on tokenized query (`len(qTokens) >= 1`). |
| **Stage 3** | SymSpell + True Damerau-Levenshtein Typo Resolution | `55.0` | `250` | Runs if `enableTypo == true` AND (`len(candidates) == 0` OR any query token had no exact/prefix match in `TokenMap`). |
| **Stage 4** | Progressive Right-to-Left Token Fallback (`qTokens[:k]`) | `25.0` | `250` | Runs only if `len(candidates) == 0` AND `len(qTokens) > 1`. |

### 3.3 True Damerau-Levenshtein Thresholds by Token Rune Length
In Stage 3 (`SymSpellLookupTokensWithDist`), edit distance is restricted by the rune count of each query token `n = utf8.RuneCountInString(qTok)`:
- `n < 4`: `0` edits allowed (skipped in Stage 3; matched via prefix or Latin diacritic folding only).
- `4 <= n <= 5`: Maximum edit distance `1` (1 insertion, 1 deletion, 1 substitution, or 1 adjacent transposition).
- `n >= 6`: Maximum edit distance `2`.

### 3.4 RefCount-Aware Sub-Phrase Pruning
Before applying `GroupCaps`, the top candidate pool (`max(limit * 4, 40)`) is pruned so shorter n-gram fragments do not clutter results alongside their full phrase:
- A shorter candidate $A$ is pruned by a longer candidate $B$ if and only if:
  1. $A.\text{Normalized} \neq \text{normQuery}$ (exact query matches are never pruned), AND
  2. $B.\text{Normalized}$ starts with $A.\text{Normalized} + \text{" "}$ OR ends with $\text{" "} + A.\text{Normalized}$, AND
  3. $A.\text{RefCount} \le B.\text{RefCount}$ (if the shorter sub-phrase is genuinely more popular across multiple documents than the longer phrase, both are kept).

---

## 4. Complete REST API Specification

### 4.1 Authentication Matrix & Query-String Security
| Route Pattern | `X-API-Key: ac_pub_...` | `?key=ac_pub_...` / `?api_key=` | `X-API-Key: ac_sec_...` | `Authorization: Bearer <jwt>` |
| :--- | :--- | :--- | :--- | :--- |
| `GET /v1/suggest` | Allowed (`public`) | Allowed (`public` only) | Allowed (`secret`) | Allowed (`secret`) |
| `GET /v1/datasets/{id}/stats` | Public (no auth required) | Public (no auth required) | Public (no auth required) | Public (no auth required) |
| `POST /v1/datasets` | `403 Forbidden` | Do not use for `ac_sec_...` | Allowed (`secret`) | Allowed (`secret`) |
| `DELETE /v1/datasets/{id}` | `403 Forbidden` | Do not use for `ac_sec_...` | Allowed (`secret`) | Allowed (`secret`) |
| `POST /v1/datasets/{id}/reset` | `403 Forbidden` | Do not use for `ac_sec_...` | Allowed (`secret`) | Allowed (`secret`) |
| `POST /v1/datasets/{id}/items:add` | `403 Forbidden` | Do not use for `ac_sec_...` | Allowed (`secret`) | Allowed (`secret`) |
| `POST /v1/datasets/{id}/items:delete` | `403 Forbidden` | Do not use for `ac_sec_...` | Allowed (`secret`) | Allowed (`secret`) |
| `PUT /v1/datasets/{id}/items:update` | `403 Forbidden` | Do not use for `ac_sec_...` | Allowed (`secret`) | Allowed (`secret`) |
| `/v1/dashboard/*` | `403 Forbidden` | Do not use for `ac_sec_...` | Allowed (`secret`) | Allowed (`secret`) |
| `POST /v1/auth/verify-email` | `403 Forbidden` | Do not use | Allowed (`secret`) | Allowed (`secret`) |
| `POST /v1/auth/resend-code` | `403 Forbidden` | Do not use | Allowed (`secret`) | Allowed (`secret`) |

> **Security Rule for `?key=` Query Parameters**: Query-string parameters (`?key=ac_pub_...` or `?api_key=ac_pub_...`) are designed strictly for read-only browser `GET /v1/suggest` requests to bypass CORS `OPTIONS` preflight round-trips. Because URL query strings can be captured in server logs, CDN logs, browser history, and APM tools, **never pass secret keys (`ac_sec_...`) or JWT tokens in URL query strings**.

---

### 4.2 Search & Dataset Endpoints

#### `GET /v1/suggest`
Executes the 4-stage autocomplete query pipeline and asynchronously increments monthly query usage.
- **Query Parameters**:
  - `dataset_id` (`string`, required): Dataset UUID.
  - `q` (`string`, optional; fallback alias `query`): Search string (max 100 bytes).
  - `limit` (`int`, optional): Max suggestions (`1..20`, default `8`).
  - `typo` (`bool`, optional): Overrides dataset `typo_tolerance` (`true`/`1` or `false`/`0`).
  - `key` (`string`, optional; alias `api_key`): Public read-only API key (`ac_pub_...`) for zero-preflight browser requests.
- **Response (`200 OK`, `models.SuggestResponse`)**:
```json
{
  "query": "macbok pro",
  "took_ms": 0.165,
  "suggestions": [
    {
      "text": "Apple MacBook Pro 16 inch M3 Max",
      "group": "product",
      "score": 91.2,
      "frequency": 25
    }
  ]
}
```
- **Error Statuses**: `400` (missing `dataset_id`), `401` (invalid/missing key), `402` (monthly search quota `10,000` exceeded or subscription paused), `403` (dataset belongs to another tenant), `404` (dataset not found), `429` (exceeded 60 req/s burst 120).

---

#### `POST /v1/datasets`
Creates a new dataset (`secret` scope required via `X-API-Key: ac_sec_...` or `Authorization: Bearer <jwt>`). Enforces the tenant dataset quota (`2` datasets on Beta plan).
- **Request Body (`models.CreateDatasetRequest`)**:
```json
{
  "name": "Pharmacy Inventory",
  "typo_tolerance": true,
  "boilerplate_regex": "(?i)^i\\s+will\\s+(be\\s+(your\\s+)?|do\\s+)?",
  "max_ngram_words": 4,
  "group_caps": {
    "pharmacy": 5
  }
}
```
- **Response (`201 Created`, `models.CreateDatasetResponse`)**:
```json
{
  "dataset": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "tenant_id": "2c9d8f10-91b2-4a88-b822-112233445566",
    "name": "Pharmacy Inventory",
    "typo_tolerance": true,
    "boilerplate_regex": "(?i)^i\\s+will\\s+(be\\s+(your\\s+)?|do\\s+)?",
    "group_caps": {
      "pharmacy": 5
    },
    "max_ngram_words": 4,
    "created_at": "2026-10-31T08:00:00Z",
    "updated_at": "2026-10-31T08:00:00Z"
  }
}
```
- **Error Statuses**: `400` (missing `name`), `401` (unauthenticated), `402` (dataset quota `2/2` reached), `403` (public key used).

---

#### `GET /v1/datasets/{dataset_id}/stats`
Returns real-time in-memory and disk snapshot telemetry for a dataset (publicly accessible for playground and monitoring).
- **Response (`200 OK`, `models.DatasetStatsResponse`)**:
```json
{
  "dataset_id": "550e8400-e29b-41d4-a716-446655440000",
  "total_rows_ingested": 150,
  "active_unique_phrases": 412,
  "unique_dictionary_tokens": 285,
  "symspell_keys": 1840,
  "queries_served": 1024,
  "avg_query_latency_ms": 0.084,
  "snapshot_size_bytes": 18432
}
```

---

#### `POST /v1/datasets/{dataset_id}/items:add`
Batch ingests up to `25,000` items (`10 MB` max body) into the dataset (`secret` scope required). Enforces the tenant row quota (`500` total rows on Beta plan).
- **Request Body (`models.IngestRequest`)**:
  Supports structured `items` (`[]IngestItem`), raw `texts` (`[]string`), or both in the same payload:
```json
{
  "items": [
    {
      "id": "pharm-001",
      "text": "Paracetamol 500mg Tablet",
      "group": "pharmacy",
      "count": 10
    }
  ],
  "texts": [
    "Amoxicillin 250mg Capsule"
  ]
}
```
- **Response (`200 OK`, `models.IngestResponse`)**:
```json
{
  "added": 2,
  "skipped_duplicates": 0,
  "active_phrases": 7,
  "took_ms": 0.42
}
```

---

#### `POST /v1/datasets/{dataset_id}/items:delete`
Batch deletes up to `25,000` items (`10 MB` max body) from the dataset (`secret` scope required) by matching `ids` (`[]string`) and/or `texts` (`[]string`), decrementing `RefCount` on all phrases generated by each deleted document.
- **Request Body (`models.DeleteRequest`)**:
```json
{
  "ids": ["pharm-001"],
  "texts": ["Amoxicillin 250mg Capsule"]
}
```
- **Response (`200 OK`, `models.DeleteResponse`)**:
```json
{
  "deleted_docs": 2,
  "purged_phrases": 7,
  "active_phrases": 0,
  "took_ms": 0.31
}
```

---

#### `PUT /v1/datasets/{dataset_id}/items:update`
Atomically updates an item by `id` (`1 MB` max body, `secret` scope required).
- **Request Body (`models.UpdateItemRequest`)**:
```json
{
  "id": "pharm-001",
  "old_text": "Paracetamol 500mg Tablet",
  "new_text": "Paracetamol 500mg Effervescent Tablet",
  "group": "pharmacy"
}
```
  - `id` (`string`, required): Document ID to update.
  - `old_text` (`string`, optional): Previous text if known.
  - `new_text` (`string`, required; alias `text`): Replacement text to index.
  - `group` (`string`, optional): Category group tag.
- **Response (`200 OK`, `models.UpdateItemResponse`)**:
```json
{
  "success": true,
  "active_phrases": 5,
  "took_ms": 0.55
}
```

---

#### `POST /v1/datasets/{dataset_id}/reset`
Wipes all in-memory index structures, deletes `tenant:{dataset_id}:zset` from Redis, and deletes `./snapshots/{dataset_id}.idx` from disk while preserving the dataset record (`secret` scope required).
- **Response (`200 OK`)**:
```json
{
  "dataset_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "reset_success",
  "active_unique_phrases": 0,
  "total_rows_ingested": 0
}
```

---

#### `DELETE /v1/datasets/{dataset_id}`
Permanently deletes the dataset from PostgreSQL, evicts it from RAM, removes its Redis `ZSET`, and deletes its `.idx` snapshot file (`secret` scope required).
- **Response (`200 OK`)**:
```json
{
  "status": "deleted",
  "dataset_id": "550e8400-e29b-41d4-a716-446655440000"
}
```

---

#### `POST /v1/datasets/{dataset_id}/seed`
Seeds synthetic multi-domain items (`models.SeedDatasetRequest`: `{"count": 500, "reset": false}`).
- **Production Restriction**: When `APP_ENV=production`, this endpoint requires either `X-Master-Admin-Key` or a Superadmin JWT; normal tenant credentials receive `403 Forbidden` (`"seeding is disabled in production; founder admin authorization required"`).

---

### 4.3 Public, Auth & Tenant Dashboard Endpoints

#### `GET /health`
Checks PostgreSQL connectivity (`2s` timeout), Redis connectivity (`2s` timeout), and `./snapshots` directory writeability. Results are cached in memory for 5 seconds under a mutex (`healthMu`), and error details are sanitized (`"unreachable"` / `"unwritable"`) so internal driver errors are never exposed publicly.
- **Response (`200 OK` when healthy, `503 Service Unavailable` when degraded, `models.HealthResponse`)**:
```json
{
  "status": "ok",
  "postgres": true,
  "redis": true,
  "snapshots": true,
  "details": {
    "postgres": "connected",
    "redis": "connected",
    "snapshots": "writable"
  }
}
```

#### `GET /v1/demo-session`
Returns the cached playground session (provisioning and seeding the demo dataset if needed) with a read-only `ac_pub_...` key.
- **Response (`200 OK`, `models.DemoSessionResponse`)**:
```json
{
  "tenant_id": "...",
  "dataset_id": "...",
  "api_key": "ac_pub_...",
  "dataset_name": "Demo Multi-Domain Dataset"
}
```

#### `POST /v1/contact`
Submits a contact or custom-plan inquiry (rate limited to `15 req/min` per IP).
- **Request Body (`models.ContactRequest`)**:
```json
{
  "name": "Jane Doe",
  "email": "jane@example.com",
  "category": "Quota Request",
  "message": "We need a 2M-row catalog plan."
}
```
- **Response (`201 Created`)**:
```json
{
  "status": "ok",
  "id": "c81d4e2e-bcf2-11e6-869b-7df92533d2db",
  "message": "Your message has been received. We will get back to you shortly."
}
```

#### `POST /v1/auth/register`
Registers a new tenant account (`name`, `email`, `password` minimum 8 characters), provisions a starter dataset (`starter_dataset_id`), issues an initial `secret_key` (`ac_sec_...`) and `public_key` (`ac_pub_...`), and dispatches a 6-digit verification code via email.
- **Request Body (`models.RegisterRequest`)**:
```json
{
  "name": "Example Org",
  "email": "dev@example.com",
  "password": "securepassword123"
}
```
- **Response (`201 Created`, `models.RegisterResponse`)**:
```json
{
  "token": "<30-day-jwt>",
  "tenant": {
    "id": "...",
    "name": "Example Org",
    "email": "dev@example.com",
    "email_verified": false,
    "plan_tier": "indie_beta",
    "subscription_status": "active",
    "max_datasets": 2,
    "max_rows": 500,
    "max_monthly_queries": 10000,
    "monthly_queries_used": 0,
    "quota_reset_at": "2026-11-30T08:00:00Z",
    "enforce_paywall": true
  },
  "starter_dataset_id": "...",
  "secret_key": "ac_sec_...",
  "public_key": "ac_pub_...",
  "email_verified": false,
  "requires_verification": true
}
```

#### `POST /v1/auth/verify-email` (Requires `Authorization: Bearer <jwt>` or `X-API-Key: ac_sec_...`)
Verifies the 6-digit OTP code (`{"code": "123456"}`) for the authenticated tenant and returns `{"verified": true, "message": "Email verified successfully"}` (`models.VerifyEmailResponse`).

#### `POST /v1/auth/resend-code` (Requires `Authorization: Bearer <jwt>` or `X-API-Key: ac_sec_...`)
Generates a fresh 6-digit OTP code (valid for 15 minutes) and resends it to the authenticated tenant's email address.

#### `POST /v1/auth/login`
Authenticates an existing tenant (`{"email": "...", "password": "..."}`) and returns `models.LoginResponse` (`{"token": "<30-day-jwt>", "tenant": {...}}`).

#### `GET /v1/dashboard/overview` (Requires `Authorization: Bearer <jwt>` or `X-API-Key: ac_sec_...`)
Returns `models.DashboardOverviewResponse` containing `tenant` (`models.TenantSummary`), `usage` (`datasets_used`, `max_datasets: 2`, `total_rows_used`, `max_rows: 500`, `monthly_queries_used`, `max_monthly_queries: 10000`), `datasets` (`[]models.DashboardDatasetItem`), and `api_keys` (`[]models.DashboardAPIKeyItem`).

#### `POST /v1/dashboard/keys` (Requires `Authorization: Bearer <jwt>` or `X-API-Key: ac_sec_...`)
Creates a new API key (`{"name": "Frontend Search Key", "role": "public"}` or `"secret"`). Returns `201 Created` with `models.CreateKeyResponse` (`id`, `name`, `role`, `key_prefix`, `raw_key`, `created_at`).

#### `DELETE /v1/dashboard/keys/{key_id}` (Requires `Authorization: Bearer <jwt>` or `X-API-Key: ac_sec_...`)
Revokes the specified API key and evicts it from the auth cache (`200 OK`: `{"status": "revoked", "key_id": "..."}`).

---

## 5. Complete Model Context Protocol (MCP) Wire Specification (`POST /mcp`)

TypoSuggest implements a stateless HTTP JSON-RPC 2.0 MCP server at `https://typosuggest.pial.digital/mcp` (`protocolVersion: "2024-11-05"`).

### 5.1 Authentication on `/mcp`
`/mcp` accepts API keys (`ac_pub_...` or `ac_sec_...`) or JWTs via headers (`X-API-Key: ac_sec_...` or `Authorization: Bearer ac_sec_...`). Query parameters (`?api_key=` / `?key=`) are also supported as a fallback for MCP clients that cannot configure custom headers, though header authentication is strongly recommended so secret keys are not recorded in URL logs.

Note: `initialize`, `notifications/initialized`, `ping`, and `tools/list` succeed without authentication so MCP clients can discover capabilities before invoking `tools/call`. `tools/call` requires authentication.

### 5.2 JSON-RPC 2.0 Methods

#### 1. `initialize`
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "initialize",
  "params": {
    "protocolVersion": "2024-11-05",
    "capabilities": {},
    "clientInfo": { "name": "claude-desktop", "version": "1.0.0" }
  }
}
```
**Response**:
```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "protocolVersion": "2024-11-05",
    "capabilities": {
      "tools": { "listChanged": false }
    },
    "serverInfo": {
      "name": "typosuggest-mcp-server",
      "version": "1.0.0"
    },
    "instructions": "TypoSuggest MCP Server: Sub-millisecond typo-tolerant autocomplete & instant search engine. Use 'list_datasets' to inspect available datasets, 'suggest' to run typo-tolerant search queries, and 'add_items' or 'create_dataset' (requires secret key ac_sec_...) to manage data."
  }
}
```

#### 2. `tools/list`
Returns the 6 registered tools (`suggest`, `list_datasets`, `get_dataset_stats`, `create_dataset`, `add_items`, `delete_dataset`).

#### 3. `tools/call` — Detailed Tool Schemas & Payloads

##### Tool 1: `suggest` (`public` or `secret` key)
```json
{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "suggest",
    "arguments": {
      "dataset_id": "550e8400-e29b-41d4-a716-446655440000",
      "query": "paracitaml",
      "limit": 5,
      "typo_tolerance": true
    }
  }
}
```

##### Tool 2: `list_datasets` (`public` or `secret` key)
```json
{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "tools/call",
  "params": {
    "name": "list_datasets",
    "arguments": {}
  }
}
```

##### Tool 3: `get_dataset_stats` (`public` or `secret` key)
```json
{
  "jsonrpc": "2.0",
  "id": 4,
  "method": "tools/call",
  "params": {
    "name": "get_dataset_stats",
    "arguments": {
      "dataset_id": "550e8400-e29b-41d4-a716-446655440000"
    }
  }
}
```

##### Tool 4: `create_dataset` (`secret` key `ac_sec_...` required)
```json
{
  "jsonrpc": "2.0",
  "id": 5,
  "method": "tools/call",
  "params": {
    "name": "create_dataset",
    "arguments": {
      "name": "AI Agent Knowledge Catalog",
      "typo_tolerance": 2,
      "max_ngram_words": 4
    }
  }
}
```
> **Implementation Detail**: `create_dataset` accepts `typo_tolerance` as either an integer (`0` = disabled, `1` or `2` = enabled) or a boolean (`true`/`false`), enabling the dataset's `TypoTolerance` flag. Actual edit distance limits (`0` for `<4` runes, `1` for `4–5` runes, `2` for `>=6` runes) are enforced dynamically per token rune length.

##### Tool 5: `add_items` (`secret` key `ac_sec_...` required)
```json
{
  "jsonrpc": "2.0",
  "id": 6,
  "method": "tools/call",
  "params": {
    "name": "add_items",
    "arguments": {
      "dataset_id": "550e8400-e29b-41d4-a716-446655440000",
      "items": [
        {
          "id": "item-1",
          "text": "Paracetamol 500mg Tablet",
          "group": "pharmacy",
          "count": 10
        }
      ]
    }
  }
}
```

##### Tool 6: `delete_dataset` (`secret` key `ac_sec_...` required)
```json
{
  "jsonrpc": "2.0",
  "id": 7,
  "method": "tools/call",
  "params": {
    "name": "delete_dataset",
    "arguments": {
      "dataset_id": "550e8400-e29b-41d4-a716-446655440000"
    }
  }
}
```

---

## 6. Production Batch Ingestion & Error-Handling Patterns

### Chunked Batch Ingestion with Rate-Limit Backoff (Node.js / TypeScript)
When ingesting large catalogs (on custom/enterprise plans), chunk payloads into batches of `5,000–10,000` items (below the `25,000` item / `10 MB` hard cap) and respect `HTTP 429` (`Retry-After: 1`):

```typescript
interface IngestItem {
  id?: string;
  text: string;
  group?: string;
  count?: number;
}

export async function ingestInBatches(
  datasetId: string,
  secretKey: string,
  allItems: IngestItem[],
  batchSize = 5000
) {
  let totalAdded = 0;
  let totalSkipped = 0;

  for (let i = 0; i < allItems.length; i += batchSize) {
    const chunk = allItems.slice(i, i + batchSize);
    let attempts = 0;

    while (true) {
      attempts++;
      const res = await fetch(
        `https://typosuggest.pial.digital/v1/datasets/${encodeURIComponent(datasetId)}/items:add`,
        {
          method: 'POST',
          headers: {
            'X-API-Key': secretKey,
            'Content-Type': 'application/json',
          },
          body: JSON.stringify({ items: chunk }),
          signal: AbortSignal.timeout(15000),
        }
      );

      if (res.status === 429 && attempts <= 3) {
        const retrySec = Number(res.headers.get('Retry-After') || '1');
        await new Promise((r) => setTimeout(r, retrySec * 1000));
        continue;
      }

      const body = await res.json().catch(() => ({}));
      if (!res.ok) {
        throw new Error(`Batch [${i}..${i + chunk.length}] failed (${res.status}): ${body.error || res.statusText}`);
      }

      totalAdded += body.added ?? 0;
      totalSkipped += body.skipped_duplicates ?? 0;
      break;
    }
  }

  return { totalAdded, totalSkipped };
}
```