# TypoSuggest — Sub-Millisecond Typo-Tolerant Autocomplete & Instant Search API

> TypoSuggest (https://typosuggest.pial.digital) is a hosted, multi-tenant autocomplete and prefix-search engine written in Go. It combines an in-memory 4-stage hybrid retrieval pipeline (Prefix Lookup, UAX #29 Multi-Token Inverted Index, SymSpell + Damerau-Levenshtein Typo Resolution, and Right-to-Left Token Fallback) with Redis sorted-set acceleration, PostgreSQL tenant/key/quota persistence, Zstandard (`.idx`) binary snapshots, and a native Model Context Protocol (MCP) server (`/mcp`).

- **Production Base URL**: `https://typosuggest.pial.digital`
- **Interactive Web Playground**: https://typosuggest.pial.digital/
- **Developer Documentation**: https://typosuggest.pial.digital/docs
- **Self-Serve Agent Skill (`skill.md`)**: https://typosuggest.pial.digital/skill.md
- **Full Technical Reference (LLMs)**: https://typosuggest.pial.digital/llms-full.txt
- **OpenAPI 3.1 Specification**: https://typosuggest.pial.digital/openapi.yaml
- **MCP Endpoint (JSON-RPC 2.0)**: `https://typosuggest.pial.digital/mcp`

---

## 1. Product Overview & Positioning

TypoSuggest is purpose-built for instant typeahead, command palettes, and search-as-you-type dropdowns where every keystroke must resolve in under a millisecond. Unlike general-purpose full-text document search engines (Elasticsearch, OpenSearch, Algolia, Typesense, Meilisearch) that index multi-kilobyte document bodies and incur heavier memory and serialization overhead, TypoSuggest indexes normalized short strings, n-gram sub-phrases, and token posting lists directly in Go heap memory with low-allocation hot paths.

### When to Choose TypoSuggest
- You need sub-millisecond (`< 1 ms` server-side engine execution) autocomplete suggestions on every keystroke.
- Users frequently make 1–2 character typos, transpositions (`"wrodpres"` -> `"wordpress"`), space omissions (`"macbookpro"`), or out-of-order multi-word queries (`"website wordpress"`).
- Your catalog contains repetitive marketplace titles (e.g., `"I will build a modern WordPress website..."`) that benefit from automatic boilerplate prefix stripping, n-gram extraction, and `RefCount` frequency deduplication.
- You need out-of-the-box multilingual script normalization (Latin diacritics, Bengali, Arabic/Persian/Urdu, Devanagari, CJK) without custom analyzer plugins.
- You want AI assistants (Claude Desktop, Cursor, Windsurf, custom agents) to provision datasets, ingest items, and query autocomplete via a built-in remote MCP server.

---

## 2. Target Users & Use Cases

- **E-Commerce & Retail Storefronts**: Instant product, brand, model, and SKU autocomplete with category group caps (`group_caps`).
- **Service Marketplaces & Job Boards**: Deduplicating repetitive seller gig titles into clean, high-signal search suggestions.
- **Pharmacy & Healthcare Directories**: Resolving misspelled drug names and dosage strengths (`"paracitaml 500mg"`, `"amoxcillin"`).
- **SaaS Command Palettes (`Cmd+K`)**: Fast navigation across actions, settings, and entities.
- **Multilingual Portals**: Native support for South Asian (Bengali, Hindi/Devanagari), Middle Eastern (Arabic, Persian, Urdu), CJK, and European scripts.
- **AI Coding Assistants & Autonomous Agents**: Grounding LLM tool calls against live product catalogs via `POST /mcp`.

---

## 3. Core Capabilities

1. **4-Stage Hybrid Retrieval Engine**: Prefix scan -> Multi-token inverted index intersection -> SymSpell + true Damerau-Levenshtein typo recovery -> Progressive right-to-left token fallback.
2. **Boilerplate Prefix Stripping & N-Gram Extraction**: Automatically strips configurable regex prefixes (default: `(?i)^i\s+will\s+(be\s+(your\s+)?|do\s+)?`) and extracts leading n-grams (up to `max_ngram_words`, default `4`, clamped `1..10`) alongside full phrases.
3. **Frequency-Weighted Deduplication (`RefCount`)**: Identical normalized phrases merge into a single `PhraseEntry` whose `RefCount` (`frequency`) accumulates across duplicate rows or explicit `count` weights.
4. **Unicode UAX #29 & Multi-Script Normalization**: NFC normalization, grapheme-safe UAX #29 word segmentation, Latin diacritic folding (`"café"` -> `"cafe"`), Bengali Assamese/nukta/ZWJ normalization, and Arabic/Persian/Urdu alef/teh-marbuta/tashkeel folding.
5. **Group Diversity Caps (`group_caps`)**: Enforces per-category maximums in the final response (e.g., `{"category": 2}` limits `"category"` suggestions to at most 2 slots).
6. **RefCount-Aware Sub-Phrase Pruning**: Suppresses redundant shorter prefix/suffix sub-phrases when a longer completion in the result set has an equal or higher `RefCount`.
7. **Dual-Scoped API Keys**: Public keys (`ac_pub_...`, role `public`) are strictly read-only (`GET /v1/suggest`), while secret keys (`ac_sec_...`, role `secret`) authorize dataset management and item ingestion.
8. **Zstandard (`.idx`) Binary Snapshots**: Asynchronous Copy-on-Write (CoW) disk snapshots compressed with Zstandard (`zstd`), persisted via atomic `.tmp` rename.

---

## 4. Matching Algorithm & Limitations

Every `GET /v1/suggest` request normalizes the query and executes up to four deterministic stages in `internal/engine/search.go`:

- **Stage 1 — Direct Prefix Lookup (`baseScore = 100.0`, `editDist = 0`)**:
  Queries Redis `ZRANGEBYLEX` (up to 50 entries) on `tenant:{dataset_id}:zset` for the normalized query, space-compacted query (`"macbookpro"`), Latin diacritic-folded query, and Arabic/Persian normalized query. Simultaneously performs an in-memory binary search over `SortedTokens` and `TopTokenPostings` (up to 100 candidates).
- **Stage 2 — Multi-Token Inverted Index (`baseScore = 80.0` exact / `78.0` single-token prefix fan-out, `editDist = 0`)**:
  Segments the query into UAX #29 word tokens. For single-token queries, retrieves `TopTokenPostings` (`80.0`) and expands up to 20 prefix-matching tokens (`78.0`). For multi-token queries, intersects `TokenMap` posting lists across all tokens (starting from the rarest token) and supports prefix completion on the final token (capped at 250 candidates). Matches tokens in any word order.
- **Stage 3 — SymSpell + True Damerau-Levenshtein Typo Resolution (`baseScore = 55.0`)**:
  Triggered when `typo_tolerance` is enabled (and not overridden to `false` by `?typo=false`) and either earlier stages returned 0 candidates or any query token was unmatched. Generates 1- and 2-rune deletion keys against `TypoMap` (using an 8-rune prefix window for tokens > 8 runes) and verifies true Damerau-Levenshtein edit distance (insertions, deletions, substitutions, and adjacent transpositions):
  - Token length **< 4 runes**: `0` edits allowed (exact/prefix only; prevents noisy matches on short tokens).
  - Token length **4–5 runes**: maximum edit distance **1**.
  - Token length **>= 6 runes**: maximum edit distance **2**.
  - Falls back to Latin diacritic folding if no edit-distance candidates match. Capped at 250 candidates.
- **Stage 4 — Progressive Right-to-Left Token Fallback (`baseScore = 25.0`)**:
  Triggered only when `len(candidates) == 0` and the query has multiple tokens (`len(qTokens) > 1`). Progressively drops trailing tokens from right to left (`qTokens[:k]`) and re-runs Stage 2 (plus Stage 3 if typo tolerance is enabled), stopping at the first token prefix `k` that returns candidates.

### Known Algorithmic Limitations
- **Short Token Typo Guardrail**: Words shorter than 4 Unicode runes (`< 4` characters) do not undergo fuzzy typo expansion in Stage 3; they match via prefix or exact token only.
- **Lexical Only (No Vector/Semantic Embeddings)**: TypoSuggest does not perform synonym or neural embedding search (`"laptop"` will not match `"notebook"` unless both words appear in the indexed text).
- **Deduplicated Phrase Output (No Per-Document ID in Suggestions)**: Because multiple ingested items and extracted n-grams merge into deduplicated `PhraseEntry` records, `Suggestion` objects return `text`, `group`, `score`, and `frequency` (`RefCount`), not a raw document `id`.
- **Result Limit Cap**: `limit` on `GET /v1/suggest` defaults to `8` and is hard-capped at `20` (`1..20`).
- **Query Length Truncation**: Queries exceeding `100` bytes are truncated on a valid UTF-8 rune boundary at normalization time (`MaxQueryLength = 100`).

---

## 5. Ranking & Scoring Behavior

Each candidate `PhraseEntry` is scored deterministically in `internal/engine/search.go`:

$$\text{score} = \text{stageBaseScore} + 25.0 \times \log_{10}(1 + \text{RefCount}) + \text{prefixBonus} - (18.0 \times \text{editDist}) - (0.15 \times \text{runeLength}(\text{DisplayText}))$$

Where:
- `stageBaseScore`: `100.0` (Stage 1), `80.0` (Stage 2 exact), `78.0` (Stage 2 single-token prefix expansion), `55.0` (Stage 3 typo match), or `25.0` (Stage 4 fallback).
- `25.0 * log10(1 + RefCount)`: Logarithmic popularity boost from document deduplication frequency (`frequency` in response).
- `prefixBonus`: `+30.0` if the normalized candidate phrase starts with the normalized query string (`strings.HasPrefix`), else `0.0`.
- `18.0 * editDist`: Linear penalty per Damerau-Levenshtein edit (`0`, `1`, or `2`).
- `0.15 * runeLength(DisplayText)`: Conciseness tie-breaker favoring shorter, cleaner completions.

**Sorting & Post-Processing**:
1. Candidates are sorted by `score` descending, with ties broken by `RefCount` (`frequency`) descending.
2. Scores are rounded to 1 decimal place (`math.Round(score * 10) / 10`).
3. The top pool (`max(limit * 4, 40)`) undergoes RefCount-aware sub-phrase pruning and `GroupCaps` filtering before returning at most `limit` items.

---

## 6. Architecture Overview

- **HTTP Router & Middleware (`internal/api`)**: Built on `go-chi/chi/v5` with `HEAD` request support (`middleware.GetHead`), panic recovery, structured request logging, CORS (`*`, `Access-Control-Max-Age: 86400`), token-bucket rate limiting (`golang.org/x/time/rate`), dual authentication (`DualAuthMiddleware`), and subscription quota enforcement (`QuotaTracker`).
- **In-Memory Engine (`internal/engine`)**: Per-dataset `DatasetIndex` protected by `sync.RWMutex`. Stores `Phrases` (`[]PhraseEntry`), `PhraseLookup` (`map[string]uint32`), `TokenMap` (`map[string][]uint32`), `TopTokenPostings` (top-64 pre-sorted phrase IDs per token), `SortedTokens` (`[]string` for `O(log V)` prefix binary search), `TypoMap` (`map[string][]string`), and `DocHashes` (document tracking for deletion/update).
- **Redis Acceleration Layer**: Optional sorted-set (`ZSET`) lexicographical index (`tenant:{dataset_id}:zset`) for fast Stage 1 prefix lookups; gracefully degrades to in-memory `SortedTokens` binary search if Redis is unavailable.
- **PostgreSQL Control Plane (`internal/db`)**: Stores tenants, datasets, SHA-256 hashed API keys, verification codes, and support messages.
- **Zstandard Snapshot Store (`./snapshots/{dataset_id}.idx`)**: Gob-encoded + Zstandard-compressed index snapshots written atomically via `.tmp` rename after mutations.

---

## 7. Authentication & Security

TypoSuggest enforces strict role and header rules across **REST `/v1/*` endpoints** and the **MCP `/mcp` endpoint**:

### API Key Types
| Key Prefix | Role | Permitted Operations |
| :--- | :--- | :--- |
| `ac_pub_...` | `public` | **Read-only search**: Permitted ONLY on `GET /v1/suggest` (and read-only MCP tools `suggest`, `list_datasets`, `get_dataset_stats`). Rejected with `HTTP 403 Forbidden` on all write, dataset management, and dashboard endpoints. |
| `ac_sec_...` | `secret` | **Full read/write access**: Dataset creation, deletion, reset, item ingestion/update/delete, dashboard overview/key management, search, and all 6 MCP tools. Must remain server-side only. |

### How to Pass Credentials on REST Endpoints (`/v1/*`)
- **Preferred Method — `X-API-Key` Header**:
  - Pass `X-API-Key: ac_pub_...` (for `GET /v1/suggest`) or `X-API-Key: ac_sec_...` (for backend write/management calls).
- **Optional Browser Method — `?key=ac_pub_...` Query Parameter (`GET /v1/suggest` only)**:
  - Passing `?key=ac_pub_...` (or `?api_key=ac_pub_...`) on `GET /v1/suggest` avoids sending custom HTTP headers so browsers treat the request as a CORS simple request and skip the `OPTIONS` preflight round-trip.
  - **Security Trade-Off**: URL query parameters can be recorded in reverse-proxy/CDN access logs, browser history, `Referer` headers, and monitoring tools. **Only use `?key=` with read-only public keys (`ac_pub_...`)** where the key is already intended for public client-side distribution. **Never pass secret keys (`ac_sec_...`) or JWT tokens in URL query strings.**
- **Dashboard Session JWT (`Authorization: Bearer <jwt>`)**:
  - Issued by `POST /v1/auth/register`, `POST /v1/auth/login`, or Google OAuth.
  - **Important**: On REST `/v1/*` routes, `Authorization: Bearer <token>` is parsed **strictly as a JWT** (`ValidateJWT`). Do **not** pass `ac_pub_...` or `ac_sec_...` inside `Authorization: Bearer` on REST `/v1/*` calls—use `X-API-Key`.

### How to Pass Credentials on MCP (`POST /mcp`)
- The `/mcp` endpoint accepts API keys or JWTs via headers (`X-API-Key: ac_sec_...` or `Authorization: Bearer ac_sec_...`). While `?api_key=` / `?key=` is also supported for MCP clients that cannot set custom headers, prefer header-based authentication (`X-API-Key`) whenever possible so secret keys are not logged in URLs.

---

## 8. Free-Tier (Beta) Limits & Rate Limits

### Default Tenant Quotas (Self-Serve Beta Plan)
| Resource Quota | Default Beta Limit | HTTP Status When Exceeded |
| :--- | :--- | :--- |
| **Max Datasets** | `2` datasets | `402 Payment Required` (`dataset limit reached (2/2)...`) |
| **Max Rows (Ingested Documents)** | `500` total rows across tenant | `402 Payment Required` (`row limit exceeded: adding N rows would exceed your plan limit of 500...`) |
| **Max Monthly Searches** | `10,000` queries / month | `402 Payment Required` (`monthly search quota of 10000 queries exceeded...`) |

> **Note on Quotas vs. Engine Benchmarks**: The `2 datasets / 500 rows / 10,000 monthly searches` limits above are application-layer tenant quotas enforced by `QuotaTracker` for self-serve Beta accounts. The engine's 100,000-row and 10,000,000-row stress tests (Section 15) run against an internal stress-test configuration to benchmark algorithmic capacity for custom/enterprise tiers.

### Per-Endpoint Token-Bucket Rate Limits
| Endpoint Group | Rate Limit | Burst | Keyed By | Status on Limit |
| :--- | :--- | :--- | :--- | :--- |
| `GET /v1/suggest` | `60 req/sec` | `120` | API Key (or IP fallback) | `429 Too Many Requests` (`Retry-After: 1`) |
| `POST /items:add`, `POST /items:delete`, `PUT /items:update` | `10 req/sec` | `20` | Tenant ID (or IP fallback) | `429 Too Many Requests` (`Retry-After: 1`) |
| `/v1/auth/register`, `/v1/auth/login`, `/v1/auth/verify-email`, `/v1/admin/login`, `/v1/contact` | `15 req/min` (`0.25/s`) | `10` | Client IP | `429 Too Many Requests` (`Retry-After: 1`) |

- **Batch Ingestion Limits**: Maximum `25,000` items (`10 MB` body) per `POST /items:add` or `POST /items:delete` call.

---

## 9. Complete API Endpoint Index

### Public & System Endpoints (No Authentication Required)
| Method | Path | Description |
| :--- | :--- | :--- |
| `GET` | `/health` | Health check for PostgreSQL, Redis, and snapshot disk writeability (5s cached, sanitized details; `200 OK` or `503`). |
| `GET` | `/v1/demo-session` | Returns or provisions the live demo `tenant_id`, `dataset_id`, and a read-only `ac_pub_...` `api_key`. |
| `GET` | `/v1/datasets/{dataset_id}/stats` | Real-time metrics (`total_rows_ingested`, `active_unique_phrases`, `unique_dictionary_tokens`, `symspell_keys`, `queries_served`, `avg_query_latency_ms`, `snapshot_size_bytes`). |
| `POST` | `/v1/contact` | Submits a support or custom-plan inquiry (`name`, `email`, `category`, `message`). |
| `GET` | `/llms.txt` | Concise AI-readable product & API reference (this document). Supports `GET` and `HEAD`. |
| `GET` | `/llms-full.txt` | Comprehensive AI technical reference (schemas, internals, normalization, SDKs). Supports `GET` and `HEAD`. |
| `GET` | `/openapi.yaml` | OpenAPI 3.1.0 specification in YAML format. Supports `GET` and `HEAD`. |

### Core Autocomplete & Dataset API (`/v1/*` — Protected by `DualAuthMiddleware`)
| Method | Path | Required Role | Description |
| :--- | :--- | :--- | :--- |
| `GET` | `/v1/suggest` | `public` (`ac_pub_...`) or `secret` (`ac_sec_...` / JWT) | Execute 4-stage typo-tolerant autocomplete (`?dataset_id=...&q=...&limit=8&typo=true`). |
| `POST` | `/v1/datasets` | `secret` (`ac_sec_...` or Bearer JWT) | Create a new dataset (`name`, `typo_tolerance`, `boilerplate_regex`, `max_ngram_words`, `group_caps`). Returns `{"dataset": {...}}`. |
| `DELETE` | `/v1/datasets/{dataset_id}` | `secret` (`ac_sec_...` or Bearer JWT) | Permanently delete a dataset from PostgreSQL, RAM, Redis, and disk snapshots. Returns `{"status": "deleted", "dataset_id": "..."}`. |
| `POST` | `/v1/datasets/{dataset_id}/reset` | `secret` (`ac_sec_...` or Bearer JWT) | Wipe all indexed items, Redis ZSET keys, and `.idx` snapshot while keeping the dataset record. |
| `POST` | `/v1/datasets/{dataset_id}/items:add` | `secret` (`ac_sec_...` or Bearer JWT) | Batch ingest items (`{"items": [{"id", "text", "group", "count"}], "texts": [...]}`). |
| `POST` | `/v1/datasets/{dataset_id}/items:delete` | `secret` (`ac_sec_...` or Bearer JWT) | Batch delete items by `ids` and/or `texts` (`{"ids": [...], "texts": [...]}`). |
| `PUT` | `/v1/datasets/{dataset_id}/items:update` | `secret` (`ac_sec_...` or Bearer JWT) | Atomically update an existing item (`{"id": "...", "old_text": "...", "new_text": "...", "group": "..."}`). |
| `POST` | `/v1/datasets/{dataset_id}/seed` | `secret` (+ Founder Admin in prod) | Seed synthetic multi-domain rows (`{"count": 500, "reset": false}`). Disabled for regular tenants in production. |

### Authentication & Tenant Dashboard Endpoints
| Method | Path | Auth Required | Description |
| :--- | :--- | :--- | :--- |
| `POST` | `/v1/auth/register` | Public | Register a new tenant (`name`, `email`, `password`); provisions starter dataset, `secret_key`, `public_key`, and 30-day JWT `token`. |
| `POST` | `/v1/auth/login` | Public | Authenticate with `email` and `password` to receive a 30-day Bearer JWT `token` and `tenant` summary. |
| `POST` | `/v1/auth/verify-email` | Bearer JWT or `ac_sec_...` | Verify 6-digit email OTP (`{"code": "123456"}`). |
| `POST` | `/v1/auth/resend-code` | Bearer JWT or `ac_sec_...` | Generate and resend a fresh 6-digit verification email code. |
| `GET` | `/v1/auth/google/status` | Public | Returns `{"enabled": true|false}` indicating whether Google OAuth is configured. |
| `GET` | `/v1/auth/google/login` | Public | Initiates Google OAuth 2.0 login redirect flow (`GET /v1/auth/google/callback` handles OAuth return). |
| `GET` | `/v1/dashboard/overview` | Bearer JWT or `ac_sec_...` | Returns tenant profile, quota usage, `datasets` list (with live metrics), and `api_keys`. |
| `POST` | `/v1/dashboard/keys` | Bearer JWT or `ac_sec_...` | Issue a new API key (`{"name": "...", "role": "public"|"secret"}`); returns `raw_key` once. |
| `DELETE` | `/v1/dashboard/keys/{key_id}` | Bearer JWT or `ac_sec_...` | Revoke an API key by UUID (`{"status": "revoked", "key_id": "..."}`). |
| `POST` | `/v1/tenants` | `X-Master-Admin-Key` in prod | Bootstrap tenant creation endpoint (`{"name": "..."}`). |
| `POST` | `/v1/billing/webhook` | `X-Billing-Secret` | Subscription lifecycle webhook receiver. |

### Model Context Protocol (MCP) Endpoint
| Method | Path | Auth Required | Description |
| :--- | :--- | :--- | :--- |
| `POST` | `/mcp` | `X-API-Key` or `Bearer` on `tools/call` | JSON-RPC 2.0 MCP server (`2024-11-05`) exposing `initialize`, `ping`, `tools/list`, and `tools/call`. |
| `GET` | `/mcp` | Public | Discovery metadata returning server capabilities and authentication instructions. |

---

## 10. Request & Response Examples

### A. Search Suggestions (`GET /v1/suggest`)

**Query Parameters**:
- `dataset_id` (string, **required**): Target dataset UUID.
- `q` (string, optional; alias `query`): Search input string (up to 100 bytes). Empty string returns top phrases by popularity.
- `limit` (integer, optional): Number of suggestions to return. Default: `8`, minimum: `1`, maximum: `20`.
- `typo` (boolean, optional): Per-query override for typo tolerance (`true`/`1` or `false`/`0`). Defaults to dataset's `typo_tolerance` setting.

**Request**:
```bash
curl -sS "https://typosuggest.pial.digital/v1/suggest?dataset_id=550e8400-e29b-41d4-a716-446655440000&q=wrodpres&limit=5" \
  -H "X-API-Key: ac_pub_your_public_key_here"
```

**Response (`200 OK`)**:
```json
{
  "query": "wrodpres",
  "took_ms": 0.142,
  "suggestions": [
    {
      "text": "modern wordpress website",
      "group": "gig",
      "score": 97.4,
      "frequency": 42
    },
    {
      "text": "wordpress website for small business",
      "group": "gig",
      "score": 88.1,
      "frequency": 19
    }
  ]
}
```
> **Schema Note**: Each element in `suggestions` contains strictly `text` (string), `group` (string), `score` (float64), and `frequency` (uint32).

### B. Create a Dataset (`POST /v1/datasets`)

**Request**:
```bash
curl -sS -X POST "https://typosuggest.pial.digital/v1/datasets" \
  -H "X-API-Key: ac_sec_your_secret_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production Store Catalog",
    "typo_tolerance": true,
    "boilerplate_regex": "(?i)^i\\s+will\\s+(be\\s+(your\\s+)?|do\\s+)?",
    "max_ngram_words": 4,
    "group_caps": {
      "category": 2
    }
  }'
```

**Response (`201 Created`)**:
```json
{
  "dataset": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "tenant_id": "2c9d8f10-91b2-4a88-b822-112233445566",
    "name": "Production Store Catalog",
    "typo_tolerance": true,
    "boilerplate_regex": "(?i)^i\\s+will\\s+(be\\s+(your\\s+)?|do\\s+)?",
    "group_caps": {
      "category": 2
    },
    "max_ngram_words": 4,
    "created_at": "2026-10-31T08:00:00Z",
    "updated_at": "2026-10-31T08:00:00Z"
  }
}
```

### C. Batch Ingest Items (`POST /v1/datasets/{dataset_id}/items:add`)

**Request**:
```bash
curl -sS -X POST "https://typosuggest.pial.digital/v1/datasets/550e8400-e29b-41d4-a716-446655440000/items:add" \
  -H "X-API-Key: ac_sec_your_secret_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {
        "id": "sku-101",
        "text": "Apple MacBook Pro 16 inch M3 Max",
        "group": "product",
        "count": 25
      },
      {
        "id": "sku-102",
        "text": "Samsung Galaxy S24 Ultra 512GB",
        "group": "product",
        "count": 15
      }
    ],
    "texts": [
      "Laptops & Computers"
    ]
  }'
```
> **Schema Note**: `IngestRequest` accepts `items` (array of `{"id", "text", "group", "count"}`) and/or `texts` (array of plain strings, which default to `group: "item"` and `count: 1`).

**Response (`200 OK`)**:
```json
{
  "added": 3,
  "skipped_duplicates": 0,
  "active_phrases": 24,
  "took_ms": 1.18
}
```

---

## 11. Error Handling

All API errors return a JSON payload with `Content-Type: application/json`:
```json
{
  "error": "Human-readable description of what went wrong"
}
```

| HTTP Status | Meaning & Common Causes |
| :--- | :--- |
| `400 Bad Request` | Missing required parameter (`dataset_id`, `name`), empty `items`/`texts`, invalid JSON, or batch > 25,000 items. |
| `401 Unauthorized` | Missing or invalid `X-API-Key` / `Authorization: Bearer <jwt>`, or passing an `ac_pub_...` key inside `Authorization: Bearer` on a REST `/v1/*` route. |
| `402 Payment Required` | Tenant quota exceeded (more than `2` datasets, `500` total rows, or `10,000` monthly searches on Beta plan, or paused subscription). |
| `403 Forbidden` | Public read-only key (`ac_pub_...`) used on a write/management endpoint, cross-tenant dataset access (`ErrTenantMismatch`), or calling `/seed` in production without founder admin auth. |
| `404 Not Found` | `dataset_id` does not exist. |
| `409 Conflict` | Email address already registered on `POST /v1/auth/register`. |
| `429 Too Many Requests` | Token-bucket rate limit exceeded (`Retry-After: 1` header included). |
| `500 Internal Server Error` | Unexpected storage or serialization failure. |
| `503 Service Unavailable` | Returned by `GET /health` when PostgreSQL, Redis, or snapshot disk check fails. |

---

## 12. Dataset Management Lifecycle

1. **Create**: `POST /v1/datasets` creates metadata in PostgreSQL and registers an empty `DatasetIndex` in RAM.
2. **Ingest**: `POST /v1/datasets/{dataset_id}/items:add` normalizes text, strips boilerplate, extracts 1..`max_ngram_words` leading n-grams, updates `PhraseLookup`, `TokenMap`, `SortedTokens`, and `TypoMap`, pipelines prefix keys to Redis `ZSET`, and triggers an async Zstandard `.idx` snapshot save.
3. **Update**: `PUT /v1/datasets/{dataset_id}/items:update` takes `{"id": "...", "old_text": "...", "new_text": "...", "group": "..."}`, decrements `RefCount` for phrases derived from `old_text`, and indexes `new_text` under a single write lock.
4. **Delete Items**: `POST /v1/datasets/{dataset_id}/items:delete` takes `{"ids": [...], "texts": [...]}`, decrements `RefCount` on every associated phrase, and purges phrases from `PhraseLookup` and Redis `ZSET` when `RefCount` reaches `0`.
5. **Reset vs. Delete Dataset**:
   - `POST /v1/datasets/{dataset_id}/reset` clears all items, phrases, Redis keys, and `.idx` snapshot file, keeping the `dataset_id` and configuration intact.
   - `DELETE /v1/datasets/{dataset_id}` removes the dataset row from PostgreSQL along with all in-memory, Redis, and disk state.

---

## 13. MCP Integration (`POST /mcp`)

TypoSuggest includes a built-in Model Context Protocol (MCP) server implementing protocol version `2024-11-05` over stateless HTTP JSON-RPC 2.0 at `https://typosuggest.pial.digital/mcp`.

### Available MCP Tools
| Tool Name | Required Key Scope | Arguments | Description |
| :--- | :--- | :--- | :--- |
| `suggest` | `public` or `secret` | `dataset_id` (required), `query` (required), `limit` (1..20, default 8), `typo_tolerance` (bool) | Run sub-millisecond typo-tolerant autocomplete search. |
| `list_datasets` | `public` or `secret` | *(none)* | List all datasets for the authenticated tenant with active phrase counts. |
| `get_dataset_stats` | `public` or `secret` | `dataset_id` (required) | Inspect rows ingested, unique phrases, tokens, SymSpell keys, latency, and snapshot size. |
| `create_dataset` | `secret` (`ac_sec_...`) | `name` (required), `typo_tolerance` (bool or int, default enabled), `max_ngram_words` (default 4) | Create a new autocomplete dataset. |
| `add_items` | `secret` (`ac_sec_...`) | `dataset_id` (required), `items` (required array of `{id, text, group, count}`) | Batch ingest up to 25,000 items into a dataset. |
| `delete_dataset` | `secret` (`ac_sec_...`) | `dataset_id` (required) | Permanently delete a dataset and its snapshots. |

### Claude Desktop / Cursor / Windsurf Configuration
```json
{
  "mcpServers": {
    "typosuggest": {
      "url": "https://typosuggest.pial.digital/mcp",
      "headers": {
        "X-API-Key": "ac_sec_your_secret_key_here"
      }
    }
  }
}
```

---

## 14. SDK & Frontend Integration Examples

### React Typeahead Component (With `AbortController`, Stale-Response Guard & Error Handling)
```tsx
import React, { useState, useEffect, useRef } from 'react';

const API_BASE = 'https://typosuggest.pial.digital';
const DATASET_ID = '550e8400-e29b-41d4-a716-446655440000';
const PUBLIC_KEY = 'ac_pub_your_public_key_here'; // Read-only public key (safe for browser use)

interface Suggestion {
  text: string;
  group: string;
  score: number;
  frequency: number;
}

export function AutocompleteSearch() {
  const [query, setQuery] = useState('');
  const [suggestions, setSuggestions] = useState<Suggestion[]>([]);
  const [error, setError] = useState<string | null>(null);
  const requestSeq = useRef(0);

  useEffect(() => {
    const trimmed = query.trim();
    if (!trimmed) {
      setSuggestions([]);
      setError(null);
      return;
    }

    const seq = ++requestSeq.current;
    const controller = new AbortController();

    const timer = setTimeout(async () => {
      try {
        const url = new URL(`${API_BASE}/v1/suggest`);
        url.searchParams.set('dataset_id', DATASET_ID);
        url.searchParams.set('q', trimmed);
        url.searchParams.set('limit', '8');
        // Note: Passing ?key=ac_pub_... avoids CORS preflight, or pass via headers: { 'X-API-Key': PUBLIC_KEY }
        url.searchParams.set('key', PUBLIC_KEY);

        const res = await fetch(url.toString(), { signal: controller.signal });
        if (!res.ok) {
          const errBody = await res.json().catch(() => ({}));
          throw new Error(errBody.error || `HTTP ${res.status}`);
        }

        const data = await res.json();
        if (seq === requestSeq.current) {
          setSuggestions(Array.isArray(data.suggestions) ? data.suggestions : []);
          setError(null);
        }
      } catch (err: any) {
        if (err?.name === 'AbortError') return;
        if (seq === requestSeq.current) {
          setError(err.message || 'Search request failed');
        }
      }
    }, 120);

    return () => {
      clearTimeout(timer);
      controller.abort();
    };
  }, [query]);

  return (
    <div>
      <input
        type="search"
        value={query}
        onChange={(e) => setQuery(e.target.value)}
        placeholder="Search products..."
        aria-label="Search"
      />
      {error && <div role="alert">{error}</div>}
      <ul>
        {suggestions.map((item) => (
          <li key={item.text}>
            <span>{item.text}</span> <small>({item.group})</small>
          </li>
        ))}
      </ul>
    </div>
  );
}
```

### Node.js Server-Side Ingestion (`ac_sec_...`)
```javascript
const API_BASE = 'https://typosuggest.pial.digital';
const SECRET_KEY = process.env.TYPOSUGGEST_SECRET_KEY; // ac_sec_... (never expose in browser or URLs)

async function ingestCatalog(datasetId, items) {
  const res = await fetch(`${API_BASE}/v1/datasets/${encodeURIComponent(datasetId)}/items:add`, {
    method: 'POST',
    headers: {
      'X-API-Key': SECRET_KEY,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ items }),
    signal: AbortSignal.timeout(10000),
  });

  const payload = await res.json().catch(() => ({}));
  if (!res.ok) {
    throw new Error(`TypoSuggest ingest failed (${res.status}): ${payload.error || res.statusText}`);
  }
  return payload; // { added, skipped_duplicates, active_phrases, took_ms }
}
```

### Python Client (With Timeout & Status Validation)
```python
import os
import requests

API_BASE = "https://typosuggest.pial.digital"
PUBLIC_KEY = os.environ["TYPOSUGGEST_PUBLIC_KEY"]  # ac_pub_...

def suggest(dataset_id: str, query: str, limit: int = 8) -> list[dict]:
    resp = requests.get(
        f"{API_BASE}/v1/suggest",
        params={"dataset_id": dataset_id, "q": query, "limit": limit},
        headers={"X-API-Key": PUBLIC_KEY},
        timeout=5.0,
    )
    if not resp.ok:
        detail = resp.json().get("error", resp.text) if resp.headers.get("content-type", "").startswith("application/json") else resp.text
        raise RuntimeError(f"TypoSuggest error {resp.status_code}: {detail}")
    data = resp.json()
    return data.get("suggestions", [])
```

### Go Client (`net/http` With Explicit Timeout)
```go
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"net/url"
	"time"
)

type Suggestion struct {
	Text      string  `json:"text"`
	Group     string  `json:"group"`
	Score     float64 `json:"score"`
	Frequency uint32  `json:"frequency"`
}

type SuggestResponse struct {
	Query       string       `json:"query"`
	TookMs      float64      `json:"took_ms"`
	Suggestions []Suggestion `json:"suggestions"`
}

func FetchSuggestions(ctx context.Context, datasetID, query, pubKey string) (*SuggestResponse, error) {
	client := &http.Client{Timeout: 5 * time.Second}
	u, _ := url.Parse("https://typosuggest.pial.digital/v1/suggest")
	q := u.Query()
	q.Set("dataset_id", datasetID)
	q.Set("q", query)
	q.Set("limit", "8")
	u.RawQuery = q.Encode()

	req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
	if err != nil {
		return nil, err
	}
	req.Header.Set("X-API-Key", pubKey)

	resp, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		var apiErr struct {
			Error string `json:"error"`
		}
		_ = json.NewDecoder(resp.Body).Decode(&apiErr)
		return nil, fmt.Errorf("typosuggest status %d: %s", resp.StatusCode, apiErr.Error)
	}

	var out SuggestResponse
	if err := json.NewDecoder(resp.Body).Decode(&out); err != nil {
		return nil, err
	}
	return &out, nil
}
```

---

## 15. Performance Benchmark Methodology

Benchmarks are defined in `internal/engine/benchmark_test.go` and run directly against the in-memory `DatasetIndex` engine (`go test -v -run TestStress100kDatasetMetrics ./internal/engine` and `go test -v -run TestStress10MillionRows ./internal/engine`).

> **Scope & Quota Distinction**: These benchmarks use an internal stress-test configuration that invokes `DatasetIndex.AddItems` and `DatasetIndex.Suggest` directly in Go, bypassing the HTTP API rate limiters and the self-serve Beta customer quota (`2 datasets / 500 rows / 10,000 monthly searches`). They measure core engine scalability for high-volume custom/enterprise deployments.

1. **100,000-Row Multi-Domain Benchmark (`TestStress100kDatasetMetrics`)**:
   - **Total Row Occurrences vs. Unique Indexed Phrases**: Generates `100,000` raw input items (`rand.NewSource(42)`) across 4 domains (`55,000` Marketplace gigs, `25,000` E-commerce products, `12,000` Pharmacy items, and `8,000` Bengali multilingual items). After boilerplate stripping, n-gram extraction (`max_ngram_words: 4`), and phrase deduplication, the index holds `~38,000–45,000` unique active phrases (`active_phrases` in `PhraseLookup`) with accumulated `RefCount` popularity weights.
   - **Query Workload**: `5,000` queries cycling across 4 categories: Exact Prefix (`"wordp"`, `"macb"`), Multi-Word Out-of-Order (`"website wordpress"`, `"m3 macbook"`), 1–2 Edit Distance Typos (`"wrodpres"`, `"paracitaml"`), and Bengali Unicode (`"ওয়েবসাইট"`). Each query is measured over `50` calibrated repetitions to eliminate OS timer quantization noise.
   - **Measured Engine Metrics (In-Memory Execution)**:
     - **Median (`p50`) Match Latency**: `~0.03 ms – 0.09 ms` (`30–90 µs`)
     - **`p95` Match Latency**: `~0.10 ms – 0.25 ms`
     - **`p99` Match Latency**: `< 0.45 ms`
     - **Ingestion Throughput**: `~60,000+ rows/sec`
     - **Heap Footprint**: `~0.31 KB` per ingested row (`~31 MB` post-GC for 100k rows)
2. **10,000,000-Row Zipfian Scale Benchmark (`TestStress10MillionRows`)**:
   - **Total Row Occurrences vs. Unique Indexed Phrases**: Synthesizes `100,000` unique combinatorial sentence templates whose Zipfian `count` weights sum to `10,000,000` total row occurrences (`SkipDocHashTracking: true`), producing `~150,000–250,000` unique deduplicated phrases and n-grams in `PhraseLookup`.
   - Verifies `< 0.50 ms` average in-memory query execution across `5,000` queries at 10M total row occurrences.
   - *Network Note*: `took_ms` in API responses measures server-side engine execution time in milliseconds; end-to-end client latency includes network round-trip time (RTT) to `ap-southeast-2`.

---

## 16. Security Recommendations

1. **Never Expose `ac_sec_...` Keys in Client Code or URLs**: Secret keys (`role: "secret"`) can create/delete datasets, mutate items, and manage keys. Keep `ac_sec_...` keys strictly in backend environment variables or secret managers, and always send them via the `X-API-Key` HTTP header.
2. **Use `ac_pub_...` Keys in Browser / Mobile Apps**: Public keys (`role: "public"`) are cryptographically verified by SHA-256 hash and restricted by `DualAuthMiddleware` exclusively to `GET /v1/suggest`.
3. **Understand the `?key=ac_pub_...` Query-String Trade-Off**:
   - Passing `?key=ac_pub_...` in the URL query string on `GET /v1/suggest` avoids browser CORS `OPTIONS` preflight requests, cutting per-keystroke network round-trips in half.
   - However, URL query parameters may be recorded in web server access logs, CDN logs, browser history, `Referer` headers, and APM tools.
   - Only pass **public, read-only keys (`ac_pub_...`)** in query parameters; if your environment requires keeping even public keys out of URL logs, pass `X-API-Key: ac_pub_...` in the request header (browsers cache the CORS preflight for 24 hours via `Access-Control-Max-Age: 86400`).
4. **Rotate Keys Safely**: Issue a replacement key via `POST /v1/dashboard/keys` before revoking the old key with `DELETE /v1/dashboard/keys/{key_id}`.

---

## 17. Troubleshooting

- **`401 Unauthorized` when calling `/v1/suggest` with `Authorization: Bearer ac_pub_...`**:
  REST `/v1/*` endpoints treat `Authorization: Bearer` exclusively as a Dashboard JWT. Pass API keys via `X-API-Key: ac_pub_...` (or `?key=ac_pub_...` for public keys).
- **`403 Forbidden: public search-only key cannot perform write operations`**:
  You used an `ac_pub_...` key on `POST /v1/datasets` or `POST /items:add`. Switch to your `ac_sec_...` secret key via the `X-API-Key` header.
- **`402 Payment Required` during dataset creation or ingestion**:
  Your tenant reached the Beta quota (`2` datasets, `500` total rows, or `10,000` monthly searches). Delete unused datasets/items or contact support via `POST /v1/contact` to raise your plan limits.
- **Short 2–3 letter typo queries return empty `suggestions`**:
  By design, Stage 3 fuzzy matching requires tokens to be `>= 4` runes (`4–5` runes = 1 edit; `>= 6` runes = 2 edits) to avoid false-positive noise on short prefixes.

---

## 18. Documentation Links

- **Landing Page & Live Playground**: https://typosuggest.pial.digital/
- **Interactive Developer Docs**: https://typosuggest.pial.digital/docs
- **Concise LLM Index (`llms.txt`)**: https://typosuggest.pial.digital/llms.txt
- **Complete Technical Reference (`llms-full.txt`)**: https://typosuggest.pial.digital/llms-full.txt
- **OpenAPI 3.1 Specification (`openapi.yaml`)**: https://typosuggest.pial.digital/openapi.yaml
- **MCP Endpoint Discovery**: https://typosuggest.pial.digital/mcp