Unicode Text Handling Reference
Text boundaries, normalization, case handling, and encoding decisions for multilingual applications.
Text boundaries, normalization, case handling, and encoding decisions for multilingual applications. Each row includes a practical application and a failure to check before using it.
Use it when
- Build search that handles composed and decomposed text.
- Choose the unit for text length or truncation.
Watch out for
- Preserve original text when using a lossy search transform.
- Identifier security requires more than normalization.
The data
10 rows
| Concept | Example | Use | Pitfall |
|---|---|---|---|
| UTF-8 bytes | é occupies two UTF-8 bytes | Measure network and file size. | A byte count is not a character count. |
| Code point | U+00E9 is LATIN SMALL LETTER E WITH ACUTE | Identify a Unicode scalar value. | A visible character can contain several code points. |
| Grapheme cluster | e plus combining acute can display as one character | Move a caret or truncate visible text. | Code-point slicing can split a visible character. |
| NFC | Compose canonically equivalent sequences where possible | Normalize text used for ordinary comparison. | It does not remove accents. |
| NFD | Decompose canonical equivalents | Inspect combining marks or build a deliberate search transform. | Removing every mark changes meaning in many languages. |
| NFKC | Apply compatibility decomposition and composition | Normalize selected identifiers under an explicit policy. | It can change distinctions the original text intended. |
| Case folding | Treat selected uppercase and lowercase forms as equivalent | Build Unicode-aware caseless matching. | Locale and security requirements still matter. |
| Turkish dotted I | I, İ, ı, i require deliberate handling | Test Turkish search and casing. | ASCII-only lowercasing is not a complete locale policy. |
| Bidirectional text | Arabic or Hebrew mixed with Latin identifiers | Render mixed-direction labels safely. | Invisible controls can obscure visual ordering. |
| Confusable characters | Latin a and Cyrillic а can look alike | Review identifiers and suspicious domains. | Visual similarity does not imply string equality. |
Fields
The shape of this dataset, read from the data itself: every column, its type and a real value. Agents can use this to plan a query without fetching the whole document.
| Section | Field | Type | Example |
|---|---|---|---|
| data | concept | string | UTF-8 bytes |
| data | example | string | é occupies two UTF-8 bytes |
| data | use | string | Measure network and file size. |
| data | pitfall | string | A byte count is not a character count. |
Fetch the same bytes
The static files contain the same reference rows as the API, with no API rate limit. Live metadata, such as view counts, may differ. Use the API for content negotiation; use the static files for a cacheable snapshot.
curl "https://yjtoon.com/api/dataset/unicode-text-handling?format=toon"
const res = await fetch( "https://yjtoon.com/static-data/dataset/unicode-text-handling.toon" ); const toon = await res.text();
Rate limit: 120 requests per minute per IP, no key and no signup. API reference →
Sources and editorial notes
Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses.
Editorial review: . Source documents keep their own licenses.