api: YAML JSON TOON Database
version: 1.0.0
format: json
dataset:
  id: 1011
  slug: unicode-text-handling
  title: Unicode Text Handling Reference
  description: "Text boundaries, normalization, case handling, and encoding decisions for multilingual applications."
  summary: "Text boundaries, normalization, case handling, and encoding decisions for multilingual applications. Each row includes a practical application and a failure to check before using it."
  use_when[2]: Build search that handles composed and decomposed text.,Choose the unit for text length or truncation.
  caveats[2]: Preserve original text when using a lossy search transform.,Identifier security requires more than normalization.
  related[2]: date-time-interchange,json-schema-keywords
  category: Programming Languages
  category_slug: programming-languages
  tags: "programming-languages,unicode text handling,reference,engineering"
  view_count: 0
  created_at: 1791798640
  updated_at: 1791798640
  reviewed_at: 2026-10-12
  sources[3]{title,url}:
    Unicode normalization standard,"https://www.unicode.org/reports/tr15/"
    Unicode text segmentation,"https://www.unicode.org/reports/tr29/"
    Unicode security mechanisms,"https://www.unicode.org/reports/tr39/"
  reading[1]: unicode-normalization-for-search-and-identifiers
  provenance: "Original editorial reference notes by YJTOON, informed by the linked primary documentation. External specifications retain their own licenses."
  entry_count: 10
  fields[4]{section,field,type,example}:
    data,concept,string,UTF-8 bytes
    data,example,string,é occupies two UTF-8 bytes
    data,use,string,Measure network and file size.
    data,pitfall,string,A byte count is not a character count.
data[10]{concept,example,use,pitfall}:
  UTF-8 bytes,é occupies two UTF-8 bytes,Measure network and file size.,A byte count is not a character count.
  Code point,U+00E9 is LATIN SMALL LETTER E WITH ACUTE,Identify a Unicode scalar value.,A visible character can contain several code points.
  Grapheme cluster,e plus combining acute can display as one character,Move a caret or truncate visible text.,Code-point slicing can split a visible character.
  NFC,Compose canonically equivalent sequences where possible,Normalize text used for ordinary comparison.,It does not remove accents.
  NFD,Decompose canonical equivalents,Inspect combining marks or build a deliberate search transform.,Removing every mark changes meaning in many languages.
  NFKC,Apply compatibility decomposition and composition,Normalize selected identifiers under an explicit policy.,It can change distinctions the original text intended.
  Case folding,Treat selected uppercase and lowercase forms as equivalent,Build Unicode-aware caseless matching.,Locale and security requirements still matter.
  Turkish dotted I,"I, İ, ı, i require deliberate handling",Test Turkish search and casing.,ASCII-only lowercasing is not a complete locale policy.
  Bidirectional text,Arabic or Hebrew mixed with Latin identifiers,Render mixed-direction labels safely.,Invisible controls can obscure visual ordering.
  Confusable characters,Latin a and Cyrillic а can look alike,Review identifiers and suspicious domains.,Visual similarity does not imply string equality.