Unicode Normalization for Search and Identifiers

Two strings can look identical and still compare differently. The accented letter é can be one code point or an e followed by a combining accent. Search, deduplication, and identifier rules need a deliberate policy for those differences.

First decide the job. Display should preserve the author’s text. Search may use a derived matching key. A security-sensitive identifier needs a narrower documented policy. Applying one aggressive normalization everywhere can damage names, change meaning, or create collisions that the application never checks.

Keep original text and a search key

const original = "café";
const canonical = original.normalize("NFC");
const searchKey = original.normalize("NFD")
  .replace(/[̀-ͯ]/g, "")
  .toLowerCase();

This JavaScript example illustrates a limited Latin-oriented accent-insensitive transform. It is not a universal multilingual search algorithm: the combining-mark range is incomplete, and removing marks can erase meaningful distinctions in other writing systems. Use it only under a defined product requirement, with language-specific examples.

Store and display the original. If the matching key collides, return all relevant results or apply a documented ranking rule. Do not silently merge records merely because a lossy search transform maps them to the same key.

Test the language you actually support

Turkish casing makes the ASCII assumption particularly visible: I, İ, ı, and i do not reduce to one simple English-style pair. Test names and queries in the target locale. Decide whether matching should be locale-sensitive, language-agnostic, or backed by a dedicated search system.

Length also needs a unit. UTF-8 bytes measure transfer size. Code points identify scalar values. Grapheme clusters more closely match the visible units a user expects a caret or truncation operation to preserve. A family emoji or a letter with combining marks can be split incorrectly by naïve slicing.

Normalization is not identifier security

Compatibility normalization can collapse distinctions that matter in a display name or code sample. Visually confusable characters from different scripts can remain distinct even after normalization. Review allowed scripts, collision handling, and invisible directional controls according to the identifier’s risk and audience.

A practical regression set should include composed and decomposed accents, Turkish names, non-Latin scripts, a combining sequence, an emoji cluster, and an intentionally confusable identifier. Assert both the search result and the preserved display value. That catches a search improvement that accidentally corrupts the content it was meant to find.

Reference tables to keep nearby

• Unicode Text Handling Reference — Text boundaries, normalization, case handling, and encoding decisions for multilingual applications.

Primary sources

These are the specifications and project documentation behind the examples. Check the relevant implementation and version before applying a pattern.

• Unicode normalization standard
• Unicode text segmentation
• Unicode security mechanisms