Unicode normalization makes different sequences of code points comparable when they represent the same text—or intentionally folds compatibility characters into a more uniform form. This guide explains NFC, NFD, NFKC, and NFKD, shows how to normalize Unicode strings in JavaScript and server-side code, and provides a practical workflow for fixing failed equality checks, searches, and database uniqueness rules without damaging multilingual data.
Overview
Unicode text can look identical while being stored differently. For example, é may be represented as one precomposed code point, U+00E9, or as the letter e followed by a combining acute accent, U+0301. A user sees the same character, but a direct string comparison can treat the sequences as different.
Unicode normalization applies a defined transformation so equivalent text follows a consistent representation. The four standard forms are divided into two groups:
- Canonical normalization: NFC and NFD preserve the distinction between characters and compatibility variants.
- Compatibility normalization: NFKC and NFKD may fold formatting or presentation distinctions when they are considered compatible.
The letter C means “composed,” while D means “decomposed.” NFC generally combines sequences where a composed character exists. NFD generally separates characters into base characters and combining marks. NFKC and NFKD perform comparable operations but also apply compatibility mappings.
| Form | Category | Typical result | Common use |
|---|---|---|---|
| NFC | Canonical composed | Combines canonically equivalent sequences | General text storage and interchange |
| NFD | Canonical decomposed | Separates base characters and combining marks | Accent-aware processing and character analysis |
| NFKC | Compatibility composed | Combines characters and applies compatibility mappings | Search, identifiers, and controlled input comparison |
| NFKD | Compatibility decomposed | Decomposes characters and applies compatibility mappings | Analysis or transformations that need compatibility folding |
Normalization is not the same as encoding. UTF-8 and UTF-16 describe how code points are represented as bytes or code units; normalization changes the arrangement or selection of Unicode characters within the text. It also does not remove every visual ambiguity. Confusable characters from different scripts may remain distinct after normalization, so security checks require additional controls. See the guide to Unicode code points, UTF-8, and UTF-16 for the encoding distinction.
How to compare options
Choose a normalization form according to the meaning of the data, not simply because one form produces fewer code points. Ask four questions before adding normalization to a pipeline:
- Is the data user-facing text or an identifier? Names, addresses, and messages may need preservation of distinctions that are meaningful to users. Login names, tags, or search keys may need a separate comparison representation.
- Should compatibility distinctions remain visible? Some characters represent formatting or presentation choices, such as fullwidth Latin letters, circled letters, or superscript characters. NFKC and NFKD can fold these distinctions, which may be useful for matching but inappropriate for display.
- Does the application need composed or decomposed output? NFC is often a practical interchange choice. NFD can be useful when code needs to inspect or process combining marks individually.
- What does the receiving system expect? Search indexes, databases, APIs, and authentication components may have different assumptions. Define the boundary where normalization occurs and test that every component follows the same rule.
A safe design usually keeps the original input for display or audit purposes and creates a normalized value for a specific operation. Do not silently replace user text with a compatibility-folded version unless that transformation is part of the product’s documented behavior.
Feature-by-feature breakdown
NFC versus NFD
NFC and NFD preserve canonical equivalence. Consider these two representations:
// Visually equivalent examples
const composed = "é"; // U+00E9
const decomposed = "e\u0301"; // U+0065 followed by U+0301
composed === decomposed; // false
composed.normalize("NFC") === decomposed.normalize("NFC"); // true
composed.normalize("NFD") === decomposed.normalize("NFD"); // true
NFC is a reasonable default for normalized text exchanged between web applications, provided the application does not have a requirement for another form. NFD is useful for operations such as separating accents from base letters, but it should not be confused with removing accents. Decomposition exposes combining marks; an additional, deliberate filtering step would be required to remove them, and that step can change meaning in some languages.
NFKC versus NFKD
Compatibility forms go further. For example, a fullwidth Latin character such as A may normalize to the ordinary Latin letter A under compatibility normalization. A superscript two, ², may be mapped toward 2. These mappings can improve matching across imported data and user input, but they may erase distinctions that matter in a document, scientific expression, brand name, or legal record.
const fullwidth = "ABC";
fullwidth.normalize("NFC"); // "ABC"
fullwidth.normalize("NFKC"); // "ABC"
Use NFKC or NFKD for a comparison key only when compatibility folding is an explicit requirement. They are not universal “clean text” functions, and they do not solve case folding, transliteration, accent removal, or homoglyph detection by themselves.
JavaScript implementation
Modern JavaScript environments provide String.prototype.normalize(). The method accepts NFC, NFD, NFKC, or NFKD; NFC is used when no form is supplied.
function normalizedKey(value, form = "NFC") {
if (typeof value !== "string") {
throw new TypeError("Expected a string");
}
return value.normalize(form);
}
const displayValue = userInput; // Preserve original text
const searchValue = normalizedKey(userInput); // Use for a defined comparison
const compatibilityKey = normalizedKey(userInput, "NFKC");
Normalize at a clear boundary, such as request validation, import processing, or search-key generation. Avoid repeatedly normalizing inside unrelated business logic, and document whether a database column contains original text, NFC text, or a derived comparison key.
Server-side patterns
Most established language runtimes expose Unicode normalization through a standard library or a Unicode-aware library. In Python, for example:
import unicodedata
original = "e\u0301"
canonical = unicodedata.normalize("NFC", original)
compatibility = unicodedata.normalize("NFKC", original)
Equivalent APIs exist in other ecosystems, but names and supported Unicode versions can vary. Confirm the runtime documentation, test the exact characters used by your application, and keep client and server behavior aligned where normalized values cross an API boundary.
Database uniqueness and search
A database uniqueness constraint operates on the values it receives. If one request stores NFC and another stores NFD, a visually identical value may bypass a uniqueness check unless normalization occurs before comparison or the database has a suitable Unicode-aware collation and expression strategy.
A robust pattern is to store the original value separately from a normalized key:
display_name: the value retained for presentation.comparison_name: a documented normalized value used for lookup or uniqueness.
Choose the key’s form and any additional case-folding policy deliberately. Test existing records before adding a unique constraint, because normalization can reveal collisions that were previously hidden.
Best fit by scenario
| Scenario | Starting point | Important caution |
|---|---|---|
| General multilingual text interchange | NFC | Preserve the original where display fidelity matters. |
| Inspecting accents and combining marks | NFD | Combining marks affect length and rendering; do not assume one code point equals one visible character. |
| Search or matching imported identifiers | NFKC, often as a derived key | Compatibility folding may merge values users consider different. |
| Data analysis requiring decomposed compatibility forms | NFKD | Apply later filtering steps only when their linguistic effects are understood. |
| Passwords and security-sensitive secrets | Define a narrowly tested policy | Do not introduce normalization casually; preserve the authentication system’s documented behavior. |
Normalization also has limits in security work. It will not identify all homoglyphs, resolve bidirectional text behavior, or make arbitrary usernames safe. For those concerns, combine normalization with script policy, confusable detection, display controls, and careful identifier rules. The related guides on Unicode security and confusables and bidirectional Unicode text cover those separate problems.
When to revisit
Revisit your normalization policy whenever you add a new input source, change a runtime or database, introduce multilingual search, or discover a duplicate that looks different only in a developer tool. Unicode data and runtime implementations can evolve, so keep normalization behavior in regression tests rather than relying on assumptions.
When diagnosing a mismatch, use this workflow:
- Capture both strings without trimming, case conversion, or transliteration.
- Inspect code points, not only rendered text. Record hexadecimal values and combining marks.
- Compare the raw strings under NFC, NFD, NFKC, and NFKD to identify which distinction matters.
- Decide whether the application needs canonical equivalence or compatibility matching.
- Test the decision with real languages, combining sequences, emoji, punctuation, and imported records.
- Apply the chosen form at one documented boundary and add database or API tests for it.
Finally, keep normalization separate from other transformations. Case folding, whitespace handling, accent removal, URL processing, and confusable detection each have different risks. A small, explicit pipeline is easier to review than a generic “sanitize Unicode” step. If text still renders or compares unexpectedly, inspect grapheme clusters and rendering behavior as well; normalization alone cannot explain every visible difference. The guides to grapheme clusters and Unicode-aware string length and Unicode encoding errors are useful next checks.