Unicode-Aware Search and Slug Generation: Case Folding, Transliteration, and Locale Rules
UnicodeSearchSEOInternationalizationText ProcessingWeb Development

Unicode-Aware Search and Slug Generation: Case Folding, Transliteration, and Locale Rules

UUnicode.live Editorial Team
2026-11-15
7 min read

A practical checklist for Unicode-aware search and multilingual URL slugs, covering case folding, locale rules, accents, transliteration, and testing.

Reliable multilingual search and URL slugs require more than converting text to lowercase. This checklist explains when to use Unicode case folding, locale-aware casing, normalization, accent handling, transliteration, and preserved original text, then turns those choices into repeatable slug and search-index workflows.

Overview

A name can have several valid representations in software. A user may search for İstanbul, istanbul, or a form produced by a keyboard with different casing rules. A title may contain accented Latin characters, Greek or Cyrillic letters, Arabic script, or CJK characters that should not be discarded simply because a slug generator expects ASCII.

The first principle is to separate display text, search text, and URL text. Preserve the original value for headings, labels, metadata, and editing. Build a search representation for matching. Build a URL representation only when a stable, readable route is needed. These representations can share normalization steps, but they should not be treated as interchangeable.

Unicode normalization can make canonically equivalent sequences comparable, such as a precomposed accented character and a base character followed by a combining mark. It does not decide whether accents are meaningful for a language, and it does not transliterate one writing system into another. For background, see Unicode Normalization Explained and the reference on code points, UTF-8, and UTF-16.

Decision table

TechniqueWhat it doesBest useWatch for
Unicode case foldingCreates a broadly comparable, case-insensitive form.General search matching and duplicate detection.It is not the same as display casing or every locale's lowercase rule.
Locale-aware lowercasingApplies casing behavior for a specified locale.User-facing transformations where language context is known.Results can differ by locale; do not use an unspecified locale for sensitive matching.
Accent handlingOptionally makes letters with and without diacritics match.Search recall when product requirements permit it.Accents can distinguish words or names. Make this a documented search policy.
TransliterationMaps text between scripts or into an approximate Latin representation.ASCII-oriented legacy systems, optional aliases, and some URL policies.It is language-dependent and lossy; different tools may produce different outputs.
Preserving original textStores and displays the user's exact Unicode text.Titles, names, labels, audit data, and canonical content.It does not by itself provide case-insensitive or accent-insensitive matching.

Checklist by scenario

  1. Store the original. Keep the source string unchanged, subject to your normal validation and storage rules. Do not replace a person's name or a page title with a transliterated value.
  2. Normalize consistently. Choose a normalization form for indexing and query processing, commonly NFC when preserving text structure is important. Apply the same choice to both indexed values and incoming queries.
  3. Choose casing deliberately. Use Unicode-aware case folding for a general comparison key, or use locale-aware casing when the application knows the language and that distinction matters.
  4. Decide whether accents should match. For example, a search product may choose to let an unaccented query find an accented title, while a dictionary or identity field may need to distinguish them. Keep the original field available for exact matching.
  5. Keep script information. Do not assume that a Latin transliteration is a replacement for Greek, Cyrillic, Arabic, or CJK text. It can be an additional searchable alias, not the only indexed value.

A practical search-index record can look like this:

{
  "display": "İstanbul Üniversitesi",
  "search": "normalized-and-case-folded-value",
  "aliases": ["optional transliteration or spelling variants"],
  "language": "tr",
  "exact": "original-value"
}

The exact field, language field, and search field serve different purposes. Your search engine's analyzer, tokenizer, stemming, and accent-folding settings must agree with the values your application sends. Test the complete path rather than testing only a helper function in isolation.

For internationalized URL slugs

  1. Generate the slug from the canonical title, not from an already transformed search key.
  2. Normalize the title before segmentation and separator handling.
  3. Choose between a Unicode slug and an ASCII transliterated slug based on routing, tooling, editorial, and compatibility requirements.
  4. Convert runs of separators to one separator, trim them, and define how punctuation, symbols, and empty results are handled.
  5. Guarantee uniqueness with a stable identifier or an explicit collision rule. Do not silently change an established slug whenever a title is edited.
  6. Store the chosen slug and redirect or resolve older slugs if URLs are migrated.

A useful data model keeps URL and content concerns separate:

{
  "title": "original Unicode title",
  "slug": "stable-route-value",
  "slugSource": "unicode-or-transliterated",
  "canonicalId": "stable-record-id"
}

Unicode slugs can preserve local spelling, while transliterated slugs may fit systems that expect a restricted character set. Neither choice is universally correct. If you use transliteration, document the library, language assumptions, and version so that regeneration does not unexpectedly change routes.

For a new locale or script

  1. Collect real examples, including uppercase and lowercase forms where applicable, combining marks, punctuation, names, and empty or symbol-only strings.
  2. Confirm the runtime's Unicode and locale-data behavior for casing and normalization.
  3. Check the search engine's analyzer configuration for the target language and script.
  4. Decide whether transliteration is an alias, a URL representation, or not used at all.
  5. Test rendering and directionality for Arabic and mixed-direction content. The bidirectional text guide is useful when display and storage appear to disagree.

What to double-check

Use a small, versioned test matrix before changing a normalizer, slug library, runtime, or search analyzer. Include at least these cases:

  • Turkish: test dotted and dotless I, including İ, i, I, and ı. Compare locale-aware behavior with your general search policy rather than assuming English casing applies.
  • Greek: include accented forms and final sigma. Check whether case folding and tokenization produce the matches your users expect.
  • German: test ß and any product-specific expectations around alternate spellings. Do not treat a transliteration choice as a universal linguistic rule.
  • Cyrillic: test both native-script search and any Latin alias. Watch for confusable characters that look similar but are different code points.
  • Arabic: include diacritics, right-to-left display, and punctuation. Verify that visual order is not being mistaken for stored character order.
  • CJK: test titles without spaces. Confirm how the search analyzer segments text and whether the slug policy should preserve characters, transliterate them, or use an identifier.

Also test equivalent Unicode sequences, combining marks, emoji or symbols adjacent to words, leading and trailing punctuation, repeated separators, very long titles, and two different titles that produce the same candidate slug. JavaScript implementations may benefit from Unicode property escapes for script-aware matching. If text crosses HTML, CSS, JavaScript, or JSON boundaries, review the appropriate character-escape guidance.

Record expected outcomes, not only expected intermediate strings. A good test says, for example, “query with the selected accent policy finds this document” and “changing the display title does not alter the existing route.” This makes failures meaningful after a dependency or locale-data update.

Common mistakes

  • Lowercasing everything with a default locale. Default locale behavior can vary between environments. Pass an explicit locale where locale-aware casing is required, and use a deliberate case-folding strategy for general matching.
  • Using transliteration as normalization. Normalization handles equivalent Unicode sequences; transliteration changes scripts or representations. They solve different problems.
  • Removing every accent. Accent-insensitive matching may improve discovery in some interfaces, but it can merge distinct words or names. Make it a field-level search decision.
  • Using the search key as the URL. Search keys are allowed to be lossy and may change as search requirements evolve. URLs need stability.
  • Assuming ASCII is safer. Restricting output can simplify legacy integrations, but it can also discard meaningful spelling and make local content less recognizable. Choose the constraint explicitly.
  • Ignoring confusables and invisible characters. Similar-looking characters can create duplicate or deceptive identifiers. Review the Unicode security guide when identifiers, usernames, or administrative workflows are involved.
  • Testing only one language. A transformation that appears correct for English can fail for Turkish, Greek, Arabic, or scripts without spaces.

When to revisit

Revisit this checklist before adding a locale, changing a search analyzer, replacing a transliteration library, upgrading a runtime with new Unicode or CLDR data, or migrating URL routes. It is also worth reviewing before seasonal planning cycles when large batches of pages, products, events, or names are imported.

Keep a short change record containing the runtime and library versions, normalization policy, locale assumptions, analyzer settings, slug rules, and representative test strings. When a duplicate, missing result, or changed URL is reported, compare each stage: original text, normalized text, case-folded value, accent policy, transliteration alias, analyzer output, and final route.

Before shipping, use this final action list:

  1. Preserve the original Unicode value.
  2. Define separate search and URL representations.
  3. Apply the same normalization and matching policy to queries and indexed data.
  4. Document locale, accent, transliteration, and collision rules.
  5. Run the Turkish, Greek, German, Cyrillic, Arabic, and CJK test matrix.
  6. Check search behavior, rendering, routing, and redirects together.
  7. Repeat the review whenever workflows, tools, locales, or Unicode-related data change.

Related Topics

#Unicode#Search#SEO#Internationalization#Text Processing#Web Development
U

Unicode.live Editorial Team

Developer Tools Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.