Build a Unicode Test Corpus for Web Apps: Fixtures for Languages, Symbols, and Edge Cases
UnicodeTestingInternationalizationQuality AssuranceWeb AppsText Processing

Build a Unicode Test Corpus for Web Apps: Fixtures for Languages, Symbols, and Edge Cases

UUnicode.live Editorial Team
2026-11-20
8 min read

Build a reusable Unicode test corpus for multilingual UIs, APIs, search, storage, serialization, emoji, RTL text, and edge cases.

A small, deliberate Unicode test corpus can expose bugs that ordinary ASCII fixtures miss. This guide gives you a reusable set of strings and a practical checklist for testing multilingual interfaces, APIs, search, storage, serialization, layout, and error handling.

Overview

Internationalized applications do not process text as a single, uniform category. A visible character may contain multiple code points; a displayed symbol may be an emoji sequence; a user-perceived character may occupy more or fewer code units than your language's string length reports. Text can also behave differently when it is sorted, normalized, wrapped, escaped, stored, or rendered with a missing font.

A Unicode test corpus is a named collection of fixtures that represents these conditions consistently across your test suite. It should include ordinary multilingual text as well as intentionally difficult input. Store each fixture with a description, expected behavior, and, where useful, the expected code points or normalized form. Keep the corpus separate from individual tests so that one update benefits UI, API, database, and search coverage.

Use stable labels rather than relying on the appearance of a string alone. A practical fixture record might look like this:

{
  "id": "combining-e-acute",
  "value": "é",
  "category": "combining-sequence",
  "description": "Latin e followed by a combining acute accent",
  "expectedNFC": "é",
  "expectedNFD": "é",
  "notes": "Compare display, length, equality, and cursor behavior"
}

Do not assume that every fixture should be rejected, normalized, transliterated, or treated as one character. The correct result depends on the field and product requirement. A display name, password, search query, URL slug, database key, and log message may legitimately have different rules.

Checklist by scenario

1. Accented text and combining sequences

  • Include precomposed text such as é, ñ, and å.
  • Include visually equivalent decomposed text such as é, ñ, and å.
  • Compare equality, search matching, sorting, validation, and database uniqueness under the normalization policy you chose.
  • Test a base letter with more than one combining mark, and test cursor movement or deletion in an editable field.

These cases are essential because two strings can look identical while containing different code-point sequences. The normalization guide on NFC, NFD, NFKC, and NFKD is useful when deciding which comparisons should be canonicalized.

2. Multiple scripts and language metadata

  • Add representative samples for every writing system your product claims to support, such as Latin, Cyrillic, Greek, Arabic, Hebrew, Devanagari, Thai, Hangul, and CJK text.
  • Include mixed-script values to test validation and spoof-resistant display rules, for example a Latin word beside a visually similar character from another script.
  • Pair strings with language tags such as en, fr, ar, ja, or a more specific BCP 47 tag where the locale affects formatting or sorting.
  • Test fallback behavior when a language tag is missing, unsupported, or more specific than the available translation.

Language identification and Unicode content are related but not interchangeable. Preserve language metadata separately from the string, then test formatting and sorting with the intended locale. See the guide to BCP 47 language tags and Unicode locale extensions for web-focused examples.

3. Right-to-left text and bidirectional content

  • Use Arabic and Hebrew sentences, not only isolated letters.
  • Test mixed-direction content such as an Arabic name with a Latin product code and digits.
  • Include punctuation, parentheses, URLs, dates, and numbers inside right-to-left text.
  • Check text alignment, truncation, caret movement, copy and paste, and the visual order of labels and values.
  • Test explicit direction settings and the behavior of direction-sensitive containers.

Capture screenshots at narrow and wide widths. A string can be semantically correct while its visual ordering makes an interface confusing. Test both natural content direction and the direction required by the component.

4. CJK, long words, and layout pressure

  • Include Japanese, simplified or traditional Chinese, and Korean samples appropriate to your supported locales.
  • Test text with no spaces, long unbroken identifiers, email-like values, URLs, and hashes.
  • Check wrapping, clipping, ellipsis, line height, minimum widths, table columns, buttons, and responsive navigation.
  • Verify that font fallback does not change the component's height or cause missing-glyph boxes.

Character count is not a reliable substitute for rendered width. If your application also produces terminal output, test display width separately; combining marks and wide CJK characters can affect alignment. The reference on Unicode character width in terminals covers that distinction.

5. Emoji, symbols, and punctuation

  • Include plain emoji such as 😀, text and emoji presentation variants such as ♥ and ♥️, and emoji with skin-tone modifiers.
  • Include joined sequences such as family or profession emoji that use zero-width joiners.
  • Test symbols from different blocks, mathematical signs, currency symbols, smart quotes, dashes, ellipses, and non-ASCII punctuation.
  • Check character limits, deletion, selection, database length constraints, and JSON or HTML escaping.

An emoji sequence may be several code points even though users perceive it as one symbol. Keep sequence fixtures intact and test user-perceived deletion where the platform supports it. The emoji sequence reference explains variation selectors, joiners, and skin tones.

6. Invisible characters and separators

  • Include ordinary spaces, non-breaking spaces, narrow no-break spaces, tabs, carriage returns, line feeds, and mixed line endings.
  • Test zero-width joiners, zero-width non-joiners, word joiners, and leading or trailing whitespace where your input policy permits them.
  • Include an empty string, whitespace-only input, repeated separators, and a string containing an unexpected control character.
  • Verify logging, trimming, display, copy and paste, tokenization, and serialization without silently losing information.

Make invisible characters visible in test failure output by printing escaped code points or a diagnostic representation. A fixture that appears blank in a failure message is difficult to maintain.

7. Malformed and adversarial input

  • Include isolated UTF-16 surrogates or invalid byte sequences at the boundary where your application decodes input.
  • Test invalid percent-encoding, invalid JSON escape sequences, truncated multibyte input, and overlong or unexpectedly large values where relevant.
  • Check that parsers fail safely, errors are actionable, and rejected data does not become replacement characters without an intentional policy.
  • Use resource limits for oversized strings and deeply repeated combining marks rather than allowing stress fixtures to destabilize the test environment.

Malformed data should be tested at system boundaries, not necessarily represented as valid in-memory strings. Define whether the boundary rejects, repairs, preserves, or reports each class of invalid input.

8. Reuse the corpus across the workflow

  1. UI snapshots: Render every fixture in inputs, labels, tables, menus, alerts, and empty states. Add narrow viewport checks and right-to-left snapshots.
  2. API validation: Send fixtures through request bodies, query parameters, headers where allowed, and error responses. Assert status, decoded value, and validation message.
  3. Search: Test exact matching, case handling, accent handling, tokenization, highlighting, prefix search, and sorting. Use Intl.Collator or another locale-aware strategy deliberately rather than assuming binary comparison is sufficient; see the Intl.Collator guide.
  4. Database round trips: Insert, retrieve, update, compare, index, sort, and enforce uniqueness for each relevant category. Confirm that the driver and database use the expected encoding and normalization behavior.
  5. Serialization: Round-trip fixtures through JSON, HTML, URL encoding, logs, queues, and exports. Verify that escaping changes representation rather than meaning. The guide to Unicode escapes in HTML, CSS, JavaScript, and JSON can help separate these contexts.
  6. Regular expressions: Test script-aware matching, boundaries, combining marks, and emoji sequences. Use Unicode property escapes where supported, and document engine differences; see the Unicode property escapes reference.

What to double-check

Before merging a corpus, record the intended behavior rather than only recording the input. For each fixture, ask:

  • What is the input encoding at this boundary, and where is decoding performed?
  • Is comparison byte-based, code-point-based, normalized, case-insensitive, or locale-aware?
  • Does the product limit bytes, code units, code points, grapheme clusters, or rendered width?
  • Should the value be preserved exactly, normalized for comparison, or transformed for a derived field such as a slug?
  • Can users select, delete, copy, paste, and edit the value without losing visible content?
  • Does the chosen font contain the glyphs, and does fallback preserve the intended layout?
  • Are escaped output and decoded output both checked in tests?

For search and slug generation, keep the original value and derived value as separate assertions. Transliteration can improve discoverability but is not a substitute for preserving the user's text. For CSS and font coverage, test the actual production font stack and any unicode-range rules rather than relying only on a system font.

Common mistakes

  • Testing only visible labels: Include API payloads, database keys, search terms, filenames, URLs, and logs.
  • Counting JavaScript string length as user characters: Supplement code-unit checks with code-point and grapheme-aware expectations where the requirement is user-facing.
  • Using one “international” sentence: A single sentence rarely covers combining marks, direction changes, scripts, emoji sequences, and no-space wrapping.
  • Normalizing everything without a rule: Normalization may be appropriate for comparison but undesirable when preserving user input or signed data.
  • Ignoring fonts: Correct code points do not guarantee a visible glyph, stable line height, or consistent fallback.
  • Hiding invisible characters in failures: Print escaped values, code points, and lengths in diagnostic output.
  • Mixing valid and malformed input without labels: Boundary tests become ambiguous when a fixture's validity is not explicit.
  • Snapshotting only one browser or viewport: Direction, font fallback, selection, and line breaking can vary across environments.

When to revisit

Review the corpus before an internationalization launch, when adding a supported language or script, after changing database or API encoding, and after replacing fonts, UI frameworks, parsers, search libraries, or serialization code. Re-run the most important fixtures during dependency and browser upgrades, especially when text rendering, regular expressions, or locale behavior is part of the change.

Also revisit it after every production text bug. Add the smallest fixture that reproduces the failure, give it a descriptive category, and attach expected behavior before fixing the implementation. During seasonal planning cycles, check whether new content types, campaigns, user-generated fields, or export formats introduce different text paths.

To keep maintenance manageable, designate an owner, store fixtures in version control, and document the Unicode data or runtime assumptions that matter to your tests. A useful final checklist is: add one sample for every supported script; add one decomposed and one precomposed pair; add right-to-left and mixed-direction values; add emoji sequences; add invisible and long inputs; add malformed boundary cases; then run the same corpus through UI, API, search, storage, and serialization tests. This turns Unicode support from an occasional manual check into a repeatable engineering practice.

Related Topics

#Unicode#Testing#Internationalization#Quality Assurance#Web Apps#Text Processing
U

Unicode.live Editorial Team

Technical Editor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.