The string you validated no longer exists

A Unicode transliterator is a string rewriting engine. ICU - the International Components for Unicode library that ships inside Java, PHP, Node bindings, ElasticSearch, and most SQL engines - implements transliteration as ordered, context-sensitive rewrite rules. Each rule matches a pattern with left and right context and substitutes a replacement. The engine runs passes until the string stops changing. That construct is a semi-Thue system. Semi-Thue systems simulate Turing machines. A crafted ruleset can encode a tag system, recurse, and never reach a fixed point.

That is the academic result. It is the least useful part.

The property that matters operationally sits upstream of Turing completeness. Transliteration, normalization, case folding, and best-fit mapping are non-injective transformations applied at trust boundaries. The byte sequence a system validates is not the byte sequence the system later executes, stores, or compares. The transformation lives between the two. Every control that runs before the transformation is reasoning about a string that no longer exists after it. The Turing completeness only proves the transformation class is unbounded - the output cannot be predicted by inspecting the input under a bounded model. Attackers do not need arbitrary computation. They need the transform to change one byte the validator already cleared.

Start with normalization forms. NFC, NFD, NFKC, NFKD. NFKC collapses compatibility characters into canonical equivalents. Fullwidth Latin U+FF21 folds to U+0041, capital A. The ligature U+FB01 folds to the two ASCII bytes f and i. Circled, superscript, and mathematical alphanumeric ranges all fold to plain ASCII. Case folding runs the other direction - U+00DF, the sharp s, expands to ss. One codepoint becomes two bytes. String length changes after the length was checked. The set is many-to-one in one pass and one-to-many in another, and the two passes rarely run at the same layer.

Best-fit mapping is the sharpest instance. When Windows converts a wide string to a narrow codepage through WideCharToMultiByte with the best-fit flag set, it approximates characters that have no exact encoding. U+00AD, the soft hyphen, best-fits to 0x2D, the ASCII hyphen-minus. U+FF0D, the fullwidth hyphen, does the same. The conversion is lossy by design and silent by default.

CVE-2024-4577 is that mechanism weaponized. PHP CGI on Windows. CVSS 9.8. The php-cgi SAPI hands the request query string to the process command line. On systems running Chinese or Japanese codepages, the OS applies best-fit mapping during command-line conversion. A soft hyphen supplied in the request converts to 0x2D. That reintroduces the argument delimiter the CVE-2012-1823 patch had filtered a decade earlier. The filter checked for 0x2D. The character supplied was U+00AD. The transformation between the check and the exec produced 0x2D. Validation and execution disagreed on the bytes, and the disagreement was the exploit. Injected -d directives set auto_prepend_file and allow_url_include, and the request reaches remote code execution. Affected builds: PHP before 8.1.29, before 8.2.20, before 8.3.8. TellYouThePass ransomware operators were mass-scanning and exploiting it within 48 hours of disclosure. MITRE T1190, exploitation of a public-facing application, into T1059 command execution.

Django CVE-2019-19844 is the same fault in an identity flow. The password-reset lookup compared submitted email against stored email using a Unicode-aware, case-insensitive match. An attacker registered an address whose codepoints normalized under NFKC to a privileged account’s address while differing byte-for-byte at registration. The reset request normalized, matched the victim, and the token routed to the attacker’s own registered mailbox. Account takeover with no credential theft. Rated critical. The remediation forced normalization to a canonical lowercase form before the lookup, then re-fetched by the stored address - canonicalize first, decide second. Identity providers that fold usernames before comparison inherit the same exposure surface. Okta, Entra, any directory that treats display-form and match-form as interchangeable is making the assumption this bug punishes.

Trojan Source, CVE-2021-42574 with the homoglyph sibling CVE-2021-42694, moves the fault into source code itself. Bidirectional control characters - RLO, LRO, and the isolates U+2066 through U+2069 - reorder how a line renders without changing the token stream the compiler parses. The reviewer reads one program. The compiler builds another. A logic inversion or an early return hides inside a comment that visually terminates where it does not. The renderer transforms; the compiler does not; the trust gap is between two views of the same bytes. Homoglyph variables push the same idea further - a Cyrillic character that renders identical to Latin resolves to a different identifier.

One mechanism runs through all four. A trust boundary, a non-injective transform, and an ordering fault. The system validates form A and operates on form B. Validate-then-transform is the anti-pattern. Transform-then-validate - canonicalize to a single normal form before every security decision, then compare canonical forms only - is the control that closes it. The order is the entire defense. Reverse it and the strongest allow-list on earth is checking a string the sink never sees.

Telemetry is where this gets ugly, because the malicious byte is invisible at the layer that logs. WAF signatures written against raw request bytes miss normalized and homoglyph payloads, because the dangerous ASCII form only materializes inside the application after conversion. Engines that normalize before matching catch more - Cloudflare’s managed rules and ModSecurity with transformations like t:urlDecodeUni and t:normalizePath will fold some of it - but a WAF that normalizes for matching and then logs the decoded form destroys the evidence. The U+00AD is gone from the log line. The analyst reviewing the SIEM sees a hyphen, or sees nothing. Request the raw, pre-transformation bytes at the edge or the artifact is already lost.

On the host, the signal moves to process creation. Sysmon Event ID 1 on a PHP server shows php-cgi.exe spawning with unexpected -d arguments, or php-cgi.exe as the parent of cmd.exe, powershell.exe, or a scripting host. That parent-child pair is the durable detection for CVE-2024-4577 post-exploitation, because the encoding trick is upstream and unlogged but the payload still has to execute something. Event ID 3 captures the outbound stage when allow_url_include pulls a remote include. For the Django class, application logs show a password reset issued to an address that normalizes onto a privileged account - anomalous, present, and almost never alerted, because no rule compares the normalized form of a reset target against the admin set. For Trojan Source, the detection is static, not runtime: scan diffs and commits for bidirectional controls in the U+202A-U+202E and U+2066-U+2069 ranges and for mixed-script identifiers. GitHub now flags bidi characters in blobs. CodeQL and dedicated linters catch them in CI. Nothing catches them at execution, because by execution the source is already compiled and the visual deception served its purpose during review.

Map the cluster to ATT&CK and the picture holds. T1027, obfuscated files or information, covers the encoding itself. T1140, deobfuscate or decode files or information, is the transform the target performs on the attacker’s behalf - the victim system does the decoding step. T1036, masquerading, covers homoglyph identifiers and lookalike identities. T1190 is the delivery for the server-side instances. The through-line is that the target’s own text pipeline completes the attack. The attacker ships an ambiguous string. The platform resolves the ambiguity in the attacker’s favor.

The patch boundaries are narrow and the residual exposure is wide. CVE-2024-4577 was fixed by changing how PHP hands arguments across the command-line boundary on Windows, not by fixing best-fit mapping, which is OS behavior and still active for every other process that performs the same conversion. Django’s fix normalized one lookup; any comparison elsewhere that folds before matching carries the identical fault. Trojan Source produced compiler and linter warnings, not a change to Unicode, because bidirectional overrides are a legitimate feature for right-to-left scripts and cannot be removed. Each patch closes one instance. The assumption underneath every instance is unpatched, because it is architectural: that the string a system checked is the string that system will use. Normalization sits under usernames, filenames, path resolution, URL parsing, TLS SNI, email routing, and JWT claim comparison. Wherever a byte sequence crosses from a validator to a sink through a codepage conversion, a normalization pass, or a case fold, the two ends can be made to disagree. UTF-8 being well-formed guarantees the bytes decode. It guarantees nothing about what they become on the other side of the transform. The engine that turns one into the other is Turing-complete, and the security model assumed it was a lookup table.

See also: NordVPN for tunneled traffic when operating outside controlled networks.

---

#ad Contains an affiliate link.