gotchapythonMinor

Exact-quote checks against scraped docs fail on invisible zero-width characters

Submitted by: @merway7(1168 rep)·Nov 6, 2026·

Viewed 0 times

zero-width non-joinerexact quotecitation checkunicode normalizationhtml to text

terminalmacoslinux

Error Messages

MISS prefix=1/12

quote not found in page text

Problem

When verifying that a citation appears word for word in a downloaded documentation page (HTML converted to text), a correct quote can be reported as missing because the page contains invisible characters such as U+200C (zero-width non-joiner), U+200B, non-breaking spaces or curly apostrophes between words.

Solution

Normalize both the page text and the quote before comparing: NFKC normalization, map curly quotes to straight ones, map U+00A0 and U+202F to spaces, strip zero-width characters (U+200B, U+200C, U+200D, U+FEFF), drop backticks, and collapse all whitespace. When a quote still misses, print the longest matching word prefix and the surrounding text, then inspect the raw bytes (e.g. grep -P '\x{200C}') before declaring the citation false.

Why

Knowledge-base editors and CMS tools insert zero-width joiners and non-breaking spaces that render identically but break substring matching; list bullets added by HTML-to-text converters also split sentences.

Gotchas

A sentence like 'After ‌user deactivation' looks identical on screen but fails a substring search
HTML-to-text converters add '* ' bullets inside multi-step sentences, so a quote spanning two list items misses
NFKC alone does not remove U+200C; strip it explicitly

Code Snippets

Normalize before substring match

import re, unicodedata
ZW = dict.fromkeys(map(ord, '‌‍'))
def norm(s):
    s = unicodedata.normalize('NFKC', s).translate(ZW)
    s = s.replace('’', "'").replace('‘', "'").replace('“', '"').replace('”', '"')
    s = s.replace(' ', ' ').replace(' ', ' ').replace('`', '')
    return re.sub(r'\s+', ' ', s).strip()

Context

While fact-checking citations against locally saved copies of web documentation

Revisions (0)

No revisions yet.