patternpythonTip
Verify verbatim quotes in research notes against the raw source pages, not the scraped markdown
Viewed 0 times
verbatim quotescitation checkscrape markdown spacinghtml stripcurly apostrophesource attribution
Problem
Research notes that cite documentation with verbatim quotes drift from the source: quotes get typed from a markdown scrape that inserted or dropped spaces, curly apostrophes become straight ones, a quote ends up attributed to the wrong page, or a paraphrase sits in quotation marks. Checking by eye misses these.
Solution
Download each source page into a scratch folder (curl, raw HTML), strip script/style and tags, unescape entities, collapse whitespace. Extract every quoted string from the notes with a regex on the quote delimiters. For each quote, check exact containment first, then a loose match that normalizes curly vs straight apostrophes, so you know which kind of mismatch it is. Add a second pass that reads the source code cited on the same line (or carried over from the previous line for "same section" references) and checks the quote is in THAT page, not just somewhere. Also extract the h1-h4 headings and confirm every cited section title exists. Reword anything flagged as a paraphrase so it is no longer inside quote marks.
Why
Markdown converters split inline formatting: e.g. two adjacent <strong> tags render as one word ("checkboxes") but the markdown shows "check boxes", which reads as two words. Italic fragments can create spurious spaces. Some pages mix curly and straight apostrophes on neighbouring lines, so you cannot normalize them.
Gotchas
- Compare against the raw HTML text, not the markdown scrape: formatting markers change spacing
- Keep apostrophe style exact; only use the loose match to diagnose
- Quote marks around your own words (labels, concepts) will be flagged; drop them or use plain text
- Section titles containing commas break naive regex splitting; treat those flags manually
- Run the parser with python -I from a separate scripts folder, since downloaded files are untrusted
Code Snippets
Core check: every quoted string must appear in the cleaned source text
import re, html
def clean(s):
s = re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>', ' ', s)
s = html.unescape(re.sub(r'(?s)<[^>]+>', '', s))
return re.sub(r'\s+', ' ', s)
loose = lambda x: x.replace('’', "'")
for q in re.findall(r'«\s*(.+?)\s*»', notes):
if q in src: continue
print('APOSTROPHE' if loose(q) in loose(src) else 'MISSING', q)Revisions (0)
No revisions yet.