Appearance
Automate OCR post-correction cleanly by only changing text where you have positive evidence both that it is wrong and what it should be — low engine confidence on the token, a high-confidence candidate within small edit distance, and a measured CER improvement on held-out data. The cardinal sin of post-correction is enthusiasm: a pass that fixes 500 errors but invents 80 new ones is often worse than the raw OCR, because the new errors look plausible and survive review. The workflow below layers cheap, safe corrections first and reserves language models for what dictionaries cannot reach.
What is OCR post-correction, exactly?
It is everything between raw recognition output and a finished transcript: removing recognition artefacts, fixing systematic glyph confusions, rejoining hyphenated line-breaks, and resolving genuine word errors. Crucially it runs on the text, so it is fast and re-runnable without touching images or the model.
A layered pipeline that won't backfire
Order matters — deterministic, lossless steps first; risky, lossy steps last:
text
1. Mechanical cleanup → de-hyphenate line breaks, normalise whitespace/Unicode (NFC)
2. Rule-based fixes → known systematic substitutions (ſ→s, fi→fi, scanner artefacts)
3. Dictionary lookup → period lexicon, only within edit distance ≤ 2
4. Confidence gating → only touch tokens below the OCR confidence threshold
5. Language-model rerank→ context disambiguation, constrained to near candidates
6. Audit & measure → CER/WER on validation set, per-rule loggingEach step is reversible and logged. If step 5 raises CER, you disable it without losing steps 1–4.
How do I avoid introducing new errors?
Gate corrections on confidence + edit distance. A token the engine emitted at 0.98 confidence is probably right; do not let a dictionary "fix" it. Conversely, a 0.4-confidence token that is one edit from a frequent dictionary word is a safe correction.
python
def safe_correct(token, conf, lexicon, max_dist=2, conf_gate=0.85):
if conf >= conf_gate:
return token # trust the engine
cands = lexicon.within_distance(token, max_dist)
if len(cands) == 1: # unambiguous nearby word
return cands[0]
return token # ambiguous → leave itThe "exactly one candidate" check is doing heavy lifting: ambiguity is where automated correction guesses, and guessing is where new errors come from.
Will a spellchecker corrupt historical spelling?
Yes, if it is a modern one. colour, publick, connexion and thousands of legitimate period forms get "fixed" into anachronisms. Build or borrow a period-appropriate lexicon — for English, derive one from a transcribed corpus of the same era, or use historical variant dictionaries. Keep variant spellings as valid; only flag forms that match neither the period lexicon nor any near-OCR-error pattern.
When does a language model earn its place?
Dictionaries are context-free, so they cannot choose between bare and bore when both exist. A language model (a small n-gram model, or a masked LM constrained to candidates near the OCR string) resolves these with context. The constraint is essential — let the LM rank only candidates the OCR plausibly produced, never generate freely, or it will smooth your text into confident fiction.
| Method | Catches | Risk | Cost |
|---|---|---|---|
| Mechanical rules | Hyphenation, artefacts | Very low | Trivial |
| Period dictionary | Single-edit typos | Low | Low |
| Confidence gating | Real recognition errors | Low | Low |
| Constrained LM | Context-dependent errors | Medium | Medium |
| Free generative LM | Everything — and hallucinations | High | High |
Measuring whether it actually helped
Without measurement, post-correction is faith. Hold out a transcribed validation set and compute CER/WER after each layer:
bash
# pseudo: compare gold vs corrected per stage
for stage in raw rules dict lm; do
python eval_cer.py gold/ out_$stage/ # expect monotonic CER drop
doneIf any stage raises CER, it is net-harmful on your material — disable or re-tune it rather than shipping it.
Key Takeaways
- Correct only where confidence is low and a near, unambiguous candidate exists.
- Apply lossless mechanical fixes first; reserve language models for context errors they alone can solve.
- Never use a modern spellchecker — build a period lexicon so historical spellings survive.
- Constrain language models to candidates near the OCR output; free generation hallucinates.
- Log every rule and measure CER/WER on held-out data after each stage.
- Leaving uncertain tokens alone is usually safer than a confident wrong fix.
Frequently Asked Questions
What is OCR post-correction?
OCR post-correction is the step after recognition that fixes systematic errors in the raw text using dictionaries, rules, confidence scores or language models. It improves accuracy without re-running the OCR engine itself.
Will a spellchecker fix OCR errors safely?
Only partly — a modern spellchecker corrupts historical spellings it treats as misspellings. Use a period-appropriate lexicon or variant dictionary, and constrain corrections to high-confidence, low-edit-distance candidates.
Should I use a language model for OCR correction?
Language models help with context-dependent errors that dictionaries miss, but they can hallucinate plausible-but-wrong text. Constrain them to candidates near the OCR output and always measure CER before and after.
How do I avoid introducing new errors during correction?
Only correct where you have evidence: low OCR confidence plus a high-confidence replacement within small edit distance. Audit every rule against held-out ground truth and keep corrections reversible.
Can I correct OCR without ground truth?
You can apply dictionary and rule-based correction without ground truth, but you cannot measure whether it helps. Always transcribe a small validation set so you can quantify CER/WER change.
Which OCR errors are best left uncorrected?
Leave rare proper nouns, genuine historical variants and anything below your confidence threshold. Over-correction of these is the main way automated cleanup makes a corpus worse than the raw OCR.