Appearance
To document a research workflow well, record three layers as you go: a project-level README that orients any reader, a dated decision log that captures every non-obvious choice, and pinned versions of the tools and data you used. Good documentation is written during the work, lives inside the version-controlled repository, and is specific enough that a competent stranger could reproduce your result without asking you a single question.
In archival and data-heavy humanities, the threat is not lost code but lost reasoning: why a date was normalised one way, why a record was excluded, which scan version fed the OCR. This guide gives a practical checklist to capture all of that.
What does "documented" actually mean here?
A workflow is documented when a reader can answer four questions from your files alone: what are the inputs, what was done to them, in what order, and why each non-trivial decision went the way it did. If any of those requires emailing you, the documentation has a hole. The test is concrete — hand the repository to a colleague and ask them to reproduce one figure; the gaps they hit are your to-do list.
Why keep a dated decision log?
The decision log is the humanities lab notebook and the highest-value habit you can adopt. Every time you make a judgement call, append a dated entry. Keep it in plain Markdown so it diffs cleanly:
markdown
## 2025-01-14
- Normalised all dates to ISO 8601. Ambiguous "Jan 03" entries in
the parish register assumed 1903 not 1803 based on ink and hand.
- Excluded 11 records with no legible surname; listed their IDs in
data/excluded.csv with reasons.
## 2025-01-16
- Switched OCR engine from Tesseract 4 to 5 — abbreviation handling
noticeably better on secretary hand. Re-ran folios 1-40.Written at the moment of decision, these entries take seconds and save hours, because reconstructed reasoning is always thinner and sometimes wrong.
How do I make manual steps reproducible?
Much humanities work is point-and-click: cropping in an imaging tool, segmenting in Transkribus, reconciling in OpenRefine. Document these as a numbered protocol with exact settings, not vague prose:
markdown
### Reconciliation protocol (OpenRefine 3.7)
1. Import `raw/persons.csv` with UTF-8 encoding.
2. Cluster the `surname` column with key collation, merge clusters
with 3+ members only.
3. Reconcile against Wikidata, type Q5 (human); accept matches
scoring >= 80, queue the rest for manual review.
4. Export operations history to `docs/openrefine-operations.json`.Naming the exact version, threshold and export makes a mouse-driven step as repeatable as a script.
Which documentation layers matter most?
Not all documentation earns its keep equally. Prioritise like this:
| Layer | Effort | Payoff | Priority |
|---|---|---|---|
| README orientation | Low | High | Essential |
| Decision / cleaning log | Low | Very high | Essential |
| Pinned versions and environment | Low | High | Essential |
| Inline code comments | Medium | Medium | Recommended |
| Full data dictionary | Medium | High | Recommended |
| Diagram of the pipeline | Medium | Medium | Optional |
If you only do three things, do the README, the decision log, and version pinning.
How do I keep documentation from rotting?
Documentation rots when it lives apart from the work. Keep it in the repository so a change to the data and a change to its description land in the same commit. Add a lightweight check to your routine: before any release, re-run the README's reproduction steps on a fresh checkout. Anything that fails is documentation that has drifted from reality, and you fix it then rather than discovering it years later.
What about documenting data provenance?
For every dataset, record where it came from, when you retrieved it, and what transformations sit between the source and the file in your repository. A short provenance note per dataset — original archive reference, retrieval date, processing scripts in order — lets a reviewer trace any value back to a folio. This is the backbone of a defensible, auditable result.
Key Takeaways
- Document during the work, not afterwards; reconstructed reasoning is thinner and error-prone.
- Keep a dated decision log — the humanities lab notebook and your highest-value habit.
- Store documentation inside the version-controlled repository so it cannot drift away from the data.
- Turn point-and-click steps into numbered protocols with exact versions, thresholds and exports.
- Prioritise README, decision log and pinned versions above all other documentation.
- Test documentation by reproducing one result on a clean checkout before every release.
Frequently Asked Questions
What is the minimum documentation a project needs?
A README explaining what the project is and how to run it, a record of every data-cleaning decision, and pinned software versions. Those three things cover most reproducibility needs.
Where should workflow documentation live?
Inside the project repository, version-controlled alongside the code and data it describes. Documentation kept in a separate wiki or shared drive drifts out of sync quickly.
How detailed should cleaning decisions be?
Detailed enough that someone else would make the same choice. Record what you changed, why, and any borderline cases, ideally as you go rather than reconstructed afterwards.
Is a lab notebook relevant to humanities work?
Yes. A dated decision log is the humanities equivalent of a lab notebook and is the single highest-value documentation habit for archival and data work.
How do I document manual steps that are not code?
Write them as a numbered, repeatable procedure in the README or a protocol file, naming exact menu paths, settings and thresholds so the manual step is as reproducible as a script.