Appearance
To pipeline crowdsourced data into your systems, build an idempotent five-stage flow — extract, validate, transform, load, verify — that keys every record on a stable identifier and keeps the raw extract immutable. The core principle is that the pipeline must be re-runnable: when volunteers edit old pages or your schema changes, you regenerate the loaded data from raw exports and versioned scripts rather than patching by hand. This guide walks the full workflow with examples a working archivist can adapt.
What are the stages of a crowdsourced data pipeline?
Five stages, each isolated and re-runnable:
text
[Platform] --extract--> raw/ (immutable)
raw/ --validate--> reports + rejects
validated --transform-> target model (TEI / CSV / annotations)
transformed --load----> catalogue / repository / database
loaded --verify----> counts, spot-checks, link integrityKeeping stages separate means a transform bug never corrupts your extract, and a failed load never loses validated data. Each stage reads its input and writes a new artefact; nothing is edited in place.
How do I extract and validate crowdsourced data?
Pull from the platform's export or API, then validate before anything else touches it. Validation against an explicit schema catches malformed records early, where they are cheap to fix:
python
import json, jsonschema
schema = json.load(open("schema.json"))
records = json.load(open("raw/2026-02-10_export.json"))
good, bad = [], []
for r in records:
try:
jsonschema.validate(r, schema)
good.append(r)
except jsonschema.ValidationError as e:
bad.append({"id": r.get("page_id"), "error": str(e)})
json.dump(good, open("validated/good.json", "w"))
json.dump(bad, open("reports/rejects.json", "w"))
print(f"{len(good)} valid, {len(bad)} rejected")The reject report is as important as the valid output — it tells you whether a guideline or interface problem is producing systematically broken records.
How do I match records to existing catalogue items?
Join on stable identifiers only. A IIIF image identifier, an accession number, or a persistent page ID survives re-imports; a title or a date does not. Carry that identifier untouched through every stage so the link from crowdsourced text back to the catalogued object is never broken. If your only common field is something volatile, fix that upstream before building the pipeline — fuzzy-matching titles at load time is a recipe for silent mis-linking.
How should the transform stage map data?
Match the output format to the destination, not to convenience:
| Target system | Move data as |
|---|---|
| Collections catalogue | CSV / JSON fields |
| Digital edition | TEI-XML |
| IIIF viewer | Web Annotation (JSON-LD) |
| Relational database | Normalised tables |
| Repository deposit | CSV plus a data dictionary |
Forcing every system to accept one universal format usually means lossy conversions. Transforming per target keeps fidelity where it matters.
Should the pipeline run automatically or with a human in the loop?
Stage your trust. For the first imports, run extract and validate automatically but gate the load behind human approval so you catch surprises before they hit production. Once the pipeline has proven stable over several cycles, schedule incremental loads — pulling only records changed since the last run via a last-modified timestamp — and reserve manual review for records the validator flags. Full automation is the destination, not the starting point.
How do I handle edits and avoid duplicates?
Volunteers revisit old pages, so loads must be upserts, not inserts. Key on the stable identifier: if the record exists, update it; if not, insert it.
sql
INSERT INTO transcriptions (page_id, text, updated_at)
VALUES (:page_id, :text, :updated_at)
ON CONFLICT (page_id)
DO UPDATE SET text = EXCLUDED.text,
updated_at = EXCLUDED.updated_at;Pulling only records with a newer updated_at than your last run keeps incremental loads fast and prevents the duplicate explosion that naive re-imports cause.
How do I keep the whole pipeline reproducible?
Treat it like software. The raw extract is immutable and dated. Every stage is a script under version control. Each run logs its metadata — input file, record counts, script commit hash — so a future you, or a successor, can rebuild the exact loaded state from raw data plus code. This is what makes a crowdsourced dataset defensible: not "trust me," but "here is the script that produced it."
Key Takeaways
- Build five idempotent stages: extract, validate, transform, load, verify.
- Validate against an explicit schema and keep the reject report.
- Join records only on stable identifiers like IIIF or accession IDs.
- Transform per target system rather than forcing one universal format.
- Gate loads behind human review until the pipeline is proven, then automate incrementally.
- Use upserts keyed on stable IDs to handle edits without duplicates.
- Keep raw extracts immutable and every stage scripted and versioned.
Frequently Asked Questions
What are the stages of a crowdsourced data pipeline?
Extract from the platform, validate against a schema, transform to your target model, load into the system, and verify the load. Each stage should be idempotent so you can re-run it safely.
How do I match crowdsourced records to existing catalogue items?
Join on stable identifiers like a IIIF image ID or accession number, never on volatile fields like titles. Carry the source identifier through every stage so the link is never lost.
Should the pipeline run automatically or on demand?
Run validation continuously but gate the load behind a human approval step for the first imports. Once you trust the pipeline, schedule incremental loads and reserve manual review for flagged records.
What format should I move data in?
Move structured data as CSV or JSON and richer transcription as TEI-XML or IIIF annotations. Match the format to the target system rather than forcing one format everywhere.
How do I handle updates when volunteers edit old pages?
Use upserts keyed on a stable identifier so a re-import updates the existing record instead of duplicating it. Track a last-modified timestamp to pull only changed records.
How do I make the pipeline reproducible?
Script every stage, keep the raw extract immutable, log run metadata, and version the transform code. Anyone should be able to rebuild the loaded data from the raw export and the scripts.