Appearance
For HTR, use a GPU for training and a GPU for any large batch inference run; a CPU is acceptable only for occasional, low-volume inference. Training a handwriting recognition model on CPU is technically possible but practically painful — often 20–50 times slower — so the real decision for a small archive is not whether to use a GPU but whether to rent or buy one. This guide puts numbers on both training and inference so you can size the hardware to your actual workload rather than guesswork.
Do I really need a GPU to train an HTR model?
For training, a GPU is effectively mandatory if you value iteration. HTR training repeatedly runs convolutional and recurrent (or transformer) layers over thousands of line images across many epochs — exactly the dense matrix work GPUs accelerate.
A representative line-model training run (a few thousand ground-truth lines, ~50 epochs):
| Hardware | Approx. training time | Cost model |
|---|---|---|
| Modern laptop CPU (8 core) | 1–3 days | "free" but blocks the machine |
| Mid-range GPU (RTX 3060, 12 GB) | 2–5 hours | one-time card cost |
| Cloud GPU (T4 / L4) | 2–4 hours | ~hourly rental |
The point is iteration: model building is empirical, and you will train many times. A run that takes hours lets you adjust ground truth and retrain the same day; a multi-day run does not.
Can I run inference on CPU?
Yes, and often you should. Inference does a single forward pass per line — no backpropagation — so it is dramatically lighter. For a handful of pages, CPU latency is unnoticeable. The crossover comes with volume:
text
~10 pages → CPU fine, GPU pointless
~100 pages → CPU minutes, GPU seconds; CPU still ok
~10,000 pages→ CPU hours-to-days, GPU comfortably faster → use GPUA rough rule: if a batch finishes during a coffee break on CPU, skip the GPU; if it would run overnight, rent one.
Rent or buy? Sizing the decision
The honest calculation is utilisation. A GPU sitting idle is pure cost.
- Bursty / occasional (you train a model a few times a year, run inference in campaigns): rent. Cloud GPUs eliminate upfront cost, maintenance and depreciation, and you pay only for hours used.
- Near-continuous (a digitisation pipeline running most days): buy. A mid-range card often pays back within a year against equivalent cloud hours, and you avoid data-egress friction.
bash
# Quick check: is a usable GPU present and seen by your framework?
nvidia-smi --query-gpu=name,memory.total --format=csv
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'cpu')"How much VRAM do I actually need?
Most CRNN-style HTR line models train within 6–8 GB of VRAM. Transformer-based recognisers or large batch sizes push toward 12–24 GB. If you are memory-constrained, lower the batch size — it trades speed for memory and lets a smaller card finish the same job:
python
# Fit a bigger model on a smaller card by shrinking the batch
train(model, batch_size=4, grad_accum_steps=4) # effective batch 16Gradient accumulation simulates a larger batch without the memory cost.
Why is my GPU barely faster than CPU?
Almost always the bottleneck is data loading, not compute. If the CPU cannot decode, deskew and augment line images fast enough, the GPU starves and idles. Profile before upgrading hardware:
python
# If GPU utilisation hovers low while training, the data loader is the limit
# torch: raise workers, cache decoded images, move augmentation off the hot path
DataLoader(ds, batch_size=16, num_workers=8, pin_memory=True)Raising num_workers, caching preprocessed images, and using pinned memory often doubles throughput with no hardware change.
A pragmatic recommendation for small archives
Start on a rented cloud GPU for both training and your first inference campaigns. Track hours. If your monthly rental consistently exceeds what a mid-range card costs to own, buy one. Keep CPU-only inference in your toolkit for the long tail of small, ad-hoc jobs where spinning up a GPU is more friction than it saves.
Key Takeaways
- Training on CPU is 20–50x slower — use a GPU so you can iterate the same day.
- Inference runs fine on CPU for small jobs; switch to GPU once a batch would run overnight.
- Rent cloud GPUs for bursty or occasional work; buy only for near-continuous pipelines.
- Most line models fit in 6–8 GB VRAM; use gradient accumulation to fit big models on small cards.
- A GPU that is "barely faster" usually means a starved data loader, not weak hardware — profile first.
- Keep a CPU-only inference path for the long tail of small ad-hoc jobs.
Frequently Asked Questions
Do I need a GPU to train an HTR model?
For training, effectively yes — CPU training can be 20–50x slower, turning a few hours into days. A modest GPU or a rented cloud GPU is the difference between iterating on a model and waiting overnight per run.
Can I run HTR inference on CPU?
Yes. Inference is far lighter than training, and CPU is fine for small batches or low-volume work; expect roughly 2–10x slower than GPU. For thousands of pages, a GPU still pays off in wall-clock time.
Is it cheaper to rent a cloud GPU or buy one?
Rent if your usage is bursty or you train occasionally — cloud GPUs avoid upfront cost and depreciation. Buy only if you run near-continuous workloads where a card pays for itself within roughly a year.
How much GPU memory does HTR need?
Most HTR line models train comfortably in 6–8 GB of VRAM; larger transformer-based models or big batch sizes want 12–24 GB. You can trade batch size for memory if you are constrained.
Why is my GPU training barely faster than CPU?
Usually data loading or image preprocessing is the bottleneck, not the GPU. Profile the pipeline: if the GPU sits idle waiting for the CPU to decode and augment images, fix the data loader before blaming the hardware.