Why it keeps happening
- The disk dies with the instance. On most pods and marketplace instances the container disk is deleted when you terminate, so the next session starts empty.
- Storage is in another datacenter. A volume can only attach to GPUs in its own datacenter. If that datacenter has no free GPUs, you rent elsewhere and download again.
- The cache path is wrong. Hugging Face libraries cache to
~/.cache/huggingfaceby default. If that path is not on persistent storage, every run fetches the model again.
What a re-download costs
| Model (16-bit weights) | Size | At 1 Gbit/s | At 10 Gbit/s |
|---|---|---|---|
| 8B | ≈ 16 GB | ≈ 2 min | ≈ 13 s |
| 70B | ≈ 140 GB | ≈ 19 min | ≈ 2 min |
| DeepSeek V3/R1 (671B, FP8) | ≈ 700 GB | ≈ 1.5 hours | ≈ 9 min |
Theoretical time at full line speed; real downloads are often slower. Size = parameters × 2 bytes in 16-bit.
The GPU bills the whole time. Twenty minutes of downloading a 70B model on one H100 costs about a third of $3.23; on an 8-GPU node it costs eight times that, every session.
Four fixes, from cheapest to simplest
- 1. Point the cache at persistent storage. Set
HF_HOME(orHF_HUB_CACHE) to a mounted volume before importing the libraries, and setHF_HUB_OFFLINE=1once the model is cached so loading does not call the Hub at all. - 2. Keep a volume in the same datacenter as your GPUs. On RunPod, network volumes cost $0.07/GB per month for the first 1 TB and $0.05/GB above that, are tied to the datacenter you pick, and do not sync between datacenters (checked 2026-11-17). Pick a datacenter that usually has your GPU free.
- 3. Bake small models into the container image. Works for models of a few GB; for 70B+ the image becomes too large to pull quickly.
- 4. Keep the server. If you use the model every day, a dedicated server with local NVMe removes the problem: weights stay on the disk between jobs, and you stop GPUs from being idle rather than giving the machine back.
How it works at Strata
- Reserved dedicated nodes: the bare-metal server and its local NVMe are allocated to you alone for the term (3, 6, 12 or 24–36 months), so weights, datasets and checkpoints stay put. An 8×H100 node costs $18,863 a month at the on-demand rate and less on longer terms; 8×H200 costs $25,170.
- On demand for tests: per-minute billing from $3.23 per H100-hour. Before you stop an on-demand server, copy what you need to your own storage in the same region.
- Region named in the reservation: Europe, North America, the Middle East and Asia-Pacific, so storage and GPUs sit in the same datacenter.


