How to stop losing work when your cloud GPU shuts down
We learned to treat a cloud GPU like a rented room. Our work needs a home outside it. These are the rules we use to keep files and jobs safe when compute changes.
We keep three things apart
Our safe GPU workspace has three layers. The first is the view we see. The second is the live machine. The third is the durable store that keeps the truth.
We tried simpler shapes first. It is easy to make the folder on the GPU the only copy. That feels fast until the GPU stops, moves, or fails.
- The workspace is the map of the work.
- The GPU is a worker that may leave.
- The durable store is the long-term record.
Saving is more than copying bytes
We need to know which file version is current. We also need to know when a write is done. A list of names is not enough if two workers can write at once.
We use a small record called a manifest. It names each file, its version, and where its bytes live. A new write moves that record forward in one safe step.
The bytes are books. The manifest is the library card that says which books are on the shelf now.
One checkpoint taught us the whole rule
Picture a model writing a 12 GB checkpoint. The bytes arrive in many parts. We do not call the save done just because most parts exist.
First, we write the parts. Next, we check their hashes. Last, we publish one new manifest. That final pointer is the small move that makes the large save clear.
If a worker stops early, the old manifest still points to the last full checkpoint. We can clean up loose parts later without guessing which version was real.
Move one small pointer only after every large part is safe.
A safe stop has an order
We learned that stopping a GPU is not one click inside the system. First, we stop new paid work. Next, we give active writes a short time to finish. Then we save the last good record. Last, we release the machine.
The order matters. If the machine dies first, the last file may not reach the durable store. If billing stops last, we may pay while the system waits.
- Block new work.
- Drain active writes once.
- Save the final file record.
- Stop billing and release compute.
Resume means rebuild, then read back
We never ask a new GPU to guess what the old one had. It reads the durable record, fetches the needed files, and reports which version it opened.
We can keep large model files lazy. The new worker fetches a block when code reads it. Small files arrive first, so the workspace can open fast.
We end resume with a readback. The system checks that the file view matches the saved record before it says the workspace is ready.
We use the same rule for long-running agents
Our agents can run longer than a browser tab. Their plan, task state, logs, and proof live outside the tab. The browser watches the work. It does not own the work.
This is the same idea as durable files. We put the truth in a place that outlives the short-lived worker. Then a healthy worker can pick up from a known point.
Five questions to ask a GPU workspace
We use these five questions when we review our own system. They test the full path, not one feature name.
- What is the source of truth for each file?
- What happens to an active write during stop?
- Can work resume on a different machine?
- Can the system show which version it opened?
- When does paid compute truly stop?