Checkpoints and retention
What a checkpoint is, where Portal keeps it, how it is protected, and how long Portal keeps checkpoints, outputs, logs and uploads.
Warnings and limits
- A suspended job's checkpoint is kept 3 days by default. Portal warns 18 hours before it expires; then the job fails with "the checkpoint expired", and its checkpoints are kept 7 more days, as after any end Portal decides. Resume or stop it before then. The job's time limit (
ttl) does not end a suspended job. - When a job ends on its own (it finished) or a person stops it, its checkpoints are deleted at once. After any other end (failed, lost, no capacity, an expired checkpoint, a policy's stop) they are kept 7 days.
- A job's output is kept 7 days, its log parts 30 days, an upload no job used 1 day.
- A checkpoint that fails its verification on arrival is kept 1 day for diagnosis, then deleted.
- Checkpoints are encrypted with a key of each job's own; outputs and logs are not.
What a checkpoint is
A checkpoint (Portal also calls it a capsule) is your program's state at a pause: its objects, tensors, loop positions and random generators, with the files of its working directory. Portal writes one when a job moves, is suspended, gets a spot notice, runs out of memory in a way it can recover from, or nears a provider's time limit. See what a checkpoint holds.
Where it is kept
In Portal's own storage, in a location close to the machine that writes it: object storage in the same cloud region for a job on a large cloud, and the nearest storage location otherwise. Moving a checkpoint between providers can add data transfer charges from the provider the machine runs on.
How it is protected
- Each job gets an encryption key of its own when it is created, and its checkpoints are encrypted with it before they leave the machine. An Owner can turn this off in Settings (
encrypt_capsules); the choice is fixed per job when the job is created. - Every transfer uses TLS.
- A checkpoint is verified when it arrives: what arrives must equal what left. A checkpoint that fails is never used.
How long Portal keeps what
| Item | Kept | Then |
|---|---|---|
| A suspended job's checkpoint | 3 days from the suspension, by default | an expiring event 18 hours before; then the job fails, and its checkpoints are kept 7 more days |
| Checkpoints of a job that finished, or that a person stopped | not kept | deleted at the end |
| Checkpoints of a job that ended any other way | 7 days | deleted |
| A failed or aborted move's checkpoint | 1 hour after the move ended | deleted, unless an ended job keeps it |
| A checkpoint that failed its verification | 1 day | deleted |
| A job's output | 7 days from its save at the job's end | deleted; downloads answer that it expired |
| A job's log | 30 days from when each part was written | deleted; reads answer "log expired" |
| An uploaded project that no job took | 1 day | deleted |
| A job's own copy of its uploaded project | until the job ends | deleted |
| A job in your history | until you delete it with portal rm | its costs still count in totals |
Continuing an ended job from its kept checkpoints is on the Coming soon page.