Moves, suspend and resume
Out-of-memory rescue
How Portal saves a job that runs out of memory, so you can resume it on a larger machine without starting over.
Warnings and limits
- Portal rescues an out-of-memory error only when Python raises it inside a loop of your script's own code. An error inside an optimizer step, inside a library's own loop, or outside any loop is not rescued, and the job fails.
- On Linux CPU machines Portal caps the program's memory a little under what the machine has (a reserve of 10%, 256 MiB to 1 GiB), so running out raises a Python
MemoryErrorthat is rescued like any other, instead of the operating system killing the process;PORTAL_MEMORY_TRIPWIRE=0in the job's environment turns the cap off. There is no cap on GPU machines: a process the operating system kills there raises no Python error, and Portal cannot rescue it. - The iteration that failed runs again from its start after the resume: its side effects (a file written, a message sent) happen twice.
- Portal suspends the job and does not pick the larger machine: you resume it on one.
- The checkpoint of a suspended job is kept 3 days.
- A rescue whose checkpoint fails twice in a row for the same reason ends the job, with the capture's own message as the error.
What counts as out of memory
- Python's
MemoryError; - PyTorch's
OutOfMemoryError; - a
RuntimeErrorthat says "CUDA out of memory".
What Portal does
- The error reaches your script's loop. Portal finds the innermost loop of your own code the error passed through and makes the start of that loop's body, with the same item, the point to continue from. The frames inside it are cleared, so the memory they held (activations, gradients) is free.
- Portal saves a checkpoint of the program at that point and suspends the job. Its machine is released.
- You resume it on a machine with more memory, or a larger GPU:
portal resume JOB --mem 64
portal resume JOB --gpu A100
portal resume JOB --on SPECThe failed iteration runs again, on the new machine, with the same item. The work before it is not done again.
When it cannot rescue
| Situation | Why |
|---|---|
The error came from inside an optimizer step (torch.optim) | the weights may be half updated |
| The error was raised outside your script's code | there is no point of yours to continue from |
| No loop of your script was running | there is no iteration to retry |
| The process was killed by the operating system | no Python error was raised |
In those cases the job fails as it would without Portal. Set the oom policy off for a job with --policy oom.on=false if you prefer that it always fails at once.