Spot interruptions and checkpoints

What happens on a notice

1. The provider tells the machine it will be taken. Portal's node on the machine sees the notice and reports it at once.
2. Portal asks your program to pause at its next safe point: a point between statements, in your script's own code, where its state can be saved. It keeps the last minute of the notice for saving and uploading.
3. The program's state is saved as a checkpoint and uploaded to storage, encrypted with a key of the job's own.
4. The machine is released and the job is suspended. The pool that reclaimed it is left out of new placements for 30 minutes.
5. You resume the job when you choose, on a machine you choose:

portal resume JOB
portal resume JOB --gpu L4
portal resume JOB --on SPEC

It continues from the statement after the safe point. No completed work is done again.

If the node loses contact with Portal during a notice, it saves the checkpoint on its own. A job is called lost only when its state is gone for certain: a provider that once says the machine is gone, or a store that cannot be read for a moment, does not end it. While its node does not answer, the job keeps its state with the time its node was last heard, and each event says what Portal waits for: a provider that says the machine is gone must say so again at the next check, about a minute later, with the node still silent, and the loss is declared no sooner than 10 minutes of silence (2 minutes past a notice's deadline). A suspend or move asked meanwhile is refused with the same words, never with "the job goes on". An Azure VM's bill ends when Azure's records say it evicted or deleted the VM, not when Portal declares the loss, and so does a checkpoint the eviction cut short; a checkpoint that is incomplete is said to be not resumable and is kept for inspection.

What a checkpoint holds

• objects reachable from your script's globals and from the locals of each paused function, with shared references kept;
• tensors and arrays, put back on a GPU when the new machine has one;
• functions and classes your program defined;
• loop positions: iterators, DataLoader positions and tqdm bars;
• random number generators (Python, NumPy, torch) and torch's numeric settings;
• interpreter settings your program changed: logging, warnings filters, print options, the decimal context, signal handlers;
• the current directory, sys.path, environment variable changes, and the files in the working directory.

What arrives must equal what left, checked by content fingerprints. If it does not, the new machine never takes over and the checkpoint is kept for diagnosis.

What a checkpoint cannot carry

• the process itself: open files, sockets, locks, pipes, shared memory, other threads and processes, network requests in flight. A pause waits while your program holds one of them in a global or a paused local;
• a native call in progress: one long C, CUDA or Rust call, or a library's own loop, runs to its end before the program can pause;
• files outside the working directory, and installed packages (the new machine installs your requirements again);
• a generator expression, coroutine or comprehension partway through, or code inside an except or finally block;
• an exception's traceback.

Imports run again on the new machine: their side effects outside your program's state (a network call, a file written outside the working directory) happen twice. Results on a different GPU model can differ in the last bits.

Long notices, short notices

A checkpoint takes the time to reach a safe point plus the time to save and upload the state. On AWS's 2 minutes that is enough for most jobs. On a notice of 30 s or less (Google Cloud's 25 s fallback, Azure spot VMs' 20 s), a large state may not make it: prefer on-demand machines for such jobs, or keep the state small. The plan warns when a spot offer's notice cannot save the job's state. portal explain JOB shows what happened at each notice.

The checkpoint check

A notice is the worst moment to learn that a checkpoint is impossible. So at a job's first pause point, and then every 10 minutes (the setting capture_probe_every_min), Portal's node walks the job's live state the way a checkpoint would, without saving anything, and reports what a checkpoint could not carry. The script pauses for the walk (5 seconds in all; a larger state is reported as partially checked) and goes on. The check never holds up a notice or a move: a pause asked for while it runs stops it at once and is taken at the same pause point, within a fraction of a second.

What you see:

• portal run and portal logs -f print the warning as a note when it appears; the console's run page shows it as a banner, and the Overview line "Checkpoint check" says when the last check passed; portal status JOB and the MCP's status(job) carry it as carry (ok, problems, blocked, warning); portal explain JOB lists it as a warn decision (policy spot on a spot machine, checkpoint elsewhere).
• A problem names the value as your script knows it, its type and why: A checkpoint of this job cannot carry digest (_hashlib.HASH: TypeError: cannot pickle '_hashlib.HASH' object). Portal cannot move or suspend it while it holds them, and it cannot be saved on a notice. If this spot machine is reclaimed (its notice is 120 s), its work since the last checkpoint would be lost. Create them again after a pause point instead of keeping them across one (checked at train.py:42 in train). One warning per machine and problem; an all-clear event follows once a later check passes.
• A script that passes no pause point for 30 seconds (an open file or a running thread held in a global, one long native call) is reported as blocked, with the reasons the pause points gave.

What the check does not do: it never moves or suspends the job for this, because a job that holds such a value cannot be moved or suspended at all. The fix is in the script: create the object after a pause point instead of keeping it across one, close files and sockets when they are not in use, or hold them inside a with block that ends before the next loop iteration.

Settings: capture_probe is on by default. full also rehearses a real checkpoint on the machine's disk (deleted after) and reports its size and the time it took, which is the most faithful test of a job's checkpoint; it costs that time and disk at every check, so use it to try a job out rather than for every run. The rehearsal runs on on-demand machines only: a real checkpoint cannot be cut short, so on a spot machine (or one whose purchase model the node was not told) the check runs without it, and the plan and the job's events say so. off skips the check. A job's own environment variable PORTAL_CAPTURE_PROBE overrides the setting for that job, and PORTAL_CAPTURE_PROBE_EVERY_S its interval.

Limits of the check: a script that stops its own child process only when a real pause is wanted (portal.runtime.rt.pending()) looks blocked to the check while that child runs, since the check wants no pause; a value whose pickling has side effects runs them at every check, as it would at every move.

State size and spot

What a spot machine can save within its notice is a bound Portal measured per provider, machine class and storage route (R x (notice - t0) x 0.5; the plan's notice field and the sentence under each spot offer say it). A brand-new job is placed on spot with a warning when the memory it asks for is above that bound, because what it asks for is a guess. Once Portal has measured the job's state, the measure decides:

• The bound depends on the machine's size. Portal takes a named type's vCPUs from its own catalog, else from the provider's listing of the type (EC2's instance types, Azure's SKUs, Lambda's instance types); a size Portal cannot look up is sized as its default class, and the offer's note and the plan say so.
• The state's size is a measure with a name. Portal takes a checkpoint's bytes when it has them (a suspended job's checkpoint as stored, the check's rehearsal, the last move's capsule, each with a tenth added for growth since); otherwise the memory the node reports (resident and GPU memory), which overstates a checkpoint, so that figure must be a quarter over the bound before Portal moves on it alone. Every warning, plan line and event says which figure it used ("its checkpoint as stored", "its resident and GPU memory now", "what it asked for").
• A resume, a move, and the cheaper and return policies leave out every spot offer whose notice cannot save that state. The job's events say spot aws c7i.large (us-east-1) skipped: this job's state (12.0 GB, its checkpoint as stored) exceeds what its 120 s notice can save (about 7.9 GB), and a plan (portal resume --plan, portal move --plan) says would not use spot ... under the D6 line. An exact --on is your choice and is kept, with the warning.
• A job running on spot whose measured state grows past what its notice can save is moved to the cheapest on-demand machine that fits: the spot policy provisions it ahead and moves the job once it is ready (portal explain JOB: provision then move under spot, the event naming the figure, what it is and the notice's bound). The measure is taken again at every check, so a job that grows after its first warning is caught. Turn this off with spot.move_when_over: false; a job pinned to an exact spot machine is warned and left where it is.
• A job that asks for spot only and finds no spot offer that can save its state stays suspended; its event names the way out: resume it with --on an exact machine, or on-demand (portal resume JOB --no-spot).