Moves, suspend and resume
What Portal does on its own
The policies that act on a running job without being asked, their defaults, and what Portal never does.
Warnings and limits
- Portal moves a running job to a cheaper offer on its own when the saving is at least 20% and at least 3 times the move's cost. Turn it off per job with
--no-cheaper, or for everyone in Settings. - A job is suspended after 30 minutes idle (no progress and CPU under 5% of a core) and stopped after 72 hours of running time. Both are on by default.
- The time limit (
ttl, 72 hours by default) counts the time machines run the job; a suspended job is not stopped by that count. While suspended, the same number of hours counted on the clock from the job's creation ends it instead, with its checkpoint kept 7 days forportal resumeto continue it; and its checkpoint expires 3 days after the suspension by default, when the job fails. See Suspend and resume. - A job started with an exact
--onis pinned: no cheaper, return or resizing moves, and no move off spot for its state size (it is warned instead). Notices, out-of-memory rescues, provider time limits, budgets, idleness and the time limit still apply. - A job running on spot whose measured state grows past what its machine's notice can save is moved to an on-demand machine (
spot.move_when_over, on by default), and a spot offer whose notice cannot save a job's measured state is left out of that job's later placements. Every 10 minutes Portal also checks whether a checkpoint could carry the job's state and warns when it could not; it never moves a job for that, since such a job cannot be moved. See Spot interruptions and checkpoints. - Portal never resumes a suspended job on its own, never powers off your own machine, never requests quota and never spends past a connection's cap.
- Every move and suspension is strict, automatic or your own: a value the checkpoint cannot carry refuses it, with the value's name, and the job stays where it is.
The policies
Portal checks each running job regularly and makes at most one decision that changes where it runs per check, in this order:
| Policy | Default | What it does |
|---|---|---|
| ttl | on, 72 hours | stops a running job after this much running time (time suspended does not count); ends a suspended job this many hours after its creation on the clock, its checkpoint kept 7 days for a resume to continue it |
| spot | on | on a spot or preemption notice, checkpoints and suspends the job; while the job runs on spot and its measured state is above what the notice can save, moves it to on-demand (move_when_over) |
| budget | on | suspends the job a little before its budget is spent (or stops it, if set to) |
| oom | on | on a recoverable out-of-memory error, checkpoints and suspends the job |
| idle | on, 30 minutes | suspends a job that made no progress and used under 5% of a CPU core for that long |
| lifetime | on, 30 minutes | checkpoints and suspends a job 30 minutes before a provider's fixed limit (Modal and Azure Container Apps 24 hours, Google Cloud VMs 24 hours by default) |
| lease | on | keeps each machine's hard stop about an hour ahead, renewed every 10 minutes |
| return | on | while a job runs off spot, looks every 10 minutes for a spot offer and moves there when it saves at least 20% and 3 times the move's cost |
| cheaper | on | looks every 10 minutes for a cheaper offer and moves there on the same terms |
| rightsize | on a move | when the job moves, sizes CPU and memory from its measured peaks with 1.3 times headroom, never below what it asked for |
After a move, a job waits 30 minutes before the next return or cheaper look. A reclaimed spot pool is left out for 30 minutes. Every decision is in portal explain JOB.
Set them for every job in the console's Settings, per project in portal.yaml, or per job with portal run flags. See Settings and Run a job.
What Portal never does
- Resume a job on its own. After a notice, an out-of-memory rescue, a budget or idleness, the job waits suspended until you resume it; its time limit keeps counting on the clock while it waits, and ends it with the checkpoint kept.
- Power off or reboot your own machine. Stopping a job there ends only the job.
- Request quota, or change your account's limits or access.
- Spend past a connection's cap. A machine whose worst-case cost would pass the cap is refused, and a running job is suspended when the cap would be reached.
- Keep a machine without a hard stop. Every machine has a lease that ends it if Portal stops renewing it.
- Delete your data in your cloud. It terminates only the machines it started and tagged.