Design checkpointing / restart

pitch-370001 · in Prescribed SST

Status: Thinking — at the foot of the hill Shaping — halfway up the hill Ready — at the top of the hill In progress — halfway down the hill Done — at the bottom, over the hill Shelved — off the hill Somebody has written this down as possible work — nobody has started shaping it yet.
Priority: Medium
Appetite: — A budget in person-weeks, not an estimate: what this is worth, not how long it takes.
Tags: icon4py, checkpointing, restart, io
Owner: jcanton
Assignees: —
Reviewers: —
Review waived: —
Parent: Prescribed SST
Blocked by: —
PRs: —
Cycle: — A bet is for one cycle. Work that overruns is re-bet, not extended by default.
Start date: —
End date: —
Scheduled: —
Blocks: —

Problem

Currently ICON4Py does not have checkpoint / restart capabilities

Appetite

2 weeks at most because we currently do not really need restart capabilities: the simulations we are running are small and short enough to fit in the 24 hours wall clock limit.

Solution

Evaluate the restart from ibm02 branch which implements serial 2-checkpoint restart. If found useful, port that to the current main (write a dedicated Config. Add distributed restart in this cycle already? use driver_io or something different dedicated to restart?), otherwise design a new solution. Investigate whether there are libraries dedicated to this instead of implementing our own from scratch ("I" did not find anything useful).

Design checkpointing / restart

Problem

Appetite

Solution

Rabbit holes

No-gos