Optimize the standalone-driver startup time on GPU

task-802688 · in Runtime improvements

Status: Thinking — at the foot of the hill Shaping — halfway up the hill Ready — at the top of the hill In progress — halfway down the hill Done — at the bottom, over the hill Shelved — off the hill Shaped, sized and argued for — waiting for a betting table to put it in a cycle.
Priority: Low
Appetite: 4.0 person-weeks A budget in person-weeks, not an estimate: what this is worth, not how long it takes.
Tags: distributed
Owner: iomaganaris
Assignees: iomaganaris
Reviewers: jcanton
Review waived: —
Parent: Runtime improvements
Blocked by: —
PRs: —
Cycle: 38 A bet is for one cycle. Work that overruns is re-bet, not extended by default.
Start date: —
End date: —
Scheduled: 2026-09-24 → 2026-11-18 · overruns cycle 38 by 8.3 weeks
Blocks: —
Progress: 0/1 items

Problem

The standalone-driver of icon4py (for the JW experiment at least) seems to be doing a lot of pack and forth with GPU memory that consumes a lot of time before the actual dycore execution. See the example here: https://c2sm.github.io/icon4py-knowledge/personal/iomaganaris/standalone-driver-startup-opt/standalone-driver-JW-nsys.png The selected area with green shows the time spent in anything before DaCe calls that suggest dycore execution.

Solution

Figure out where these copies come from using NVTX markers in Python and eliminate/reduce them.

Progress

Not started. The shaping note is: https://c2sm.github.io/icon4py-knowledge/personal/iomaganaris/standalone-driver-startup-opt/standalone-driver-startup-opt