Driver Parallel Output

pitch-370007 · in warm_bubble

Status: Thinking — at the foot of the hill Shaping — halfway up the hill Ready — at the top of the hill In progress — halfway down the hill Done — at the bottom, over the hill Shelved — off the hill Built and finished inside its cycle — anything still wanted from it is a new bet.
Priority: Medium
Appetite: 4.0 person-weeks A budget in person-weeks, not an estimate: what this is worth, not how long it takes.
Tags: icon4py, standalone-driver, io, parallel-output, netcdf, zarr, distributed
Owner: kotsaloscv
Assignees: kotsaloscv
Reviewers: jcanton, msimberg
Review waived: —
Parent: warm_bubble
Blocked by: —
PRs: C2SM/icon4py#1307, C2SM/icon4py#1403
Cycle: 37 A bet is for one cycle. Work that overruns is re-bet, not extended by default.
Start date: 14.07.2026
End date: 17.08.2026
Scheduled: 2026-07-14 → 2026-08-17 · overruns cycle 37 by 1.0 weeks
Blocks: Async distributed output
Progress: 1/2 items

Problem

The standalone driver has serial output only. The goal is to extend the output to parallel one.

Solution

Parallel Output:

NetCDF: gather all in rank 0 and output
Same for Zarr
Zarr: Each rank outputs its own part [no ghost cells]
parallelNetCDF: like above

Rabbit holes

No-gos

Progress

Serial IO (PR#1307)
Parallel IO: Follow-up PR [WIP]

The checklist above is the author's, left as he wrote it. The review deck of 11.08.2026 reported the follow-up PR #1403 as "To be Merged - very soon"; it has since been merged (2026-08-17). The deck's own summary of where that leaves the driver:

Gather to rank 0 and write from there either netCDF or Zarr (v3)
Every rank writes its own block of every field in a common store either netCDF (parallel netCDF) or Zarr
Next & Final step → Async Distributed Output (dedicated ranks for the output)

Since the review

That final step is its own bet now, pitch-9c4e12, carried into cycle 38. The gridlook export that was split out of #1403 along the way is unrelated to output distribution and is parked in Miscellaneous as task-7a3f05.

## Problem

The standalone driver has serial output only. The goal is to extend the output to parallel one.

## Solution

Parallel Output:
1. NetCDF: gather all in rank 0 and output
2. Same for Zarr
3. Zarr: Each rank outputs its own part [no ghost cells]
4. parallelNetCDF: like above

## Rabbit holes

## No-gos

## Progress

- [x] Serial IO ([PR#1307](https://github.com/C2SM/icon4py/pull/1307))
- [ ] Parallel IO: Follow-up PR [WIP]

The checklist above is the author's, left as he wrote it. The review deck of
11.08.2026 reported the follow-up PR
[#1403](https://github.com/C2SM/icon4py/pull/1403) as "To be Merged - very soon";
it has since been merged (2026-08-17). The deck's own summary of where that
leaves the driver:

- Gather to rank 0 and write from there either netCDF or Zarr (v3)
- Every rank writes its own block of every field in a common store either netCDF (parallel netCDF) or Zarr
- Next & Final step → Async Distributed Output (dedicated ranks for the output)

## Since the review

That final step is its own bet now, pitch-9c4e12, carried into cycle 38. The
gridlook export that was split out of #1403 along the way is unrelated to output
distribution and is parked in Miscellaneous as task-7a3f05.