GPU bitwise reproducibility

pitch-370002 · in warm_bubble

Status: Thinking — at the foot of the hill Shaping — halfway up the hill Ready — at the top of the hill In progress — halfway down the hill Done — at the bottom, over the hill Shelved — off the hill Built and finished inside its cycle — anything still wanted from it is a new bet.
Priority: Medium
Appetite: 1.5 person-weeks A budget in person-weeks, not an estimate: what this is worth, not how long it takes.
Tags: icon4py, gpu, bitwise-reproducibility, distributed, standalone-driver
Owner: msimberg
Assignees: msimberg
Reviewers: jcanton
Review waived: —
Parent: warm_bubble
Blocked by: —
PRs: C2SM/icon4py#1368
Cycle: 37 A bet is for one cycle. Work that overruns is re-bet, not extended by default.
Start date: 14.07.2026
End date: 28.07.2026
Scheduled: 2026-07-14 → 2026-07-28
Blocks: —
Progress: 1/1 items

Problem

In https://github.com/C2SM/icon4py/pull/1303 we added a 7-day standalone driver validation test that expects bitwise identical results between single- and multi-rank runs (1 vs 4 ranks). However, this was only enabled for CPU backends with CXXFLAGS=-ffp-contract=off.

This project is about achieving the same for GPU backends.

Appetite

One cycle.

Solution

Unknown. Needs research into which flags affect floating point arithmetic, indeterministic reductions etc. on GPUs.

Some work already been done in ICON Fortran to achieve the same with with OpenACC/nvhpc. Find out what flags and options are used there to achieve bitwise identical results.

Rabbit holes

If bitwise identical results can't be achieved reasonably for the full standalone driver, understanding which fields are not bitwise identical would already be a useful result.

No-gos

Progress

merged https://github.com/C2SM/icon4py/pull/1368
- enables bitwise identical checks for standalone driver and most static fields
- moved rbf computation to host behind an env var, together with nvcc flags these were the source of nodeterminism on gpus

As reported at the review of 11.08.2026: standalone driver tests single- vs multi-rank runs now bitwise reproducible using GPU backends, with one caveat. CXXFLAGS="-ffp-contract=off" already added before for CPU reproducibility; added NVCC_APPEND_FLAGS="--fmad=false" for GPU reproducibility; and RBF interpolation static fields are computed on CPU with ICON4PY_DETERMINISTIC_RBF_COEFFS=1 (off by default).

## Problem

This project is about achieving the same for GPU backends.

## Appetite

One cycle.

## Solution

Unknown. Needs research into which flags affect floating point arithmetic, indeterministic reductions etc. on GPUs.

Some work already been done in ICON Fortran to achieve the same with with OpenACC/nvhpc. Find out what flags and options are used there to achieve bitwise identical results.

## Rabbit holes

If bitwise identical results can't be achieved reasonably for the full standalone driver, understanding which fields are not bitwise identical would already be a useful result.

## No-gos

## Progress

- [x] merged https://github.com/C2SM/icon4py/pull/1368
  - enables bitwise identical checks for standalone driver and most static fields
  - moved rbf computation to host behind an env var, together with nvcc flags these were the source of nodeterminism on gpus

As reported at the review of 11.08.2026: standalone driver tests single- vs
multi-rank runs now bitwise reproducible using GPU backends, with one caveat.
`CXXFLAGS="-ffp-contract=off"` already added before for CPU reproducibility;
added `NVCC_APPEND_FLAGS="--fmad=false"` for GPU reproducibility; and RBF
interpolation static fields are computed on CPU with
`ICON4PY_DETERMINISTIC_RBF_COEFFS=1` (off by default).