- No Rabbit holes section. The pitch template asks for one — it is what keeps the appetite honest.
Problem
The driver with the physics driver enabled -- muphys and tmx, the stack
#1359/#1436/#1360 -- is validated single-rank against the v08 reference, but it
is not multi-rank correct. Physics computes and applies on owned points only and
PhysicsDriver.run performs no halo exchange, so after physics the halo rows of
every field it wrote hold pre-physics values: exner, theta_v, the six tracers,
and with tmx also w and vn. The next step's dycore and tracer advection read
those stale halos, and the single-versus-multi-rank comparison fails.
Everything around the physics driver is already multi-rank aware. The tmx granule
ports ICON's own sync points, fourteen exchanges including its u,v input sync;
tracer advection exchanges its output; the entry diagnostics come out halo-valid
by construction. What is missing belongs to the physics driver alone.
Appetite
Two weeks, correctness only.
Solution
Three pieces, all in PhysicsDriver, as another PR on the #1359/#1436/#1360
stack:
- Wiring --
PhysicsDriver takes an ExchangeRuntime and driver_utils passes
the driver's existing exchange object.
- End-of-run bundled sync -- after
ApplyToPrognostic, one exchange per
dimension over exactly the fields apply wrote. No per-process declaration
protocol: under apply-once the writer's return value is the declaration.
- Pre-projection tendency sync -- before apply, exchange the accumulated
tend_u/tend_v. The cells-to-edges projection reads first-halo-row cells
from owned edges, and wrong owned values cannot be repaired by the final sync.
Validation extends the existing single-versus-multi-rank driver comparison test
with EXCLAIM_APE_AES (v08), one step, tracers among the compared fields and
check_halos=True. At unit level, a recording exchange stub asserts the
exchanged field sets and their ordering around apply.
Rabbit holes
No-gos
Multi-node performance and scaling runs. Any change to the granules' internal
sync points -- jcanton's tmx exchanges stay as they are.
Progress