Scientific validation

pitch-370005 · in warm_bubble

Status: Thinking — at the foot of the hill Shaping — halfway up the hill Ready — at the top of the hill In progress — halfway down the hill Done — at the bottom, over the hill Shelved — off the hill A decision, not a failure: we looked, and we are not doing this.
Priority: Medium
Appetite: 8.0 person-weeks A budget in person-weeks, not an estimate: what this is worth, not how long it takes.
Tags: icon4py, scientific-validation, tracer-advection, convergence, torus, testing
Owner: jcanton
Assignees: jcanton, OngChia
Reviewers: msimberg, anuragdipankar
Review waived: —
Parent: warm_bubble
Blocked by: Plug in turbulence - TMX
PRs: C2SM/icon4py#580, C2SM/icon4py#597, C2SM/icon4py#625
Cycle: 37 A bet is for one cycle. Work that overruns is re-bet, not extended by default.
Start date: 14.07.2026
End date: —
Scheduled: —
Blocks: —

Problem

We currently rely solely on fortran serialized data for the verification of ICON4Py.

Pros:

We can compare full fields and pinpoint problems accurately. On the contrary, the scientific tests described here only provide coarser (global or 1D) measurements
We can test individual stencils/components in isolation

Cons:

ICON4Py is tied to ICON (fortran)
- whatever bug is in ICON is also in ICON4Py
- the code doesn't have its own idependence / identity: it's always seen in comparison to
Tests are slow(er) than simply running ICON4Py
Requires generation, hosting and maintenance of serialized data

Appetite

One cycle.

Solution

A. reference experiments

Draft a list of scientific testcases commonly used for verification of atmospheric models, e.g. WK82; 2D and 3D mountains.
Select and extract data such that it can be used in automated tests, using web plot digitizer Screenshot 2026-06-23 at 13.36.48 (external image)
Implement tests comparing with said data. Reasoning about meaningful distance measurements and allowed discrepancies.

B. conserved quantities

Add tests (or add to existing tests) that verify that global conserved quantities (total energy, internal energy, mass) are actually conserved or evolve in space/time as they should.

C. numerical tests

Implement experiment-independent tests, e.g. spatial/temporal convergence and measure and verify orders. Some work was started on this and should be at a good stage

https://github.com/C2SM/icon4py/pull/625
https://github.com/C2SM/icon4py/pull/580
https://github.com/C2SM/icon4py/pull/597

D. evaluate scores

scores is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models. Evaluate if it makes sense for our needs.

Rabbit holes

If using AI to extract data, absolutely plot it on top of the reference plots to verify correct extraction (last time I tried it was trash).

No-gos

List of scientific testcases with increasing model's complexity by Anurag:

Dynamical core
- Jablonowski-Williamson baroclinic instability testcase. doi: 10.1256/qj.06.12
Dynamical core + Tracer advection
- A class of deformational flow test cases for linear transport problems on the sphere. doi:10.1016/j.jcp.2010.08.014. ICON testcase identifier: DF1-4
- Schär 2D mountain testcase (Scha ̈r et al. (2002)). Tests advection and grid.
Dynamical core + Turbulence
- Dry convective boundary layer simulations: several papers. See Moeng, C.-H. (1984), A large-eddy-simulation model for the study of planetary boundary layer turbulence, J. Atmos. Sci., 41, 2052–2062, for example. ICON identifier: CBL
Dynamical core + Tracer advection + Turbulence + Microphysics
- Moist convective boundary layer. See On the growth of layers of nonprecipitating cumulus convection, J. Atmos. Sci., 64, 2916–2931 and Dipankar et al. (2015). It is an easy extension to the dry case but also can be skipped in favour of the next. ICON identifier: CBL_flxconst
- Warm bubble experiment following Weismann & Klemp 1982. ICON identifier wk82 I think.
Dynamical core + Tracer advection + Turbulence + Microphysics + Radiation
- APE is a testcase but I'd skip it if we can get LAND ready by then

Progress

Reported at the review of 11.08.2026, in the deck's own words.

Horizontal and vertical tracer advection convergence tests based on David's PRs #580 and #625.
The tests run the driver with tracer advection only. Torus grid will be generated internally (see next slide).
Time-dependent velocity field is postponed until component? Is ready.

Torus grid generator (claude) — not purely an LLM port:

specify num rows/cols instead of domain extents
can do triangle bisection, similarly to RnBx/y/z on icosahedron
Short: ~500 lines including the usual verbose comments (to be shortened)

What this bet delivered, and what it did not

The bet was placed at the cycle-37 table and was not re-bet at cycle 38. It is shelved rather than done, because only one of its four sections was touched.

Delivered, from section C (numerical tests): David's #580, the convergence study in tracer advection, merged on 11.09.2026. #597 and #625 are still open, and the work around them carries on outside a pull request on the advection_convergence branch, which jcanton, OngChia and nfarabullini have each moved forward.

Not started: section A (reference experiments and the data extracted from their plots), section B (conserved quantities) and section D (evaluating scores). The one piece of section A that exists is OngChia's own list of the test cases to build, at https://c2sm.github.io/icon4py-knowledge/personal/OngChia/scientific-validation

Where it continues: pitch-ae33ab carries the warm-bubble slice of this. The rest is unshaped and will need a new bet.

## Problem

We currently rely solely on fortran serialized data for the verification of ICON4Py.

Pros:
- We can compare full fields and pinpoint problems accurately. On the contrary, the scientific tests described here only provide coarser (global or 1D) measurements
- We can test individual stencils/components in isolation

Cons:
- ICON4Py is tied to ICON (fortran)
    - whatever bug is in ICON is also in ICON4Py
    - the code doesn't have its own idependence / identity: it's always seen in comparison to
- Tests are slow(er) than simply running ICON4Py
- Requires generation, hosting and maintenance of serialized data

## Appetite

One cycle.

## Solution

### A. reference experiments

1. Draft a list of scientific testcases commonly used for verification of atmospheric models, e.g. [WK82](https://journals.ametsoc.org/downloadpdf/view/journals/mwre/110/6/1520-0493_1982_110_0504_tdonsc_2_0_co_2.pdf); [2D](http://journals.ametsoc.org/doi/10.1175/1520-0493(2002)130<2459:ANTFVC>2.0.CO;2) and [3D](https://onlinelibrary.wiley.com/doi/abs/10.1002/qj.2378) mountains.
2. Select and extract data such that it can be used in automated tests, using [web plot digitizer](https://automeris.io/)
![Screenshot 2026-06-23 at 13.36.48](https://hackmd.io/_uploads/r1j-xguzfg.png)
3. Implement tests comparing with said data. Reasoning about meaningful distance measurements and allowed discrepancies.

### B. conserved quantities

Add tests (or add to existing tests) that verify that global conserved quantities (total energy, internal energy, mass) are actually conserved or evolve in space/time as they should.

### C. numerical tests

Implement experiment-independent tests, e.g. spatial/temporal convergence and measure and verify orders.
Some work was started on this and should be at a good stage
- https://github.com/C2SM/icon4py/pull/625
- https://github.com/C2SM/icon4py/pull/580
- https://github.com/C2SM/icon4py/pull/597

### D. evaluate scores

[scores](https://scores.readthedocs.io/en/stable/index.html) is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models.
Evaluate if it makes sense for our needs.

## Rabbit holes

If using AI to extract data, absolutely plot it on top of the reference plots to verify correct extraction (last time I tried it was trash).

## No-gos

## List of scientific testcases with increasing model's complexity by Anurag:

- Dynamical core
    - Jablonowski-Williamson baroclinic instability testcase. doi: 10.1256/qj.06.12

- Dynamical core + Tracer advection
    - A class of deformational flow test cases for linear transport problems on the sphere. doi:10.1016/j.jcp.2010.08.014. ICON testcase identifier: DF1-4
    - Schär 2D mountain testcase (Scha ̈r et al. (2002)). Tests advection and grid.

- Dynamical core + Turbulence
    - Dry convective boundary layer simulations: several papers. See Moeng, C.-H. (1984), A large-eddy-simulation model for the study of planetary boundary layer turbulence, J. Atmos. Sci., 41, 2052–2062, for example. ICON identifier: CBL

- Dynamical core + Tracer advection + Turbulence + Microphysics
    - Moist convective boundary layer. See On the growth of layers of nonprecipitating cumulus convection, J. Atmos. Sci., 64, 2916–2931 and Dipankar et al. (2015). It is an easy extension to the dry case but also can be skipped in favour of the next. ICON identifier: CBL\_flxconst
    - Warm bubble experiment following Weismann & Klemp 1982. ICON identifier wk82 I think.

- Dynamical core + Tracer advection + Turbulence + Microphysics + Radiation
    - APE is a testcase but I'd skip it if we can get LAND ready by then

## Progress

Reported at the review of 11.08.2026, in the deck's own words.

- Horizontal and vertical tracer advection convergence tests based on David's PRs #580 and #625.
- The tests run the driver with tracer advection only. Torus grid will be generated internally (see next slide).
- Time-dependent velocity field is postponed until component? Is ready.

Torus grid generator (claude) — not purely an LLM port:

- specify num rows/cols instead of domain extents
- can do triangle bisection, similarly to RnBx/y/z on icosahedron
- Short: ~500 lines including the usual verbose comments (to be shortened)

## What this bet delivered, and what it did not

The bet was placed at the cycle-37 table and was not re-bet at cycle 38. It is
shelved rather than done, because only one of its four sections was touched.

Delivered, from section C (numerical tests): David's #580, the convergence study
in tracer advection, merged on 11.09.2026. #597 and #625 are still open, and the
work around them carries on outside a pull request on the `advection_convergence`
branch, which jcanton, OngChia and nfarabullini have each moved forward.

Not started: section A (reference experiments and the data extracted from their
plots), section B (conserved quantities) and section D (evaluating `scores`).
The one piece of section A that exists is OngChia's own list of the test cases to
build, at
https://c2sm.github.io/icon4py-knowledge/personal/OngChia/scientific-validation

Where it continues: pitch-ae33ab carries the warm-bubble slice of this. The rest
is unshaped and will need a new bet.