Jablonowski-Williamson baroclinic wave test case produces NaN on first timestep in v8.3.0 and later (works in v8.2.3)

[ac-neuralterrena]

Summary

The official Jablonowski & Williamson baroclinic wave test case produces NaN in all
prognostic fields within the first timestep on MPAS v8.3.0 and later. The same case,
with the same unmodified namelists and mesh, runs correctly on v8.2.3 and v8.2.2.

The model exits with status 0 and reports zero errors, zero critical errors and zero
warnings, so the failure is not visible without inspecting the output values.

Versions

Bisected across releases, using the unmodified test case in every run:

Version	Result at first output after t=0
v8.2.2	OK
v8.2.3	OK
v8.3.0	100% NaN
v8.4.0	100% NaN
v8.4.1	100% NaN

The change is therefore between the v8.2.3 and v8.3.0 tags.

Reproducer

curl -O https://www2.mmm.ucar.edu/projects/mpas/test_cases/v7.0/jw_baroclinic_wave.tar.gz
tar xzf jw_baroclinic_wave.tar.gz
cd jw_baroclinic_wave

# link init_atmosphere_model and atmosphere_model built from v8.4.1
./init_atmosphere_model     # exit 0
./atmosphere_model          # exit 0

Nothing in namelist.init_atmosphere, namelist.atmosphere, streams.* or the
x1.40962 mesh was modified. The only change made for convenience was shortening
config_run_duration from 16_00:00:00 to 1_00:00:00; the failure occurs on the
first timestep either way.

Observed behaviour

import netCDF4, numpy as np
d = netCDF4.Dataset("output.nc")
th = d.variables["theta"]
for t in range(th.shape[0]):
    a = np.array(th[t])
    print(t, f"NaN={np.isnan(a).mean()*100:.2f}%")

0 NaN=0.00%
1 NaN=100.00%

theta, pressure, rho and u are all 100% NaN; w is 96.3% NaN with the
remainder exactly zero. zz (time-invariant) stays finite.

With output every 30 minutes and config_dt = 1800, the damage is already present
after a single integration step: 60.9% of theta is NaN at t = 00:30, rising to 100%
by t = 01:00.

Spatial pattern after one timestep

Uniform in the vertical, monotonic in latitude:

NaN fraction by vertical level:  60.9% at every one of the 26 levels

NaN fraction by latitude band:
  -90..-60   100.0%
  -60..-30   100.0%
  -30..  0    74.2%
    0..+30    52.1%
  +30..+60    24.0%
  +60..+90     0.0%

Whole columns fail together, and the affected fraction increases monotonically from
the north pole to the south pole.

Ruled out

Each of these was tested individually against v8.4.1; all still produce NaN:

• config_dt reduced from 1800 to 900, 450 and 150 s — identical result, so this does
  not behave like a CFL violation
• config_visc4_2dsmag reduced to 0.01 and to 0.0
• config_horiz_mixing = 'none'
• config_len_disp omitted (derived from nominalMinDc) and set explicitly
• config_init_case = 1 (no perturbation) as well as 2
• PRECISION=double as well as the default single
• -O0 as well as -O3
• config_epssm is deprecated in 8.4.1; setting the level-dependent equivalent
  (config_epssm_minimum = config_epssm_maximum = 0.1, transition above model top) to
  reproduce the old config_epssm = 0.1 behaviour
• config_hybrid_coordinate = false in &vertical_grid (the v7.0 test case supplies
  no &vertical_grid group, so this defaults to true on 8.3.0+)

Environments

Reproduced on two independent toolchains:

macOS 26.5, arm64 (Apple M5 Pro)

• GNU Fortran (Homebrew GCC 16.2.0) 16.2.0
• MPICH 5.0.1
• PnetCDF 1.15.0, SMIOL
• CC_SERIAL=gcc-16 CXX_SERIAL=g++-16 (on macOS, gcc is Apple Clang, which rejects
  -fopenmp)

Debian trixie, aarch64 (container)

• GNU Fortran (Debian 14.2.0-19) 14.2.0
• Open MPI (Debian), PnetCDF 1.14.0, SMIOL
• built with the stock gfortran target

On Linux, gfortran's end-of-run summary reports
IEEE_INVALID_FLAG IEEE_UNDERFLOW_FLAG. On macOS no such notice is printed, so there
is no indication at all that anything went wrong.

Secondary observation: the runtime diagnostic hides the failure

With config_print_global_minmax_vel = true, the log prints:

Begin timestep 2010-10-23_00:00:00
 global min, max w -0.203183E-02 0.262211E-02      <- step 1, healthy
Begin timestep 2010-10-23_00:30:00
 global min, max w  0.00000      0.00000           <- step 2 onward
 global min, max u  0.00000      0.00000

and repeats 0.00000 0.00000 for every remaining step. At that point w is 96.3%
NaN. Because every comparison against NaN is false, the running min/max never updates
and retains its initial value of zero.

The practical consequence is that a diverged field is displayed identically to a
perfectly quiescent atmosphere, which reads as the opposite of a problem. A NaN count
taken before the reduction would make this visible; this is independent of the
regression itself and would be worth guarding regardless of the outcome here.

Not investigated

The root cause. The bisection above narrows it to the v8.2.3 → v8.3.0 range but no
further; I did not bisect individual commits.