Agreement Is the Null
Engine comparisons, rethought: robustness, accuracy, and the floor between them
v1.0.0 · 2026-10-03

When the physics forces two engines to the same answer, their agreement describes the network. The finding is where they part.

EXTRAN drew fifty years of complaints about robustness and kept matching other models on accuracy. Those are different properties. Most sewer flow is set by node continuity, Manning friction and the boundary conditions, so any scheme that conserves mass lands close to the same place. A comparison that reports only how close two engines are has measured the regime, and nothing yet about either solver.

Predict before you look

EXTRAN manual example 1, nine conduits, three of them pressurizing. Switch the inertial terms off entirely (INERTIAL_DAMPING FULL in place of NONE). The median conduit moves by some amount. Expressed as the uniform change in Manning's n that moves it just as far, how big is that?

One model, eight differences, one scale

Six shifts in how to compare engines

On Monday: run six, not two. A alone, A again, A at half the step, B, B at half the step, and A with every conduit's n raised 5%. Two runs give a number. Six tell you what the number is made of. The checklist and the file slots are on the Your runs tab.
This tab teaches one thing: count what the test network contains before you say what a comparison covers. The engine already writes the count. Its Flow Classification Summary gives the fraction of time each conduit spends in each flow class.

Where each model spends its conduit-hours

Each bar is the mean over conduits of the time fractions in the .rpt of the most refined baseline run. "Pressurizes" counts conduits with either end full for at least 1% of the run.

The sentence the census forces

Agreement by regime, every conduit in the sample

Census your own model

Load a SWMM5 .rpt from a dynamic wave run. Nothing leaves your device.

This tab teaches one thing: a difference between two runs is two gaps added together. The discretization gap falls when you refine the step. The formulation gap stays where it is. The engine's own quality numbers belong on a separate card.

Refine the step: one gap falls, the other holds

Robustness card: what the engine says about itself

From each run's own .rpt. Baseline configuration, fixed step.

Accuracy card: distance from the most refined run

Conduits only, every reporting period, each conduit normalised to its own peak. Median over conduits.

Formulation switches against the floor

Convention used here: a gap under 1× the floor is below it, 1× to 3× is marginal, 3× or more is resolved. The floor is the difference between the two finest baseline steps, same metric.

This tab teaches one thing: state an engine difference as the change in Manning's n that moves the same model just as far. Then set it beside how well you know n.
n_eq = n-change at which the model's own response equals the gap
Response = median over conduits of the chosen metric, baseline vs baseline with every conduit's n multiplied by 1.01, 1.02, 1.05 and 1.10. Linear between measured points, through the origin.

The model's response to roughness, with the gaps placed on it

Every model, every gap, in percent of n

This tab teaches one thing: a comparison is six runs. Load what you have. Two .out files give the difference; each extra file adds one line to the verdict.

The six runs

  1. A, the baseline engine or option.
  2. A again. The .out must be byte-identical. If it is not, stop.
  3. A at half the routing step, fixed step. A against this is the floor.
  4. B, the other engine or option, same step as A.
  5. B at half the step. Compare the two refined runs to see what survives refinement.
  6. A with every conduit n × 1.05. Prices the gap.

Set REPORT_STEP near the routing step and report all nodes and links, or the files have nothing to compare.

Load .out files

A.outrequired
B.outrequired
A at half step .outoptional: gives the floor
B at half step .outoptional: needs A at half step too
A with n raised .outoptional: prices the gap. n multiplier
A.rptoptional: splits the result by regime

Files are read in your browser and go nowhere. Conduits only; flow floor 1 L/s in the file's units; depth floor 0.001.

This tab teaches one thing: agreement between engines that share code is regression testing. Only an independent formulation can validate. Pick two engines and read what their agreement would be evidence of.

The pair matrix

The scheme summaries on this tab are recalled from published documentation. Nothing on this tab was run or re-verified in this build. Check the vendor's own manual before quoting one.
This tab teaches one thing: what the numbers in this app were measured on, how to rerun them, and what they cannot tell you.

What this does not show

    Reproduction

    Controls that ran

    Observations register

    Run ledger