dem_benchmark

The dem_benchmark command-line tool scores many DEMs against one altimetry sample and puts the results side by side. The asp_report altimetry pages assess a single DEM; this tool compares several, for example different scene combinations, processing flows (joint multi-view triangulation vs. pairwise stereo merged with dem_mosaic), or parameter settings.

Every DEM is scored with the report’s recipe — the cached ICESat-2 ATL06-SR parquet replayed, ESA WorldCover water returns dropped, residual outliers removed per DEM — and gets:

• Coverage inside the common footprint of all the DEMs (percent valid and km²), so runs with different crop windows compare fairly.

• Triangulation error, the median and NMAD of the *-IntersectionErr.tif from point2dem --errorimage. A mosaic has none.

• Altimetry residuals (altimetry minus DEM): count, median, NMAD and RMSE, before and after a per-DEM pc_align --compute-translation-only. A translation cannot change NMAD, so the difference between the two columns is the bias that alignment removes.

• Optionally, the difference against one candidate named as the reference.

Six same-pass WorldView-2 DEMs of Atlanta — three single pairs at 5°, 22° and 27° convergence, the three pairs merged with dem_mosaic, and 3- and 5-scene multi-view runs — scored against one ICESat-2 sample and sorted best-first by post-alignment NMAD. From notebooks/WorldView/worldview_spacenet_atlanta_mvs.ipynb.

Basic usage

Point the tool at the DEMs and the ICESat-2 parquet cache that a previous asp_report run wrote next to its report:

dem_benchmark stereo_mvs3/run-DEM.tif stereo_mvs5/run-DEM.tif pairwise_mosaic-DEM.tif \
              --parquet atl06sr_all.parquet

This prints the stats table and writes dem_benchmark.png (summary figure), dem_benchmark_histogram.png (overlaid residual histograms) and dem_benchmark.csv (one row per DEM) to the working directory. An unlabelled ASP run-DEM.tif is labelled by its folder; any other DEM by its filename. Give your own labels as LABEL=PATH:

dem_benchmark "MVS 3-scene=stereo_mvs3/run-DEM.tif" \
              "3 pairs + mosaic=pairwise_mosaic-DEM.tif" \
              "pair 13-16 (5.1°)=stereo_pair_13_16/run-DEM.tif" \
              --parquet atl06sr_all.parquet \
              --title "Atlanta WV2: same-pass scene combinations"

pc_align products go under <directory>/dem_benchmark/<label>/, never into the DEMs’ own folders, and are reused on a re-run. Skip alignment with --no-pc-align.

Comparing DEMs to one of them

Name one candidate as the reference to add its difference against every other DEM (vs_ref_median_m, vs_ref_nmad_m):

dem_benchmark "MVS 5-scene=stereo_mvs5/run-DEM.tif" "MVS 3-scene=stereo_mvs3/run-DEM.tif" \
              --parquet atl06sr_all.parquet --reference "MVS 5-scene"

Planetary DEMs

For Moon or Mars DEMs pass the LOLA/MOLA CSV from request_planetary_altimetry instead of a parquet; the body is detected from the DEMs:

dem_benchmark run_a/run-DEM.tif run_b/run-DEM.tif --altimetry-csv lola_pts_csv.csv

Reading the figure

• Rows are sorted best-first by post-alignment NMAD (pre-alignment with --no-pc-align).

• Coverage is percent valid inside the common footprint, with the valid km² printed beside it. --own-extent scores each DEM over its own footprint instead.

• IntersectionErr is the median triangulation error, NMAD in parentheses. A narrow-convergence pair has a small intersection error because its rays barely diverge, so read it next to the residual panels rather than as a ranking.

• dh median / dh NMAD are altimetry minus DEM, open marker before pc_align and filled after. Translation-only alignment leaves NMAD unchanged, so those markers coincide.

Full options

Usage: dem_benchmark [OPTIONS] DEMS...

  Score many DEMs against one altimetry sample.

  DEMS are paths, optionally labelled as LABEL=PATH (e.g.
  "MVS=stereo_mvs3/run-DEM.tif"); an unlabelled ASP run-DEM.tif is labelled by
  its folder. Every DEM gets: coverage inside the common footprint, the median
  triangulation error from its IntersectionErr raster when present, and the
  altimetry-minus-DEM median / NMAD / RMSE before and (unless --no-pc-align)
  after a pc_align translation. Writes a one-row-per-DEM summary figure, an
  overlaid residual histogram, and the stats table as CSV.

Options:
  --parquet TEXT           ICESat-2 ATL06-SR parquet cache to score Earth DEMs
                           against (the atl06sr_all.parquet a previous
                           asp_report run wrote next to its report, or from Al
                           timetry.request_atl06sr_multi_processing(save_to_pa
                           rquet=True)). The same points are replayed for
                           every DEM; no SlideRule request is made.
  --altimetry-csv TEXT     LOLA/MOLA CSV to score Moon/Mars DEMs against (see
                           request_planetary_altimetry). Use instead of
                           --parquet for planetary DEMs.
  --directory TEXT         Working directory. pc_align products and the
                           translated DEM copies go under
                           <directory>/dem_benchmark/<label>/, never into the
                           DEMs' own folders. Default: current directory.
  --reference TEXT         Label of one of the DEMs to difference the others
                           against (vs_ref columns of the stats table).
                           Default: none.
  --no-pc-align            Skip the per-DEM pc_align translation; report pre-
                           alignment residuals only.
  --own-extent             Compute coverage and triangulation-error statistics
                           over each DEM's own extent instead of the
                           intersection of all DEM footprints.
  --title TEXT             Figure title. Default: none.
  --output-directory TEXT  Directory for the figure and stats CSV. Default:
                           --directory.
  --output-filename TEXT   Figure filename; the stats CSV takes the same name
                           with a .csv extension, and the residual histogram
                           figure a _histogram suffix. Default:
                           dem_benchmark.png.
  --help                   Show this message and exit.

Python API

The same functionality is available via the DEMBenchmark class, which also keeps one Altimetry object per DEM (bench.altimetry[label]) so the usual per-DEM figures — mapview_plot_atl06sr_to_dem(), histogram_by_landcover() — can be drawn for any candidate afterwards:

from asp_plot.dem_benchmark import DEMBenchmark

bench = DEMBenchmark(
    directory="atlanta_mvs",
    dems={
        "MVS 3-scene": "atlanta_mvs/stereo_mvs3/run-DEM.tif",
        "MVS 5-scene": "atlanta_mvs/stereo_mvs5/run-DEM.tif",
        "3 pairs + mosaic": "atlanta_mvs/pairwise_mosaic-DEM.tif",
    },
    parquet="atlanta_mvs/atl06sr_all.parquet",
    reference="MVS 5-scene",
)
stats = bench.run()                    # one row per DEM
bench.summary_plot(save_dir="atlanta_mvs", fig_fn="dem_benchmark.png")
bench.histogram_plot()
bench.altimetry["MVS 5-scene"].histogram_by_landcover(key="all")