15 min read 3237 words

Adding Spatial Regression Tests to a CI Pipeline

The scenario: a dependency bump passes every unit test, deploys cleanly, and changes a reported setback area by 0.4 percent. Nobody notices for two months, and then a reviewer compares two versions of the same submission. Spatial regressions are hard to catch with ordinary tests because the outputs are geometries rather than values, and because “close enough” is a real requirement rather than a smell. This page builds the fixture suite that catches them, and it is the CI half of spatial pipeline orchestration and deployment.

Root-cause analysis

Spatial regressions escape ordinary testing for three structural reasons.

  1. Geometry equality is the wrong assertion. Two polygons that differ by a floating-point ulp in one vertex are not equal, and a test that demands equality fails on every platform change. Tests have to assert on measurable properties — area, length, centroid, validity, CRS — with explicit tolerances.
  2. The defect is in a dependency, not in the diff. A PROJ upgrade, a GEOS overlay change or a GDAL resampling fix produces different output from unchanged code. Tests that only exercise the repository’s own logic never see it.
  3. The magnitude is below the noise floor of a visual check. An 8-centimetre coordinate shift and a 0.4 percent area change are invisible on a map and material in a submission, so the review that would catch a big error catches nothing.
Three test tiers: fixture, contract and scale Three tiers described side by side. Property tests over a committed fixture: under one second, run on every commit, catching reprojection drift, geometry repair changes and overlay arithmetic regressions. Contract tests over a recorded portal response: a few seconds, run on every commit, catching schema drift and parsing regressions with no network access. Scale tests over one real partition: several minutes, run nightly, catching memory growth and performance regressions. Each tier lists what it cannot catch, so the three are read as complements rather than alternatives. Three tiers, three failure classes, three cadences property tests < 1 s · every commit reprojection drift geometry repair changes overlay arithmetic cannot catch memory growth contract tests seconds · every commit portal schema drift parsing regressions unit changes cannot catch a live outage scale tests minutes · nightly peak memory growth wall-clock regressions partition skew too slow to gate a commit The three are complements. A team with only property tests ships a memory regression; a team with only scale tests waits until midnight to learn that a reprojection moved. None of the three should call a live portal — a recorded response is the contract.

Pre-flight validation: what belongs in the fixture

The fixture should be small enough to commit and pathological enough to be interesting. Six items cover most of the surface: a control point with a known reprojection, a polygon with a known area in an equal-area frame, a self-intersecting bowtie, a polygon with a hole, a pair of layers that overlap along a shared edge, and a small raster with a known windowed-read result. Together they are a few kilobytes and exercise every library in the stack.

python
from pathlib import Path

import geopandas as gpd
import pytest

FIXTURES = Path(__file__).parent / "fixtures"


@pytest.fixture(scope="session")
def parcels() -> gpd.GeoDataFrame:
    """Twelve parcels: two invalid, one with a hole, one straddling a UTM zone edge."""
    return gpd.read_file(FIXTURES / "parcels.gpkg")


@pytest.fixture(scope="session")
def constraints() -> gpd.GeoDataFrame:
    """Three constraint layers that overlap along shared edges, as real ones do."""
    return gpd.read_file(FIXTURES / "constraints.gpkg")

Fix implementation: the four assertions

The tests below are deliberately property-based rather than snapshot-based. Each one asserts something that must remain true, with a tolerance chosen from what the domain actually needs — a centimetre for coordinates, a hundredth of a percent for areas.

python
import math

import geopandas as gpd
import pyproj
import pytest

CONTROL_LONLAT = (-101.8313, 35.2220)
CONTROL_UTM14N = (334_936.15, 3_899_889.52)
KNOWN_PARCEL_HA = 42.187          # measured once in EPSG:5070, checked forever after


def test_control_point_reprojects_to_the_same_metre():
    """Catches a PROJ pipeline or datum-grid change — the invisible regression."""
    t = pyproj.Transformer.from_crs(4326, 32614, always_xy=True)
    x, y = t.transform(*CONTROL_LONLAT)
    assert math.isclose(x, CONTROL_UTM14N[0], abs_tol=0.01)
    assert math.isclose(y, CONTROL_UTM14N[1], abs_tol=0.01)


def test_parcel_area_is_stable(parcels):
    """Catches an equal-area frame change or a geometry repair that moved a boundary."""
    p = parcels.loc[parcels["parcel_id"] == "P-0007"].to_crs(5070)
    ha = float(p.area.iloc[0]) / 10_000.0
    assert ha == pytest.approx(KNOWN_PARCEL_HA, rel=1e-4)


def test_repair_preserves_area(parcels):
    """Catches a make_valid change that turns a bowtie into a different shape."""
    bad = parcels.loc[~parcels.is_valid].to_crs(5070)
    assert len(bad) == 2, "fixture should carry exactly two invalid geometries"
    repaired = bad.geometry.make_valid()
    assert repaired.is_valid.all()
    # A bowtie repair legitimately changes area; a hole repair must not.
    hole = parcels.loc[parcels["parcel_id"] == "P-0011"].to_crs(5070)
    assert float(hole.geometry.make_valid().area.iloc[0]) == pytest.approx(
        float(hole.area.iloc[0]), rel=1e-9
    )


def test_overlay_area_reconciles(parcels, constraints):
    """Catches a GEOS overlay change and any union/difference arithmetic regression."""
    study = parcels.to_crs(5070).union_all()
    excl = constraints.to_crs(5070).clip(study).union_all()
    buildable = study.difference(excl)
    gross_ha = study.area / 10_000.0
    excl_ha = excl.area / 10_000.0
    build_ha = buildable.area / 10_000.0
    assert build_ha + excl_ha == pytest.approx(gross_ha, rel=1e-9)
    assert build_ha <= gross_ha
Where to put a tolerance on a logarithmic scale of things that move geometry A logarithmic scale of displacement magnitudes from a micrometre to a hundred metres, with five regions marked. Below a micrometre: floating-point noise, which a test must ignore. Around a centimetre: a PROJ datum realisation change, which a test must catch. A few centimetres: survey staking tolerance. Around a metre: a boundary vertex crossing a setback line. Above ten metres: a wrong CRS entirely. A band between one millimetre and one centimetre is marked as where a coordinate tolerance belongs, and a separate note gives one hundredth of a percent as the equivalent for areas. A tolerance has to sit between noise and consequence put the tolerance here 1 µm 1 mm 1 cm 10 cm 1 m 10 m 100 m below 1 µm floating-point noise a test must ignore it ≈ 1 cm PROJ datum realisation change a test must catch it 2–5 cm survey staking tolerance the domain limit ≈ 1 m a boundary vertex crosses a setback a test must catch it > 10 m the wrong CRS entirely any test catches it coordinates: abs_tol = 0.01 m tight enough to catch a datum change areas: rel = 1e-4 loose enough to survive a GEOS update

Fallback routing and performance tuning

  • Keep the fixture in the repository, not in object storage. A test that needs network access is a test that fails for reasons unrelated to the change under review.
  • Never let CI call a public portal. Portal outages and rate limits become build failures, and the failure mode teaches the team to ignore red builds. Record a response once and replay it.
  • Separate fast from slow. The property tests above run in under a second and belong on every commit; a scale test over a real partition belongs in a nightly job where a failure is informative rather than blocking.
  • Assert on properties, never on WKT strings. A snapshot of well-known text is a test that fails on every platform, coordinate-precision or library change, which trains everyone to regenerate it without reading the diff.
  • Pin the container in CI too. Running the tests in the same image the pipeline deploys is what makes them meaningful; running them against whatever the runner has installed tests the runner.

Downstream validation

When a spatial test fails, the useful output is the magnitude and the direction of the change, not merely that it changed. A failure that reports “area 42.187 → 42.203 ha (+0.038 percent, +0.016 ha)” is triageable in seconds; one that reports “assertion failed” starts an investigation.

python
def report_delta(name: str, expected: float, actual: float, unit: str) -> str:
    """Format a spatial regression so the reviewer can judge it without rerunning anything."""
    delta = actual - expected
    pct = (delta / expected * 100.0) if expected else float("inf")
    return f"{name}: {expected:.4f} → {actual:.4f} {unit} ({delta:+.4f}, {pct:+.3f}%)"
The delivery flow, and where each kind of spatial test runs A left-to-right flow. A commit enters a CI job that runs inside the pinned container image, where property tests and contract tests run in under ten seconds. A pass publishes the image to the registry and the pipeline deploys. A separate nightly job pulls the published image and runs a scale test over one real partition, emitting a delta report rather than a binary result. A dependency upgrade branch is shown taking the same path, with the expected fixture values updated in the same commit as the upgrade so the change is recorded rather than hidden. Same container in CI as in production — or the tests test the runner commit code or lockfile change CI in the pinned image property + contract · < 10 s publish image digest recorded nightly scale test one partition · delta a dependency upgrade takes the same path — with the expected fixture values updated in the same commit as the upgrade The nightly job reports a delta rather than a pass: a 3% wall-clock change is information, not a failure.

Frequently asked questions

What tolerance should a coordinate assertion use?

One centimetre for projected coordinates is a good default: it is far tighter than any datum realisation change that matters and far looser than floating-point noise. For geographic coordinates, express the tolerance in metres by converting rather than in degrees, because a degree of longitude is not a fixed distance and a degree-based tolerance is latitude-dependent.

Should the fixture include real project data?

No. Use synthetic geometries with the same pathologies — an invalid ring, a hole, a shared edge, a zone-straddling extent — because real parcels carry ownership information, cannot always be redistributed, and are far larger than a test needs. Synthetic fixtures also let you construct the awkward cases deliberately rather than hoping a real extract contains them.

How do I test a raster pipeline without committing a large raster?

Commit a small one: a 64 by 64 float32 GeoTIFF with a known mean, a known nodata pattern and a known windowed-read result is under 20 kilobytes and exercises the same code path as a national product. What it will not exercise is memory behaviour, which belongs in the nightly scale test.

What should happen when a dependency upgrade legitimately changes a result?

Update the expected value in the same commit as the upgrade, with the delta in the commit message. That is the whole workflow the fixture exists to support: the test does not prevent change, it forces the change to be seen and recorded. A silent 8-centimetre shift becomes a line in the history explaining why the number moved.

How many fixture geometries are enough?

A dozen, chosen for pathology rather than for coverage. Two invalid rings, one polygon with a hole, one pair sharing an edge, one geometry straddling a UTM zone boundary and a handful of ordinary parcels will exercise every branch a spatial pipeline has. Adding a hundred well-behaved parcels adds runtime and finds nothing, because well-behaved geometry is not where the defects live.

Should the tests assert on the number of features?

Yes, and it is one of the cheapest assertions available. A repair that silently splits a bowtie into two polygons, an overlay that explodes one parcel into fragments, or a filter that drops rows all show up as a changed feature count long before they show up as a changed area. Assert the count alongside the area and the two together pin the behaviour.

What about testing the CRS handling itself?

Test that the output CRS is the declared one, and test one reprojected coordinate — not the transformation machinery, which is PROJ’s job. The failure mode worth catching is a pipeline that loses or overwrites a CRS somewhere in the middle, which shows up as an output whose declared frame is right and whose coordinates are in a different one. The control point catches exactly that.

Should a spatial test run against the real object store?

No — mock the boundary and test the parsing. What the pipeline reads from object storage is bytes in a known format, and a fixture file exercises every code path that matters without a network call, a credential or a bucket that someone can empty. The one thing worth testing against a real store is the credential and permission wiring, and that belongs in a smoke test at deploy time rather than in the commit path.

How do these tests interact with the run-record assertions?

They are the same assertions at two moments. The CI tests run them against a fixture on every commit; the pipeline runs a subset of them against real partitions on every run. Writing them once as functions that take data and raise, rather than as test bodies, is what makes that reuse possible — and it means a production failure and a CI failure produce the same message.

Related articles

Back to Spatial Pipeline Orchestration & Deployment