20 min read 4395 words

Spatial Pipeline Orchestration & Deployment

Orchestration is the stage where a working notebook becomes a service that runs at 03:00 without anyone watching, and it is the deployment half of the core energy-GIS data and spatial fundamentals pipeline. The failure modes here are different in kind from the ones upstream. A geometry bug produces a wrong shape; an orchestration bug produces a shape that is correct, a schedule that ran, and a result that silently describes last month. The three that recur are version drift in the native geospatial stack, tasks that are not safe to re-run, and logging that records that a job finished without recording what it did.

Version drift is the one that surprises teams. geopandas and rasterio are thin Python bindings over GDAL, PROJ and GEOS, and the numbers they return depend on the C libraries underneath. A container rebuilt six months later with an unpinned base image can select a different datum transformation pipeline for the same pair of EPSG codes, shifting coordinates by centimetres — invisible on a map, material at survey-staking tolerance, and completely undetectable from the Python dependency list. Pinning the Python packages is not enough; the native stack has to be pinned too, and a fixture test has to assert that a known reprojection still lands where it did.

What a requirements file pins, and what actually decides the answer A four-layer stack. The top layer is the analysis code. Below it, the Python bindings — geopandas, rasterio, pyproj — marked as pinned by a requirements file. Below those, the native libraries GDAL, PROJ and GEOS, marked as unpinned unless the container pins them. At the base, the PROJ datum grids, marked as downloaded at runtime unless baked into the image. Callouts note that the bottom two layers decide the coordinates, and that a fixture test asserting a known transformation is the only thing that catches a change in them. Four layers; a requirements file pins one of them analysis code your repository pinned by git geopandas · rasterio · pyproj Python bindings pinned by requirements.txt GDAL · PROJ · GEOS native C libraries pinned only by the container PROJ datum grids transformation accuracy downloaded at runtime unless baked in These two decide what a reprojection actually returns A fixture test on a known transformation is the only check that sees a change here A centimetre-scale datum shift is invisible on a map, material at staking tolerance, and absent from every Python dependency list.

The task graph is the contract

A spatial pipeline is a directed graph of tasks whose edges are datasets, and the useful discipline is to make each edge an explicit, addressable artefact rather than an in-memory handoff. A task that reads s3://.../interconnection-queue/2026-07/ and writes s3://.../screened/2026-07/ can be re-run in isolation, backfilled for an old month, and reasoned about without running anything. A task that reads a DataFrame from the previous step in the same process can do none of those things.

That framing decides the orchestration questions that follow. Retries become safe when every task is idempotent — running it twice with the same inputs produces the same artefact and no duplicates — which in practice means writing to a staging key and renaming, as described in geospatial data ingestion pipelines. Backfills become ordinary runs with a different partition parameter. And a partial failure leaves the graph in a state a scheduler can resume rather than a state a human has to reconstruct.

The unit of partitioning should follow the natural grain of the data, which in this domain is almost always geography crossed with time: a state per month, a balancing area per day, a project per run. Partitioning by an artificial batch identifier makes backfills unnatural and makes it impossible to answer “is Texas up to date” without reading everything.

Prerequisites and data requirements

The workflow assumes a container runtime, an object store, and a scheduler — Prefect, Airflow, Dagster or a cron-driven container are all workable, and the choice matters far less than the idempotency discipline. Inputs are the upstream artefacts described elsewhere in this section; outputs are versioned artefacts plus a run record.

The container is the part worth specifying precisely. Build on a slim Python base, install the geospatial stack from wheels that bundle their native libraries, pin every version including the wheel build, and set PROJ_NETWORK=OFF with the datum grids baked into the image unless the runtime genuinely has network access to the CDN. A pipeline that silently falls back to a lower-accuracy transformation because a grid download failed is a defect that only manifests in production.

Core implementation: an idempotent, partition-addressed task

The function below is deliberately boring: it derives its own output path from its parameters, short-circuits when the artefact already exists unless forced, writes to a staging key, renames, and returns a record rather than a DataFrame. Almost every orchestration property worth having follows from those five decisions.

python
from __future__ import annotations

import hashlib
import json
import logging
import time
import uuid
from dataclasses import asdict, dataclass

import geopandas as gpd

log = logging.getLogger("siting.pipeline")


@dataclass(frozen=True)
class TaskRecord:
    task: str
    partition: str
    output_uri: str
    rows_in: int
    rows_out: int
    rows_quarantined: int
    input_fingerprint: str
    proj_version: str
    duration_s: float


def screen_partition(
    state: str,
    month: str,
    *,
    root: str,
    target_epsg: int,
    force: bool = False,
    storage_options: dict | None = None,
) -> TaskRecord:
    """Screen one state-month partition. Safe to re-run; safe to backfill."""
    started = time.perf_counter()
    src = f"{root}/queue/state={state}/month={month}/part.parquet"
    dst = f"{root}/screened/state={state}/month={month}/part.parquet"

    fingerprint = _fingerprint(src, storage_options)
    if not force and _artefact_matches(dst, fingerprint, storage_options):
        log.info("skip: %s %s already built from %s", state, month, fingerprint[:12])
        return _load_record(dst, storage_options)

    queue = gpd.read_parquet(src, storage_options=storage_options)
    rows_in = len(queue)

    projected = queue.to_crs(target_epsg)
    keep = projected[projected.geometry.notna() & projected.is_valid]
    quarantined = rows_in - len(keep)

    staging = f"{dst}.staging-{uuid.uuid4().hex}"
    keep.to_parquet(staging, storage_options=storage_options)
    _atomic_rename(staging, dst, storage_options)

    import pyproj

    record = TaskRecord(
        task="screen_partition",
        partition=f"{state}/{month}",
        output_uri=dst,
        rows_in=rows_in,
        rows_out=len(keep),
        rows_quarantined=quarantined,
        input_fingerprint=fingerprint,
        proj_version=pyproj.proj_version_str,
        duration_s=round(time.perf_counter() - started, 3),
    )
    log.info("built %s", json.dumps(asdict(record)))
    return record


def _fingerprint(uri: str, storage_options: dict | None) -> str:
    """Content address for the input, so a re-run is a no-op when nothing changed."""
    import fsspec

    fs, path = fsspec.core.url_to_fs(uri, **(storage_options or {}))
    info = fs.info(path)
    key = f"{info['size']}:{info.get('mtime', '')}:{info.get('etag', '')}"
    return hashlib.sha256(key.encode()).hexdigest()

The fingerprint deserves comment. Content-addressing the input rather than checking a timestamp is what makes a re-run cheap and a backfill correct: a scheduled job that fires hourly against an unchanged input does nothing and says so, while a job whose upstream was revised rebuilds without anyone remembering to clear a cache. Recording the fingerprint on the output is what lets the next run make that decision without reading the data.

A second run touches only what changed A task graph over four state-month partitions. Each partition shows its input fingerprint and its output artefact. On the second run, Texas, New Mexico and Oklahoma have unchanged fingerprints and are skipped in milliseconds; Kansas has a revised upstream fingerprint and is rebuilt; a previously failed Colorado partition retries from a clean state because its earlier attempt wrote to a staging key that was never renamed. A legend maps the three outcomes: skipped, rebuilt and retried. Second run: same code, same schedule, different work TX / 2026-07 fingerprint unchanged skipped · 40 ms NM / 2026-07 fingerprint unchanged skipped · 38 ms OK / 2026-07 fingerprint unchanged skipped · 41 ms KS / 2026-07 upstream revised rebuilt · 92 s CO / 2026-07 previous attempt failed retried · 88 s Why the skip is safe the output records the input fingerprint it was built from Why the retry is safe the failed attempt wrote to a staging key, never to the target Why the rebuild happened the upstream artefact changed — no cache to clear by hand

Scheduling, backfills and the late-arriving partition

Energy data arrives late and gets revised. EIA back-revises monthly series, interconnection queues are republished with corrections, and a resource product can be reprocessed for a whole year at once. A scheduler that assumes a partition is final once built will serve stale results indefinitely; one that rebuilds everything nightly will spend most of its time recomputing unchanged history.

The workable middle is a freshness window plus fingerprint checking. Re-examine the last N periods on every run — three months is a common choice for monthly energy data — and rebuild only those whose input fingerprint changed. Everything older is rebuilt only when explicitly backfilled. This keeps the nightly cost proportional to the freshness window rather than to the history, and it makes revision handling automatic rather than a manual chore.

Backfills should use the same task the scheduler uses, with the partition as a parameter. A separate backfill script is a second implementation that drifts from the first, and the drift is discovered when the two produce different numbers for the same month.

Error handling and edge cases

A task that fails halfway through a write. The staging-then-rename pattern makes this a non-event: the target still holds the previous version, and the orphaned staging object is garbage collected by a lifecycle rule. Without it, a killed job leaves a truncated artefact that every downstream reader treats as complete.

A partition with no rows. Distinguish “no data yet” from “no rows qualify”. The first should leave the partition unbuilt so the next run retries; the second should write an empty artefact with a record, so downstream tasks can tell that the screen ran and found nothing. Conflating them produces a pipeline that either retries forever or silently reports zero.

A dependency upgrade that changes a result. This is what the fixture test exists for: a handful of geometries with known reprojected coordinates, areas and distances, asserted to a fixed tolerance in CI. It turns a silent centimetre-scale drift into a failing build, and it is the only mechanism that catches a PROJ pipeline change.

Clock and timezone assumptions in the schedule. A daily job partitioned by local date runs twice on one autumn day and zero times on one spring day in a DST-observing zone. Partition on UTC dates and let presentation handle local time, exactly as with the timeseries handling described under temporal data aggregation.

Observability for spatial pipelines

Generic pipeline metrics — did it run, how long did it take — say nothing about whether the spatial work was correct. Four spatial signals belong in every run record, and all four are cheap.

The quarantine rate is the earliest warning that an upstream provider changed something; alert on its delta rather than its level. The CRS transformation path actually used, as a string, catches the PROJ drift case in production rather than only in CI. The geometry repair count distinguishes a source that is degrading from one that is merely messy. And the bounding box of the output, logged per partition, catches the whole class of failures where a coordinate error moves a state’s worth of assets into the ocean — a check that costs one line and catches the most embarrassing possible bug.

Four spatial log fields and the bug each one catches A table of four structured log fields. Quarantine rate, an example value of 2.4 percent, catches upstream schema drift and should be alerted on as a delta rather than a level. Transformation pipeline, an example PROJ pipeline string, catches a native-library change in production. Geometries repaired, an example of 118, distinguishes a degrading source from a messy one. Output bounding box, an example west Texas extent, catches a coordinate error that relocates the whole partition. A note advises emitting all four as one JSON line, since a log aggregator can alert on a field but not on a sentence. Generic metrics say it ran; these say what it did quarantine_rate 0.024 upstream schema drift — alert on the delta proj_pipeline "+proj=pipeline +step …" a native-library change in production geometries_repaired 118 a degrading source versus a messy one output_bbox (-106.4, 31.2, -101.1, 36.0) a coordinate error that moves the partition Emit them as one JSON line per run: a log aggregator can alert on a field, never on a sentence.

Emit these as structured JSON on a single line per run, not as formatted prose. A log aggregator can alert on a field; it cannot alert on a sentence.

python
def assert_pipeline_invariants(record: TaskRecord, bbox: tuple[float, float, float, float],
                               study_bbox: tuple[float, float, float, float]) -> None:
    """CI and runtime gate — the four assertions worth failing a run over."""
    assert record.rows_out > 0, f"{record.partition}: screen produced no rows"
    rate = record.rows_quarantined / max(record.rows_in, 1)
    assert rate < 0.15, f"{record.partition}: quarantine rate {rate:.1%} — upstream schema drift?"
    assert record.proj_version.startswith("9."), (
        f"unexpected PROJ {record.proj_version} — pin the native stack, not just the wheels"
    )
    minx, miny, maxx, maxy = bbox
    sminx, sminy, smaxx, smaxy = study_bbox
    assert sminx <= minx and maxx <= smaxx and sminy <= miny and maxy <= smaxy, (
        f"{record.partition}: output bbox {bbox} escapes the study area {study_bbox}"
    )

Performance and cost

The dominant cost in a scheduled spatial pipeline is usually neither CPU nor storage but redundant reads. A task that re-reads a national constraint layer for every one of 3,000 parcels pays the transfer 3,000 times; the same task with the layer read once per worker pays it once per worker. The second largest is over-partitioning: tens of thousands of tiny artefacts cost more in object listing and metadata than they save in parallelism, which is the same effect described for streaming GeoParquet from cloud object storage.

Container cold start matters more than it does for non-spatial work, because the PROJ datum grids and the GDAL driver registry are not free to load. For short tasks that dominates the run; the fix is either a warm worker pool or fewer, larger tasks. Serverless functions are a poor fit for raster stages for exactly this reason and a reasonable fit for per-partition vector work.

Frequently asked questions

Airflow, Prefect or Dagster for a spatial pipeline?

Any of them, and the decision should turn on what the team already runs. None of the three has meaningful geospatial support, because the properties that matter — idempotent tasks, addressable artefacts, parameterised backfills, structured run records — are properties of how tasks are written rather than of the scheduler. A pipeline built on those properties can be moved between schedulers in an afternoon; one built without them cannot be made reliable on any of them.

How do I pin GDAL and PROJ properly?

Install from wheels that vendor their native libraries — rasterio and pyproj both publish these — pin the exact wheel versions, and record pyproj.proj_version_str and rasterio.__gdal_version__ in the run record. Then add a fixture test asserting a known transformation to a fixed tolerance. The pin prevents the drift; the record proves which version produced a given output; the fixture catches the case where the pin was changed deliberately and the consequences were not noticed.

Should the pipeline write to a database or to object storage?

Object storage for artefacts, a database for the index of them. Spatial artefacts are large, immutable and read whole; object storage is built for that and a database is not. What a database does well is answer “which partitions exist, when were they built, from what inputs” — which is exactly the run-record table, and it is small.

What belongs in a CI run versus a scheduled run?

CI runs the fixture tests, the schema assertions and a screening pass over a small committed sample; it must never touch production data or the real object store. The scheduled run does the real work with the same code paths. The most common mistake is letting CI depend on a live external portal, which makes the build fail for reasons that have nothing to do with the change under test.

How should secrets and API keys be provided?

Through the runtime’s secret mechanism, injected as environment variables, with a distinct key per job rather than a shared one. Portal quotas are per key, so a shared key means an interactive session can exhaust the budget a nightly job depends on — and a per-job key makes the log record identify which job hit the limit.

Is it worth containerising a pipeline that only ever runs on one machine?

Yes, for the pinning alone. The container is where the GDAL, PROJ and GEOS versions are fixed, and those are the dependencies most likely to change under a pipeline without anyone editing a requirements file. Running it on one machine is a scheduling decision; pinning the native stack is a correctness one.

Should the pipeline own its own copy of upstream data?

Yes, and the copy is the boundary. Reading a public portal directly from an analysis task couples the analysis to the portal’s uptime, its rate limits and its revision schedule all at once. An ingestion task that fetches once, validates, and writes a versioned artefact converts three sources of non-determinism into one dated file, and every downstream task then reads something that cannot change under it mid-run.

How much history should the run records keep?

Enough to answer the questions that get asked, which in practice means years rather than weeks. The records are small — one JSON line per partition per run — and the questions they answer are exactly the ones with commercial weight: which version of which input produced the figure in a submission, when a quarantine rate started drifting, and whether a result changed because the data changed or because the code did. Rotate the logs, keep the records.

What is the smallest useful orchestration setup?

A container with a pinned native stack, a cron trigger, tasks that derive their own output paths, and a JSON run record appended to object storage. That is enough to get idempotency, backfills and an audit trail without any scheduler at all. Schedulers earn their keep when the graph has branches, retries need policies, and more than one person needs to see why a run failed — but they add nothing that the task discipline does not already provide.

How do I test a pipeline whose inputs are hundreds of gigabytes?

By committing a fixture that is three orders of magnitude smaller and exercises the same code paths. A dozen parcels spanning a zone boundary, one invalid ring, one null geometry, one record with a mis-keyed voltage and one partition with no qualifying rows will find more defects than a full-scale run, and it runs in seconds on every commit. Scale tests belong in a nightly job against a real partition, where slowness is acceptable and a failure is informative rather than blocking.

Should tasks be retried automatically?

Yes, with a bounded count and only where the task is genuinely idempotent — which is the whole reason the write pattern matters. Transient object-store errors and portal rate limits are the normal case and resolve on the second attempt; a schema violation or a topology failure will not, so a retry policy that treats all failures alike simply delays the alert by the retry budget. Separate the two by exception type and let the deterministic failures fail fast.

What does a good run record look like six months later?

Readable without the code. It should name the task, the partition, the input fingerprint, the output URI, the row counts in and out, the quarantine count, the native library versions, and the wall-clock duration — and nothing that requires the reader to know how the pipeline is implemented. The test is whether an engineer who has never seen the repository can answer “what produced this file, from what, and when” from the record alone.

Explore this section

Back to Core Energy-GIS Data & Spatial Fundamentals