13 min read 2946 words

Containerizing a GeoPandas Pipeline with Docker and GDAL

The scenario: a pipeline that produced a permitting submission in March is rebuilt in September to answer a reviewer’s question, and the reprojected coordinates come out 8 centimetres from where they were. Nothing in the repository changed. The requirements.txt is identical. The difference is a PROJ version inside the base image, and it selected a different datum transformation pipeline for the same pair of EPSG codes. This page is the deployment detail behind spatial pipeline orchestration and deployment.

Root-cause analysis

Three layers decide what a geospatial container returns, and a typical Dockerfile pins one of them.

  1. The Python packages. geopandas, rasterio, pyproj, shapely — pinned by requirements.txt or a lockfile, and the only layer most teams think about.
  2. The native libraries. GDAL, PROJ and GEOS are C libraries that the Python packages bind to. Modern wheels vendor them, which means the wheel version pins the native version — but only if the wheel is actually used, and a build that falls back to a source install picks up whatever the base image provides.
  3. The PROJ datum grids. Transformation accuracy between datums depends on grid files that PROJ downloads on demand when PROJ_NETWORK=ON. A container that can reach the CDN gets high-accuracy transformations; the same container in a locked-down VPC silently falls back to a lower-accuracy path, and the difference is centimetres.
Four pinning mechanisms and what each one covers A coverage matrix with four pinning mechanisms as rows and four things they might pin as columns: Python package versions, the exact wheel artefact, the operating system libraries, and transformation accuracy. A plain requirements file covers only package versions. A hashed lockfile covers packages and the exact wheel. A base image pinned by digest covers the operating system libraries. Baked PROJ datum grids with network disabled cover transformation accuracy. A note marks the last row as the one most often omitted, and the one whose absence produces a silent centimetre-scale shift. Reproducibility is four separate pins, not one package versions exact wheel OS libraries transform accuracy requirements.txt lockfile with hashes base image by digest baked PROJ grids · network OFF The bottom row is the one usually missing, and the only one whose absence is invisible: PROJ falls back to a lower-accuracy transformation without raising, and the coordinates move by centimetres.

Pre-flight validation

The check that matters is not “does it import” but “does it transform to the same place”. Assert the versions and one known transformation at container start, and fail fast rather than producing subtly different coordinates for a week.

python
import pyproj
import rasterio
import shapely

# One control point with a known answer: NAD83(2011) geographic to UTM 14N, in metres.
CONTROL_LONLAT = (-101.8313, 35.2220)
CONTROL_UTM14N = (334_936.15, 3_899_889.52)   # metres, to 1 cm


def assert_geospatial_stack(*, expect_proj_major: int = 9, tol_m: float = 0.01) -> dict:
    """Refuse to run if the native stack is not the one this pipeline was validated against."""
    versions = {
        "pyproj": pyproj.__version__,
        "proj": pyproj.proj_version_str,
        "gdal": rasterio.__gdal_version__,
        "geos": shapely.geos_version_string,
        "network": pyproj.network.is_network_enabled(),
    }
    major = int(versions["proj"].split(".")[0])
    if major != expect_proj_major:
        raise RuntimeError(f"PROJ {versions['proj']} — pipeline validated against {expect_proj_major}.x")

    transformer = pyproj.Transformer.from_crs(4326, 32614, always_xy=True)
    x, y = transformer.transform(*CONTROL_LONLAT)
    dx, dy = abs(x - CONTROL_UTM14N[0]), abs(y - CONTROL_UTM14N[1])
    if max(dx, dy) > tol_m:
        raise RuntimeError(
            f"control point moved {max(dx, dy):.3f} m — datum grids or PROJ pipeline differ"
        )
    return versions

Fix implementation

The Dockerfile below pins all three layers. It builds on a slim Python base, installs from wheels that vendor their native libraries, copies the datum grids into the image rather than fetching them at runtime, and runs the assertion above as the last build step so a bad image fails at build time rather than in production.

dockerfile
FROM python:3.11.9-slim-bookworm AS base

# Native libraries arrive vendored inside the wheels; these are only what GDAL's
# runtime needs for HTTP and compression, pinned to the distribution release.
RUN apt-get update && apt-get install -y --no-install-recommends \
        ca-certificates=20230311 \
        libexpat1 \
    && rm -rf /var/lib/apt/lists/*

ENV PIP_NO_CACHE_DIR=1 \
    PROJ_NETWORK=OFF \
    PROJ_DATA=/opt/proj \
    GDAL_CACHEMAX=512 \
    GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR \
    VSI_CACHE=TRUE

# Wheels only: a source build would link against whatever GDAL the base image has.
COPY requirements.lock /tmp/requirements.lock
RUN pip install --only-binary=:all: --require-hashes -r /tmp/requirements.lock

# Bake the datum grids so the transformation is identical with or without network.
RUN python -c "import pyproj, pathlib; print(pyproj.datadir.get_data_dir())" \
    && mkdir -p /opt/proj \
    && cp -r "$(python -c 'import pyproj; print(pyproj.datadir.get_data_dir())')"/. /opt/proj/
COPY grids/ /opt/proj/

COPY src/ /app/src/
WORKDIR /app

# Fail the build, not the run, if the stack does not reproduce the control point.
RUN python -c "from src.stack import assert_geospatial_stack; print(assert_geospatial_stack())"

ENTRYPOINT ["python", "-m", "src.run"]

The four environment variables above are not decoration. GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR stops GDAL from listing an entire object-store prefix every time it opens one file, which on a bucket with tens of thousands of objects is the difference between a one-second and a one-minute open. VSI_CACHE=TRUE and GDAL_CACHEMAX bound the block cache so a windowed read does not quietly grow to fill the container’s memory limit — the same memory discipline described in geospatial data ingestion pipelines.

Image size and cold start across four container strategies A paired chart over four container strategies. The osgeo/gdal base image is 1.42 gigabytes with a 2.9 second cold start. A slim Python base with vendored wheels is 486 megabytes and 1.4 seconds. Adding baked PROJ datum grids brings it to 578 megabytes with no change to cold start. Stripping pip caches and apt lists returns it to 517 megabytes. The third configuration is marked as the working choice, with a note that the grids buy reproducibility rather than speed. Four strategies, by image size and cold start osgeo/gdal base 1420 MB · 2.9 s cold start slim + vendored wheels 486 MB · 1.4 s cold start + baked PROJ grids 578 MB · 1.4 s cold start + caches stripped 517 MB · 1.4 s cold start The grids add 92 MB and no start time — the only size increase worth taking Cold start is PROJ init and the GDAL driver registry, not the image size

Fallback routing and performance tuning

  • Multi-stage builds save less than expected. The wheels are the image, and they are needed at runtime; a builder stage helps only if something is compiled. Removing pip caches and apt lists saves more.
  • Layer order decides rebuild time. Copy the lockfile and install before copying source, so a code change rebuilds one small layer rather than reinstalling GDAL.
  • Cold start is dominated by PROJ and the driver registry. For short tasks, keep a warm worker rather than shrinking the image; for long tasks, the image size barely matters.
  • Pin the base image by digest, not by tag. python:3.11-slim moves; python@sha256:… does not, and the whole point of this exercise is that it does not move.
  • Keep the grids in the image for reproducibility, not for speed. A cached CDN fetch is fast; what it is not is identical across environments and over time.

Downstream validation

The container should emit its versions with every run, so an artefact can always be traced back to the stack that produced it. Combined with the run record described in the parent page, that gives a complete answer to “what produced this file”.

python
import json
import logging

log = logging.getLogger("siting.stack")


def log_stack_provenance(run_id: str) -> None:
    """One structured line per run — the cheapest reproducibility insurance available."""
    versions = assert_geospatial_stack()
    log.info(json.dumps({"event": "stack", "run_id": run_id, **versions}))
The same control point under four stack configurations A table of one control point reprojected under four configurations. PROJ 9.2 with grids present: the reference position. PROJ 9.3 with grids present: 0.004 metres away, within tolerance. PROJ 9.3 with grids absent and network enabled: 0.004 metres, because the grid was fetched. PROJ 9.3 with grids absent and network disabled: 0.084 metres away, using a ballpark transformation, with no warning raised. The last row is marked as the failure the control-point assertion exists to catch. One control point, four stack configurations PROJ 9.2 · grids present reference — PROJ 9.3 · grids present 0.004 m within tolerance PROJ 9.3 · no grids · network ON 0.004 m grid fetched at runtime PROJ 9.3 · no grids · network OFF 0.084 m ballpark fallback — silent 84 millimetres is invisible on a map, inside survey-staking tolerance, and enough to move a boundary vertex across a setback line. The assertion at container start is what turns it into a failed build.

Frequently asked questions

Should I use the official GDAL image instead of a Python base?

Only if the pipeline needs GDAL command-line tools. The osgeo/gdal images are large and pin GDAL tightly, which is helpful, but they bring a full toolchain most Python pipelines never call. Starting from a slim Python base with vendored wheels gives the same pinning at a fraction of the size, and the wheel is what rasterio actually binds to either way.

How do I produce a lockfile with hashes?

pip-compile --generate-hashes from pip-tools, or uv pip compile --generate-hashes. The hashes matter more here than in most projects: they are what stops a wheel from being silently replaced by a rebuild for a different platform, which is one of the few remaining ways the native stack can change without the version changing.

What happens if the container cannot reach the PROJ CDN?

With PROJ_NETWORK=OFF and baked grids, nothing — which is the point. With network on and no reachable CDN, PROJ falls back to a lower-accuracy transformation and does not raise, so the run succeeds and the coordinates move. That silent fallback is the single strongest argument for baking the grids.

Does this apply to serverless deployments?

The pinning does; the shape changes. A Lambda-style function still needs the same wheels and grids, usually shipped as a layer or a container image, and still benefits from the control-point assertion — run it at cold start rather than at build. What does not carry over is the assumption of a warm process: PROJ initialisation and the GDAL driver registry are paid per cold start, which is why raster work fits serverless poorly.

Does a bigger image cost anything besides pull time?

Mostly it costs cache churn and attack surface rather than run time. A 1.4 gigabyte image pulls slowly on a cold node and evicts other images from the node’s cache, which shows up as unpredictable start latency for everything else on the host. It also ships a compiler toolchain and a set of command-line utilities that a Python pipeline never invokes, each of which is something to patch. The runtime difference between a 500 megabyte and a 1.4 gigabyte image on a warm node is close to zero.

How should the image be tagged?

By content, not by intent. A tag like latest or prod tells a reader nothing about what is inside, and both move. Tagging by the commit SHA and recording the resulting digest in the run record means an artefact from six months ago names the exact image that produced it, and pulling that digest reproduces the stack byte for byte — which is the whole point of the pinning above.

Can the same image serve both the pipeline and interactive analysis?

Yes, and it is worth arranging. An analyst working in a notebook against a different GDAL than the pipeline uses will eventually produce a number that the pipeline cannot reproduce, and the investigation is expensive. Publishing the same image with a Jupyter entry point costs one extra build stage and removes an entire category of “it works in my notebook” discrepancy.

Related articles

Back to Spatial Pipeline Orchestration & Deployment