13 min read 2799 words

Best practices for cleaning messy shapefiles in GeoPandas

Scenario / symptom: a regulatory boundary or substation footprint shapefile loads fine, but the next overlay raises shapely.errors.TopologyException: Input geom 1 is invalid: Self-intersection, or gdf.to_crs(...) throws pyproj.exceptions.CRSError: Invalid projection, or capacity attributes silently vanish because the .dbf field name was truncated past 10 characters. This failure lands in the ingestion and validation stage of a renewable siting pipeline — the moment a 1990s-era shapefile feeds modern Python geometry operations without first being repaired. It is a recurring case of the broader data-integrity problem covered by the parent workflow on spatial data quality and validation: an unclean input passes the read, then surfaces as a confident-but-wrong buffer, a misaligned parcel, or a crashed overlay several stages downstream.

Shapefiles remain the default exchange format for regulatory agencies, utility operators, and legacy environmental databases, so an energy GIS team cannot simply refuse them. In transmission corridor routing, interconnection queue modeling, and constraint screening, unvalidated geometries cascade into erroneous setback zones, distorted area calculations, and flawed yield estimates. The fix is a deterministic, idempotent cleaning routine that repairs geometry before it normalises attributes, enforces an explicit coordinate frame, and quarantines anything it cannot safely repair instead of dropping it silently.

Root-cause analysis

Three structural deficiencies compound to produce the symptoms above, and each passes silently on its own:

  1. Invalid topologies. Self-intersections, bowtie polygons, and duplicate vertices originate from CAD exports, manual digitizing, or coordinate rounding. They survive read_file() untouched and only break when a spatial predicate (overlay, clip, intersects) evaluates them — exactly when the geometry feeds a grid capacity buffer analysis or a constraint overlay.
  2. CRS ambiguity. A missing, corrupted, or implicit .prj sidecar forces downstream operations into unprojected lat/lon space. Area and distance computed on degrees are geometrically meaningless, which is why every cleaning routine has to resolve coordinate reference system alignment before any metric calculation runs.
  3. Attribute corruption. Mixed character encodings (CP1252 vs UTF-8), null geometries, and string contamination in numeric MW-capacity or queue-position fields all flow through the legacy .dbf container, whose 10-character field-name limit truncates columns and collides keys.

Addressing these requires ordering the repair correctly — geometry first, then CRS, then attributes — with explicit fallback routing when automated validation fails.

Three messy-shapefile defects mapped to their repair stages Left column: three warning boxes naming a defect and the error it raises — invalid topology raising TopologyException, CRS ambiguity raising CRSError and degree-space area, and attribute corruption truncating and contaminating fields. Each arrows right into a matching repair stage — make_valid plus buffer(0), set_crs then to_crs EPSG:5070, and a 10-character truncate plus numeric coercion. The three repair stages converge with arrows into a single success node on the right: a clean, projected, audited GeoDataFrame. Defect → error Repair stage Output Invalid topology TopologyException CRS ambiguity CRSError · degree-space area Attribute corruption truncated · contaminated fields make_valid + buffer(0) fallback set_crs → to_crs EPSG:5070 (equal-area) truncate 10-char + numeric coerce Clean GeoDataFrame valid · projected · audited

Pre-flight validation

Surface the root cause before the cleaning routine runs, so a broken input is diagnosed rather than half-repaired. The check below inspects geometry validity, CRS presence, and .dbf field-name length, and returns a diagnostic dict without mutating the source.

python
import geopandas as gpd


def preflight_shapefile_check(input_path: str) -> dict:
    """Diagnose a shapefile's integrity before any cleaning runs.

    Returns a report dict; never mutates the source layer.
    """
    gdf = gpd.read_file(input_path, engine="pyogrio")

    null_geom = int(gdf.geometry.isna().sum() + gdf.geometry.is_empty.sum())
    invalid_geom = int((~gdf.geometry.is_valid).sum())
    long_fields = [c for c in gdf.columns if c != "geometry" and len(c) > 10]

    report = {
        "feature_count": len(gdf),
        "crs": str(gdf.crs) if gdf.crs is not None else None,
        "null_or_empty_geometries": null_geom,
        "invalid_geometries": invalid_geom,
        "fields_exceeding_dbf_10char": long_fields,
        "needs_cleaning": bool(null_geom or invalid_geom or long_fields or gdf.crs is None),
    }

    if gdf.crs is None:
        report["crs_warning"] = "No .prj/CRS metadata — distance & area will be wrong until set."
    return report

Running preflight_shapefile_check against a raw regulatory layer reports the exact invalid-geometry count and any over-length field names instead of letting the first overlay throw a TopologyException deep in the pipeline.

Fix implementation

The corrected routine isolates geometry repair, CRS enforcement, and attribute sanitization into discrete, auditable steps. It is engineered for batch processing of regulatory boundary layers, substation footprints, and land-use constraint datasets, with explicit memory controls and quarantine routing. Geometry is repaired before attributes are touched, EPSG:5070 (NAD83 / Conus Albers, an equal-area frame) is enforced so area-based capacity-density figures are trustworthy, and out-of-bounds or null records are written to dedicated quarantine layers rather than dropped.

Deterministic shapefile cleaning pipeline with quarantine routing A top-to-bottom flow down the centre — raw shapefile, load via pyogrio, a null-or-empty-geometry decision, make_valid plus buffer(0) fallback, CRS enforce to EPSG:5070, a within-expected-bounds decision, sanitize attributes (10-character truncate plus numeric coercion), and a clean GeoDataFrame. The null decision branches right on yes to a quarantine box, null_geometries.shp; the bounds decision branches right on no to a quarantine box, out_of_bounds.shp. Repair steps are blue, quarantine boxes are amber, and the final clean output is green. no yes yes no Raw shapefile 1 · Load via pyogrio 2 · Null / empty geometry? 3 · make_valid + buffer(0) fallback 4 · CRS enforce to EPSG:5070 5 · Within expected bounds? 6 · Sanitize attrs 10-char + numeric coerce Clean GeoDataFrame Quarantine null_geometries.shp Quarantine out_of_bounds.shp
python
import geopandas as gpd
import pandas as pd
from shapely.validation import make_valid
from shapely.geometry import box
import logging
from pathlib import Path

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)


def clean_shapefile_pipeline(
    input_path: str,
    target_crs: str = "EPSG:5070",
    encoding: str = "utf-8",
    expected_bounds: tuple = None,
    quarantine_dir: str = "quarantine",
) -> gpd.GeoDataFrame:
    """Deterministic cleaning routine for messy shapefiles in energy GIS pipelines.

    Enforces topology repair, CRS normalization, and attribute sanitization.
    Returns a validated GeoDataFrame and logs/quarantines failed records.
    """
    input_path = Path(input_path)
    quarantine_path = Path(quarantine_dir)
    quarantine_path.mkdir(parents=True, exist_ok=True)

    # 1. Load with explicit engine and encoding; fall back to fiona on failure
    try:
        gdf = gpd.read_file(input_path, engine="pyogrio", encoding=encoding)
    except Exception as e:
        logging.error(f"pyogrio read failed: {e}. Attempting fiona fallback...")
        gdf = gpd.read_file(input_path, encoding=encoding)

    if gdf.empty:
        raise ValueError("Empty dataset or failed attribute read. Verify shapefile integrity.")

    # 2. Null geometry handling & topology repair
    null_mask = gdf.geometry.isna() | gdf.geometry.is_empty
    if null_mask.any():
        logging.warning(f"Quarantining {null_mask.sum()} records with null/empty geometries.")
        gdf[null_mask].to_file(quarantine_path / "null_geometries.shp", driver="ESRI Shapefile")
        gdf = gdf.loc[~null_mask].copy()

    # Shapely 2.x make_valid as primary repair, buffer(0) as fallback
    valid_mask = gdf.geometry.is_valid
    if not valid_mask.all():
        invalid_count = int((~valid_mask).sum())
        logging.info(f"Repairing {invalid_count} invalid geometries via make_valid.")
        gdf.loc[~valid_mask, "geometry"] = gdf.loc[~valid_mask].geometry.apply(make_valid)

        still_invalid = ~gdf.geometry.is_valid
        if still_invalid.any():
            logging.warning(f"Applying zero-buffer fallback to {still_invalid.sum()} geometries.")
            gdf.loc[still_invalid, "geometry"] = gdf.loc[still_invalid].geometry.buffer(0)

    # 3. CRS enforcement & validation
    if gdf.crs is None:
        logging.warning("Missing CRS metadata. Assuming EPSG:4326 before reprojection.")
        gdf.set_crs("EPSG:4326", inplace=True)

    if str(gdf.crs) != target_crs:
        logging.info(f"Transforming from {gdf.crs} to {target_crs}.")
        gdf = gdf.to_crs(target_crs)

    # 4. Spatial bounds validation (upstream/downstream alignment)
    if expected_bounds:
        bounds_box = box(*expected_bounds)
        out_of_bounds = ~gdf.geometry.intersects(bounds_box)
        if out_of_bounds.any():
            logging.warning(f"Quarantining {out_of_bounds.sum()} records outside project bounds.")
            gdf[out_of_bounds].to_file(quarantine_path / "out_of_bounds.shp", driver="ESRI Shapefile")
            gdf = gdf[~out_of_bounds]

    # 5. Attribute sanitization (10-char limit, numeric coercion, encoding safety)
    gdf.columns = [col[:10] if (col != "geometry" and len(col) > 10) else col for col in gdf.columns]

    for col in gdf.select_dtypes(include=["object"]).columns:
        if col == "geometry":
            continue
        coerced = pd.to_numeric(gdf[col], errors="coerce")
        # Only adopt coercion when it does not destroy a genuinely textual column
        if coerced.notna().mean() >= 0.9:
            gdf[col] = coerced

    gdf = gdf.reset_index(drop=True)
    logging.info(f"Pipeline complete. {len(gdf)} valid records retained.")
    return gdf

Why these parameter choices

  • make_valid before buffer(0). shapely.validation.make_valid decomposes invalid rings into valid components while preserving topology; buffer(0) is a cruder ring-normaliser kept only as a fallback for geometries make_valid cannot resolve. After repair, assert gdf.geometry.area > 0 — zero- or negative-area geometries are collapsed rings that will skew capacity-density figures. For transmission corridor routing, explode multipart geometries with gdf.explode(index_parts=True) before a routing algorithm consumes them, consistent with the transmission line and substation mapping workflow.
  • EPSG:5070 as the target. An equal-area frame keeps continental-US area calculations honest. Default to EPSG:4326 only when metadata is absent, then reproject. After transformation, projected metres should fall roughly within [-2e6, 3e6] for CONUS; lat/lon-range values surviving in a projected CRS signal a failed transform.
  • Conditional numeric coercion. pd.to_numeric(..., errors="coerce") is only adopted when at least 90% of values parse, so a genuinely textual land-use or regulatory-ID column is never silently nulled. Maintain a companion metadata CSV mapping full column names to their truncated .dbf headers to preserve audit traceability without violating the shapefile spec.

Fallback routing & performance tuning

For national-scale layers, CI/CD runs, and memory-constrained cloud nodes, layer these strategies on top of the core routine:

  • Prune columns at read time. Pass columns=["geometry", "OBJECTID", "CAP_MW"] to gpd.read_file() so the heavy .dbf attributes never enter memory before repair — this cuts peak RAM sharply on wide regulatory tables.
  • Chunk or convert beyond ~500k features. Topology validation is the memory hot spot; for very large environmental layers, process in batches or stage intermediates as GeoParquet rather than re-reading the shapefile, which also sidesteps the .dbf encoding round-trip.
  • Quarantine, never drop. null_geometries.shp and out_of_bounds.shp carry the same attribute schema as the source, so an analyst can correct source digitizing errors and re-ingest without halting the automated run.
  • Pin the GDAL/PROJ stack. Pin pyogrio/pyproj to exact versions in requirements.txt so the bundled PROJ datum database is identical across CI/CD and production, keeping reprojection deterministic.
  • Isolate repair failures. Wrap the per-feature repair in a try/except for shapely.errors.GEOSException; on failure, route the feature to quarantine and continue rather than crashing the whole batch.

Downstream validation

Gate the cleaned output in CI/CD with an assertion that fails the build on residual invalidity, CRS drift, or attribute regressions — the same audit posture used across the grid capacity buffer analysis and proximity workflows.

python
from datetime import datetime, timezone


def audit_clean_shapefile(gdf, expected_crs: str = "EPSG:5070") -> dict:
    """Assert cleaning integrity. Raises AssertionError on any CI/CD-blocking issue."""
    assert gdf.crs is not None, "Output CRS is undefined."
    assert str(gdf.crs) == expected_crs, f"CRS drift: expected {expected_crs}, got {gdf.crs}"

    invalid = int((~gdf.geometry.is_valid).sum())
    assert invalid == 0, f"{invalid} invalid geometries remain after repair."

    null_geom = int(gdf.geometry.isna().sum() + gdf.geometry.is_empty.sum())
    assert null_geom == 0, f"{null_geom} null/empty geometries leaked past quarantine."

    long_fields = [c for c in gdf.columns if c != "geometry" and len(c) > 10]
    assert not long_fields, f"Fields exceed .dbf 10-char limit: {long_fields}"

    return {
        "feature_count": len(gdf),
        "target_crs": str(gdf.crs),
        "all_geometries_valid": invalid == 0,
        "min_area_m2": round(float(gdf.geometry.area.min()), 2),
        "timestamp": datetime.now(timezone.utc).isoformat(),
    }

Attaching the returned audit dictionary to each deliverable preserves data lineage, satisfies ISO 19115 metadata expectations, and lets a permitting authority or independent engineer reproduce exactly how a cleaned layer was derived before it feeds an interconnection study or environmental screening.

Related articles

Back to Spatial Data Quality & Validation