Deduplicating Overlapping Transmission Segments from OpenStreetMap #

A total circuit length that comes back 15–40% too high, and a corridor count that overstates how many distinct rights-of-way cross a study area, is the failure signature this page exists to eliminate. It breaks the asset-inventory stage of transmission line and substation mapping: OpenStreetMap power=line data routinely carries two, three, or four LineString ways tracing the same physical corridor. Multiple mappers digitize the same towers years apart, a double-circuit line gets one way per circuit stacked on shared towers, and bulk imports overlay hand-traced geometry. None of this raises an error — every duplicate is a topologically valid line — so total_length_km = lines_gdf.geometry.length.sum() / 1000 silently double- or triple-counts, and any downstream corridor tally treats one right-of-way as several.

The naive fix, dropping rows where geometry is exactly equal, catches almost none of these. Two mappers never place vertices identically, so near-duplicates differ by a few metres at every node and survive an equality test untouched. Robust deduplication has to measure geometric near-equality rather than test for it, and it has to do so without collapsing a genuine parallel circuit into one line.

Root-cause analysis #

Three compounding causes account for the inflation, and each maps to a distinct detection or fix stage below.

1. Exact-equality matching misses near-duplicates. geometry.duplicated() and drop_duplicates(subset="geometry") compare vertex arrays byte-for-byte. Independently digitized copies of one corridor differ by sub-metre jitter at every node, so the equality filter passes all of them through and the length inflation is untouched. Near-equality must be measured with a distance metric and a tolerance, not asserted.
2. Over-aggressive merging drops a real parallel double-circuit. Two circuits on shared towers, or two nearby corridors in the same right-of-way, are supposed to remain two records — they are distinct assets carrying distinct load. A dedup rule tuned only on geometric proximity collapses them into one, deleting a live circuit from the inventory. The distinguishing signal is attribute-level: a differing circuit_id or voltage_kv marks a genuine parallel line that must be preserved.
3. Length double counting propagates downstream. Every retained duplicate adds its full length to the corridor total and its full geometry to any network attribute validation count, then to buffer areas and interconnection-screening tallies. The error is not cosmetic; it changes which corridors a siting model reports as available.

Detection needs two independent geometric tests run together. Two lines are near-duplicates only if their corridor buffers overlap heavily and their Hausdorff distance — the largest gap between the two point sets — is small. The Hausdorff distance between line sets $A$ and $B$ is

Buffer overlap alone flags two crossing lines that share a junction; Hausdorff alone is fooled by a short stub lying near a long line. Requiring both — a high buffer intersection-over-union together with a Hausdorff value under a tower-spacing tolerance — isolates true collinear copies.

Pre-flight validation #

Detection must run in a projected metric frame — buffers and Hausdorff distances are meaningless in degrees, so the same coordinate reference system alignment discipline that governs every distance operation applies here. The detector below reprojects to EPSG:32614 (UTM Zone 14N, a central-US grid footprint), builds flat-capped corridor buffers, prunes candidate pairs with the spatial index, and flags a pair only when the buffer overlap fraction and the Hausdorff distance both clear their thresholds. It returns a report rather than mutating anything, so a CI run can inspect exactly which segments would merge before the fix touches the data.

import geopandas as gpd
import pandas as pd
from shapely import hausdorff_distance

METRIC_EPSG = 32614  # UTM Zone 14N — central-US transmission footprint


def find_overlapping_segments(
    lines_gdf: gpd.GeoDataFrame,
    corridor_buffer_m: float = 10.0,
    hausdorff_eps_m: float = 25.0,
    min_overlap_frac: float = 0.30,
) -> pd.DataFrame:
    """Flag candidate duplicate / near-collinear power=line pairs.

    Returns one row per pair (i, j, hausdorff_m, overlap_frac) whose corridor
    buffers overlap by >= min_overlap_frac AND whose Hausdorff distance <= eps.
    Positional indices reference lines_gdf.reset_index(drop=True).
    """
    if lines_gdf.crs is None or not lines_gdf.crs.is_projected:
        raise ValueError(
            f"Duplicate detection needs a projected metric CRS; got {lines_gdf.crs}. "
            f"Reproject to EPSG:{METRIC_EPSG} first."
        )

    geom = lines_gdf.geometry.reset_index(drop=True)
    buffers = geom.buffer(corridor_buffer_m, cap_style="flat")
    sindex = buffers.sindex

    pairs = []
    for i in range(len(geom)):
        for j in sindex.query(buffers.iloc[i], predicate="intersects"):
            if j <= i:
                continue  # skip self-pairs and mirror duplicates
            inter = buffers.iloc[i].intersection(buffers.iloc[j]).area
            union = buffers.iloc[i].union(buffers.iloc[j]).area
            overlap_frac = inter / union if union else 0.0
            d_h = hausdorff_distance(geom.iloc[i], geom.iloc[j])
            if overlap_frac >= min_overlap_frac and d_h <= hausdorff_eps_m:
                pairs.append((int(i), int(j), round(d_h, 2), round(overlap_frac, 3)))

    return pd.DataFrame(pairs, columns=["i", "j", "hausdorff_m", "overlap_frac"])

The buffer overlap fraction is a directional intersection-over-union: $\mathrm{IoU}=|A\cap B|/|A\cup B|$ on the two corridor polygons. Two collinear copies score near 1.0; a line that merely clips another’s buffer near a junction scores low and is rejected before the more expensive Hausdorff call. cap_style="flat" keeps the buffer from ballooning past the endpoints, which otherwise inflates overlap for two lines that meet end-to-end.

Fix implementation #

Grouping is the safe operation, not deletion. Build connected components over the flagged pairs with a union-find, then keep one representative per component. Two parameter choices carry the correctness: pairs whose endpoints carry distinct non-null circuit_id values are refused from merging — that is the real parallel double-circuit case from cause 2 — and within each merged group the representative is chosen by highest voltage_kv, breaking ties on attribute completeness so the record a permitting reviewer can actually trace survives.

def deduplicate_segments(
    lines_gdf: gpd.GeoDataFrame,
    pairs: pd.DataFrame,
    id_col: str = "circuit_id",
    voltage_col: str = "voltage_kv",
) -> gpd.GeoDataFrame:
    """Collapse near-duplicate groups, keeping the best-attributed record.

    Genuine parallel circuits (distinct circuit_id on a near-duplicate pair) are
    never merged, so a real double-circuit is preserved rather than dropped.
    """
    lines_gdf = lines_gdf.reset_index(drop=True).copy()
    parent = list(range(len(lines_gdf)))

    def find(x: int) -> int:
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    for row in pairs.itertuples(index=False):
        id_a, id_b = lines_gdf.at[row.i, id_col], lines_gdf.at[row.j, id_col]
        if pd.notna(id_a) and pd.notna(id_b) and id_a != id_b:
            continue  # distinct real circuits — keep both, do not union
        parent[find(row.i)] = find(row.j)

    lines_gdf["_group"] = [find(i) for i in range(len(lines_gdf))]
    lines_gdf["_completeness"] = lines_gdf.notna().sum(axis=1)

    # Highest voltage wins; ties break toward the most complete record
    ranked = lines_gdf.sort_values(
        [voltage_col, "_completeness"], ascending=False, kind="stable"
    )
    keep_idx = ranked.groupby("_group", sort=False).head(1).index

    deduped = lines_gdf.loc[keep_idx].drop(columns=["_group", "_completeness"])
    return deduped.sort_index().reset_index(drop=True)

Because the merge decision is attribute-aware, this is the fix for both the length inflation and the double-circuit hazard at once: identical geometry with the same or missing circuit_id collapses to one length, while identical geometry carrying two real circuit identities is left intact. The retained representative keeps its geometry unchanged — no averaging of vertex positions, which would fabricate a centerline that matches neither survey — so the downstream snapping of transmission lines to substation nodes still lands on a real endpoint.

Fallback routing & performance tuning #

• Tune ε to tower spacing, not intuition. Set hausdorff_eps_m from the survey resolution of the source: 25 m absorbs typical OSM mapper jitter, but sub-metre LiDAR-derived corridors want 5–10 m so two genuinely adjacent circuits are not fused. Log the ε used into the provenance trail.
• Densify before Hausdorff on sparse geometry. OSM ways with widely spaced vertices can report a misleadingly small Hausdorff distance. Pass hausdorff_distance(a, b, densify=0.1) to interpolate points along each segment before measuring, at the cost of extra compute.
• Prune with the spatial index first. The sindex.query bounding-box filter keeps the pairwise Hausdorff work near $O(n\log n)$ instead of the $O(n^2)$ a brute-force cross-join implies; skip it and continental datasets stall.
• Partition by region for out-of-core runs. Duplicates only occur between geographically co-located ways, so tile the network by UTM zone or a coarse grid and run detection per tile in parallel — no cross-tile pair is ever a duplicate.
• Fall back to endpoint hashing when attributes are absent. If circuit_id and voltage_kv are both null across a group, rank the representative by geometry length (longest, most-complete trace) so the fix still returns a deterministic winner rather than an arbitrary row.

Downstream validation #

Gate the deduped layer before it reaches any corridor count or buffer analysis. The assertion below fails the build on two independent conditions: a total circuit_km outside the range an analyst expects for the study area — the direct symptom of over- or under-merging — and any residual near-duplicate pair that is not a legitimate distinct-circuit overlap. This is the same audit posture used when detecting and removing sliver polygons in GeoPandas: assert the artifact is gone, do not assume the cleaner removed it.

def assert_dedup_integrity(
    deduped_gdf: gpd.GeoDataFrame,
    expected_km_range: tuple[float, float],
    id_col: str = "circuit_id",
) -> None:
    """CI/CD gate: circuit_km within range and no duplicate pairs left over."""
    total_km = float(deduped_gdf.geometry.length.sum()) / 1000.0
    lo, hi = expected_km_range
    assert lo <= total_km <= hi, (
        f"circuit_km {total_km:.1f} outside expected [{lo}, {hi}] — "
        "over-merged (dropped real circuits) or under-merged (double-counted)."
    )

    residual = find_overlapping_segments(deduped_gdf)
    leaked = []
    for row in residual.itertuples(index=False):
        id_a = deduped_gdf.iloc[row.i][id_col]
        id_b = deduped_gdf.iloc[row.j][id_col]
        distinct_circuits = pd.notna(id_a) and pd.notna(id_b) and id_a != id_b
        if not distinct_circuits:
            leaked.append((row.i, row.j))
    assert not leaked, (
        f"{len(leaked)} duplicate pairs survived dedup (sample {leaked[:5]}); "
        "lower ε or widen the buffer and re-run."
    )

Logging total_km before and after, alongside the count of groups collapsed, gives an interconnection or environmental reviewer a one-line audit of how much of the raw OSM length was redundant. Pin shapely >= 2.0 and geopandas >= 0.14 in pyproject.toml so the vectorized hausdorff_distance signature and buffer semantics cannot shift the merge decision between runs.

Related #

• Transmission Line & Substation Mapping — the asset-inventory workflow this dedup step feeds a clean, non-inflated layer into.
• Snapping Transmission Lines to Substation Nodes with Shapely — the endpoint-alignment stage that consumes the deduplicated representatives.
• Network Attribute Validation — schema enforcement for the circuit_id and voltage_kv fields the merge decision keys off.
• Detecting and Removing Sliver Polygons in GeoPandas — the same assert-the-artifact-is-gone audit posture applied to geometry cleaning.