Validating NREL Solar Datasets with Python #

The failure usually surfaces as one of two exceptions at the ingestion boundary: RuntimeError: Spatial intersection yielded no valid irradiance values when a project polygon silently drifts out of the raster extent, or a MemoryError (or container OOM kill) when a multi-year NSRDB array is materialized in full. A quieter third failure produces no exception at all — stripped quality flags let night-time GHI spikes and negative irradiance leak into the aggregate, corrupting P50/P90 yields by several percent. All three break the same pipeline stage: validation of NREL solar irradiance data (NSRDB, PVWatts, and TMY3) before it reaches a yield model or an interconnection filing. This page is part of the open energy data portals ingestion reference and isolates each fault, surfaces it pre-flight, and delivers a validated fix plus a compliance-safe fallback path.

Root-Cause Analysis: Why NSRDB Validation Pipelines Fail #

When sourcing multi-year irradiance grids from public repositories, three compounding causes dominate automated validation failures. They rarely raise an exception at the point of error — each produces a plausible-looking output that only diverges from truth downstream, during a permit reviewer’s recomputation.

1. Implicit CRS Assumptions: NREL datasets default to WGS84 (EPSG:4326). Downstream GIS layers often use projected systems (e.g. EPSG:32612 or a state-plane zone). Without explicit coordinate reference system alignment, silent coordinate drift causes bounding-box clipping to return empty geometries or misaligned raster windows, corrupting spatial joins before they execute.
2. Memory Overflow on Full-Extent Reads: Loading 4 km-resolution, 20+ year NSRDB arrays into pandas or rasterio without windowing triggers OOM kills during spatial joins or temporal aggregation. Unmanaged array expansion is the leading cause of pipeline crashes in containerized environments.
3. Temporal & Quality-Flag Misalignment: NSRDB quality flags (Clearness Index, Solar Zenith Angle, and ghi_flag/dni_flag) are frequently stripped during CSV/GeoTIFF conversion. Unfiltered negative irradiance values or night-time GHI spikes corrupt yield calculations and violate IEC 61724-1 monitoring standards. A physically valid clearness index must satisfy $0\le K_t=\frac{\text{GHI}}{{\text{GHI}}_{\text{toa}}}\le 1$; any pixel outside that band is a flag failure, not a data point.

Understanding the underlying core energy-GIS data and spatial fundamentals prevents these silent failures by enforcing explicit coordinate validation, lazy evaluation, and flag-aware filtering before any spatial operation runs.

Pre-Flight Validation: Surface the Fault Before You Read the Raster #

The cheapest fix is to fail at the boundary. The function below runs before any pixel is read: it asserts both CRS are explicit and equal, confirms the project polygon actually overlaps the raster extent, and estimates the read footprint so an over-large request is rejected instead of OOM-killing the worker.

import geopandas as gpd
import rasterio
import pyproj
from shapely.geometry import box

def preflight_nrel(tif_path: str, boundary_path: str,
                   target_epsg: int = 4326,
                   max_pixels: int = 50_000_000) -> None:
    """Surface CRS drift, extent miss, and memory blowup before the main read."""
    target_crs = pyproj.CRS.from_epsg(target_epsg)

    boundary_gdf = gpd.read_file(boundary_path)
    if boundary_gdf.crs is None:
        raise ValueError("Boundary CRS is undefined. Assign explicitly before ingestion.")

    with rasterio.open(tif_path) as src:
        if src.crs is None:
            raise ValueError(f"Raster {tif_path} has no CRS tag; cannot validate alignment.")

        # 1. Reproject the boundary into the raster frame for an honest extent test
        boundary_aligned = boundary_gdf.to_crs(src.crs)
        raster_extent = box(*src.bounds)
        if not boundary_aligned.union_all().intersects(raster_extent):
            raise RuntimeError(
                "Project polygon lies outside the NSRDB raster extent — "
                "the intersection would yield zero valid pixels."
            )

        # 2. Estimate the read footprint in the target frame
        minx, miny, maxx, maxy = boundary_gdf.to_crs(target_crs).total_bounds
        px_w = abs((maxx - minx) / src.res[0])
        px_h = abs((maxy - miny) / src.res[1])
        if px_w * px_h > max_pixels:
            raise MemoryError(
                f"Requested window ~{int(px_w * px_h):,} px exceeds {max_pixels:,}; "
                "switch to a windowed or dask-backed read."
            )

Calling preflight_nrel() converts the two runtime exceptions into deterministic, early failures with actionable messages — exactly the contract a CI/CD ingestion gate needs. The same overlap-then-read discipline underpins cleaning messy shapefiles in geopandas.

Fix Implementation: A Production-Grade Validation Pipeline #

The corrected pipeline replaces the common failure patterns with explicit spatial alignment, memory-safe windowing, and compliance-aware filtering. The parameter choices are deliberate: boundless=True with fill_value=np.nan keeps the window read inside raster bounds while marking out-of-coverage cells as nodata rather than zero (a zero would be read as a valid 0 W/m² reading and bias the mean downward), and float casts guard the JSON-serializable return contract used by downstream audit logs.

import geopandas as gpd
import rasterio
import numpy as np
import pyproj
from rasterio.windows import from_bounds
import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

def validate_nrel_ghi(tif_path: str, boundary_path: str,
                      target_epsg: int = 4326) -> dict:
    """
    Validate an NREL GHI raster against a project boundary with explicit CRS
    alignment, memory-safe windowing, and quality-aware filtering.
    """
    target_crs = pyproj.CRS.from_epsg(target_epsg)

    # 1. Load and explicitly align the boundary CRS
    boundary_gdf = gpd.read_file(boundary_path)
    if boundary_gdf.crs is None:
        raise ValueError("Boundary CRS is undefined. Assign explicitly before ingestion.")
    if boundary_gdf.crs != target_crs:
        boundary_gdf = boundary_gdf.to_crs(target_crs)
        logging.info("Reprojected boundary to EPSG:%d", target_epsg)

    # 2. Open raster & assert spatial alignment (never auto-project silently)
    with rasterio.open(tif_path) as src:
        if src.crs != target_crs:
            raise ValueError(f"Raster CRS {src.crs} does not match target EPSG:{target_epsg}.")

        # 3. Memory-safe windowed read, clamped to the raster's own window
        bounds = boundary_gdf.total_bounds
        window = from_bounds(*bounds, src.transform)
        window = window.intersection(src.window(*src.bounds))

        ghi_array = src.read(1, window=window, boundless=True, fill_value=np.nan)
        out_transform = src.window_transform(window)

        # 4. Flag-aware masking: drop night-time, negative, and nodata cells
        ghi_flat = ghi_array.flatten()
        valid_mask = (ghi_flat > 0) & np.isfinite(ghi_flat)
        ghi_clean = ghi_flat[valid_mask]

        if ghi_clean.size == 0:
            raise RuntimeError("Spatial intersection yielded no valid irradiance values.")

        return {
            "mean_ghi": float(np.mean(ghi_clean)),
            "valid_count": int(valid_mask.sum()),
            "coverage_ratio": float(valid_mask.mean()),
            "crs": str(src.crs),
            "window_shape": ghi_array.shape,
            "transform": out_transform.to_gdal(),
        }

Key Validation Checkpoints #

• CRS Enforcement: The pipeline raises a hard exception on undefined or mismatched coordinate systems. Never rely on implicit geopandas or rasterio auto-projection — see the EPSG:4326 / EPSG:3857 alignment walkthrough for the failure signature this guards against.
• Window Intersection: window.intersection() guarantees the read stays within raster bounds, preventing boundless=True from introducing artificial edge artifacts.
• Flag-Aware Masking: The valid_mask removes negative values and NaN placeholders. In production, extend this to parse embedded ghi_flag arrays or companion CSV metadata so a flagged pixel never enters the mean.

Compliance-Safe Fallback Routing #

Production pipelines must degrade gracefully when primary datasets are incomplete, spatially misaligned, or fail quality thresholds. Implement a tiered fallback strategy and record which tier produced each value:

• Primary: NSRDB GeoTIFF/Parquet, validated via the pipeline above.
• Secondary: PVWatts API point-query fallback. When raster coverage is sparse, query the nearest valid grid cell using scipy.spatial.KDTree over project centroids — the same nearest-feature pattern used in proximity buffer analysis around substations.
• Tertiary: TMY3 synthetic generation. For preliminary siting, interpolate from historical TMY3 stations within a 50 km radius, applying elevation and albedo corrections.

Document every fallback activation in the project metadata. Regulatory bodies and interconnection authorities require transparent provenance when primary portal data is substituted; never mask fallback usage — append a data_source_tier field to the output schema instead.

Spatial Debugging & Memory-Tuning Protocols #

When validation returns unexpected zeros, empty arrays, or memory spikes, work through these strategies before touching the data:

• Verify bounding-box overlap first: test shapely.geometry.box(*bounds).intersects(raster_extent) before reading. A False here means the project polygon lies outside the dataset extent — the cause of most no valid irradiance values runtimes.
• Pin pixel alignment: misaligned transforms usually stem from floating-point precision drift. Round coordinates to 6 decimal places before constructing from_bounds() windows.
• Chunk temporal aggregation: for multi-year NSRDB stacks, avoid full-array loads. Use dask.array with rasterio’s block_shapes to compute monthly or seasonal aggregates lazily — the same windowed discipline applied to stacking NASA POWER and PVGIS rasters and resampling hourly solar data to monthly averages.
• Cap the GDAL cache: export GDAL_CACHEMAX (e.g. 512) in CI workers so block reads do not balloon resident memory under parallel windows.
• Validate downstream units: ensure yield models (pvlib, SAM) receive data in W/m² with consistent time zones. Mismatched UTC offsets between raster timestamps and local solar time introduce systematic bias in capacity-factor calculations. Reference the Rasterio windowed-read documentation for block-aligned chunking.

Downstream Validation: A CI/CD Integrity Gate #

Before a validated result is allowed into a yield model, assert its integrity. This compact gate is cheap enough to run on every ingestion and specific enough to fail a bad raster loudly — checking dtype, CRS, nodata bleed, and minimum spatial coverage.

import numpy as np
import rasterio
import pyproj

def assert_ghi_integrity(result: dict, tif_path: str,
                         target_epsg: int = 4326,
                         min_coverage: float = 0.85) -> None:
    """Post-read assertions suitable for a CI/CD ingestion gate."""
    with rasterio.open(tif_path) as src:
        assert src.crs == pyproj.CRS.from_epsg(target_epsg), \
            f"CRS drift: {src.crs} != EPSG:{target_epsg}"
        assert np.issubdtype(src.dtypes[0], np.floating), \
            "NSRDB GHI must be float to preserve NaN nodata; integer dtype bleeds zeros."

    assert result["coverage_ratio"] >= min_coverage, (
        f"Spatial coverage {result['coverage_ratio']:.1%} below {min_coverage:.0%} "
        "threshold — route to manual review, do not auto-approve."
    )
    assert 0.0 < result["mean_ghi"] < 12.0, (
        f"mean_ghi {result['mean_ghi']:.2f} kWh/m^2/day outside physical bounds — "
        "check unit conversion and flag masking."
    )

Flag any dataset where coverage drops below 85 % of the project area or more than 5 % of pixels fail quality checks, and route it to a manual review queue rather than auto-approving the yield estimate.

Audit-Ready Documentation & Provenance #

Environmental compliance and grid-interconnection filings demand reproducible, version-controlled validation artifacts:

• Checksum verification: generate SHA-256 hashes for all ingested rasters and boundary files, stored in a validation_manifest.json alongside pipeline outputs.
• Dependency pinning: export exact environment states with pip freeze > requirements.lock, including pyproj, rasterio, and geopandas minor versions to prevent silent CRS-library regressions.
• Structured logging: replace print() with structured JSON logs capturing timestamp, crs, valid_pixel_count, mean_ghi, and fallback_triggered for downstream audit trails.
• Quality-threshold reporting: record the coverage_ratio and data_source_tier on every output row so a reviewer can reconstruct exactly which tier and which pixels produced each number.

For authoritative guidance on solar data-quality metrics, consult the NREL NSRDB Technical Reference. By enforcing explicit spatial alignment, memory-safe ingestion, and compliance-aware fallback routing, engineering teams eliminate silent validation failures and deliver audit-ready solar yield models at scale.

Related #

• Open Energy Data Portals — the parent ingestion pattern this NSRDB workflow plugs into.
• How to align EPSG:4326 and EPSG:3857 for solar site mapping — the CRS-drift failure signature behind empty intersections.
• Stacking NASA POWER and PVGIS rasters in Rasterio — multi-source irradiance harmonization once NSRDB validation passes.
• Resampling hourly solar data to monthly averages — the temporal aggregation stage downstream of validation.
• Solar & Wind Resource Modeling Workflows — the resource-modeling domain these validated rasters feed.