16 min read 3412 words

Downloading EIA and OpenEI Datasets with Python Requests

A requests.get(url).json() one-liner against the EIA API v2 or an OpenEI/NREL endpoint works exactly once — in a notebook, on a small query, on a good day. In an unattended batch it fails in ways that never raise an exception: a truncated result set because you never paginated past the first 5,000 rows, an HTTP 200 whose body is an HTML rate-limit page rather than JSON, or a silently gzipped payload that json.loads chokes on. This is the ingestion boundary that feeds every downstream workflow described in the open energy data portals pattern, so a defect here propagates straight into capacity aggregates, resource rasters, and permitting deliverables. This page hardens the download step itself: a resilient requests.Session helper that survives real portal behavior and hands off a schema-checked, cached, coordinate-tagged table.

The failure scenario

The concrete signatures you will hit against api.eia.gov/v2/ and developer.nrel.gov / OpenEI:

  • json.decoder.JSONDecodeError: Expecting value: line 1 column 1 — the response is HTML (a rate-limit or maintenance page) or gzip bytes, not JSON, yet response.status_code == 200.
  • A DataFrame that is exactly 5,000 rows when the series has 40,000 — EIA v2 caps length per request and you took the first page as the whole.
  • HTTP 429 Too Many Requests mid-batch, or an NREL 403 once the daily key quota is spent, killing an overnight run with no retry.
  • A KeyError: 'response' because the JSON came back shaped {"error": "invalid api_key"} and you indexed straight into the happy-path structure.

Root-cause analysis

Four compounding causes account for nearly every broken portal download, and each maps to a fix stage below:

  1. API-key and envelope handling. EIA v2 takes the key as an api_key query parameter and wraps data in a {"response": {"data": [...], "total": N}} envelope; NREL/OpenEI also key on api_key but return errors as 200 OK with an errors array. A script that reads payload["response"]["data"] without first checking for an error envelope raises an opaque KeyError instead of the real message.
  2. Rate limiting and quotas. Both portals throttle. EIA returns 429; NREL enforces a rolling hourly and daily cap and returns 403/429 with X-RateLimit-Remaining headers. Without honoring Retry-After and backing off, a burst of parallel requests gets the whole IP throttled.
  3. Pagination. EIA v2 paginates with offset/length and reports response.total; taking only the first page silently truncates every series longer than one page.
  4. Content-type and encoding traps. A throttle or WAF response is text/html with a 200, and gzip-encoded bodies decode to bytes. Trusting status_code == 200 as “success” and calling .json() blindly turns both into a JSONDecodeError far from the cause.
Resilient portal download: status gate, content-type gate, error-envelope gate, then schema-validate and cache A top-to-bottom flow with a right-hand exception lane. The request passes through a retrying session. A status gate checks for HTTP 200; 429 or 503 routes right to a backoff-and-retry node, other non-200 codes raise. A content-type gate checks for application/json; an HTML or gzip body routes right to a decode-and-detect node that raises the underlying portal error. An error-envelope gate checks the parsed JSON for an errors array; if present it raises the portal message. Otherwise the data array is schema-validated, cached to disk idempotently, and returned as a validated table. Session.get() Retry adapter mounted status == 200? 429 / 503 honor Retry-After exponential backoff, retry yes JSON content-type? html / gzip decode & detect error raise portal message yes error envelope? yes raise portal error e.g. invalid api_key no schema-check + cache validated table returned

Pre-flight validation

Before a batch spends a quota, confirm the key is live and the endpoint is reachable. EIA v2 exposes a cheap metadata route (/v2/<route> with no data rows) that returns the series envelope; a single HEAD-like probe surfaces an invalid key or a dead route in one call instead of failing 400 requests deep.

Five response shapes, four gates, one success path A fan diagram. A single request node on the left branches into five possible responses: 200 with JSON data, 200 with text or HTML, 429 too many requests, 200 with a JSON error envelope, and a gzip body. Each response connects to the gate that detects it — the status gate, the content-type gate, the envelope gate, or the encoding gate — and each gate routes either to backoff and retry, to a raised error, or onward. The single path that clears every gate arrives at the schema validator and the local cache. Every one of these arrives with a 200 unless the status gate says otherwise GET request api_key + params 200 · application/json envelope + data 200 · text/html WAF or throttle page 429 · too many requests quota exhausted 200 · JSON error envelope {"error": "invalid key"} 200 · gzip body decoded as text = mojibake status gate raise_for_status() content-type gate assert json in ctype envelope gate check payload["error"] encoding gate resp.content, not .text backoff and retry raise fail loud schema validate
python
import os
import requests


def preflight_portal(base_url: str, api_key: str, probe_route: str) -> None:
    """Fail fast if the API key is missing/invalid or the endpoint is unreachable."""
    if not api_key:
        raise ValueError("No API key set. Export EIA_API_KEY / NREL_API_KEY.")

    probe = requests.get(
        f"{base_url}/{probe_route}",
        params={"api_key": api_key, "length": 1},
        headers={"Accept": "application/json"},
        timeout=15,
    )
    ctype = probe.headers.get("Content-Type", "")
    if "json" not in ctype:
        # A 200 with text/html is a throttle or WAF page, not data.
        raise RuntimeError(
            f"Non-JSON probe response ({ctype}); endpoint likely throttling or down."
        )
    body = probe.json()
    if isinstance(body, dict) and (body.get("error") or body.get("errors")):
        raise RuntimeError(f"Portal rejected the key/route: {body.get('error') or body['errors']}")
    print(f"[preflight] {base_url}/{probe_route} reachable; key accepted.")

Fix implementation

The resilient helper mounts a urllib3 Retry adapter so transient 429/5xx responses back off automatically, then explicitly gates content-type and the error envelope before touching the data. raise_on_status=False lets the retry logic exhaust its budget before we inspect the final response ourselves; respect_retry_after_header=True honors the portal’s own Retry-After.

The backoff between attempt retries follows urllib3’s formula, so with backoff_factor = 0.5 the delays grow geometrically:

giving 0.5s, 1s, 2s, 4s — enough to clear a short EIA throttle without stalling a pipeline for minutes.

python
import gzip
import json
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


def build_session(total_retries: int = 5, backoff_factor: float = 0.5) -> requests.Session:
    """A Session that retries 429/5xx with exponential backoff and honors Retry-After."""
    retry = Retry(
        total=total_retries,
        backoff_factor=backoff_factor,
        status_forcelist=(429, 500, 502, 503, 504),
        allowed_methods=frozenset({"GET"}),
        respect_retry_after_header=True,
        raise_on_status=False,
    )
    session = requests.Session()
    adapter = HTTPAdapter(max_retries=retry, pool_connections=8, pool_maxsize=8)
    session.mount("https://", adapter)
    session.headers.update({"Accept": "application/json", "Accept-Encoding": "gzip"})
    return session


def _parse_portal_json(resp: requests.Response) -> dict:
    """Decode a portal response, catching HTML/gzip-as-200 and error envelopes."""
    ctype = resp.headers.get("Content-Type", "")
    if "json" not in ctype:
        # requests auto-inflates declared gzip; a raw gzip body with a wrong
        # header will not be, so detect the magic bytes and inflate manually.
        raw = resp.content
        if raw[:2] == b"\x1f\x8b":
            raw = gzip.decompress(raw)
        try:
            body = json.loads(raw)
        except json.JSONDecodeError as exc:
            snippet = resp.text[:200].replace("\n", " ")
            raise RuntimeError(f"Expected JSON, got {ctype}: {snippet!r}") from exc
    else:
        body = resp.json()

    # EIA wraps errors as {"error": ...}; NREL/OpenEI as {"errors": [...]}.
    if isinstance(body, dict) and (body.get("error") or body.get("errors")):
        raise RuntimeError(f"Portal error: {body.get('error') or body['errors']}")
    return body

With the transport hardened, the paginated fetch walks EIA v2’s offset/length window until response.total is exhausted, accumulating every page instead of truncating at the first:

python
def fetch_eia_series(
    session: requests.Session,
    base_url: str,
    route: str,
    api_key: str,
    params: dict,
    page_size: int = 5000,
) -> list[dict]:
    """Fetch a full EIA v2 series across all pages; returns the flat data rows."""
    rows: list[dict] = []
    offset = 0
    while True:
        page_params = {**params, "api_key": api_key, "offset": offset, "length": page_size}
        resp = session.get(f"{base_url}/{route}", params=page_params, timeout=30)
        if resp.status_code != 200:
            raise RuntimeError(f"HTTP {resp.status_code} at offset {offset}: {resp.text[:200]}")

        envelope = _parse_portal_json(resp)["response"]
        rows.extend(envelope["data"])

        total = int(envelope.get("total", len(rows)))
        offset += page_size
        if offset >= total or not envelope["data"]:
            break
    return rows

Downloads are cached to disk idempotently — a deterministic filename keyed on the query means a re-run reads the parquet instead of re-hitting the quota, and a partial file is written atomically so an interrupted run never leaves corrupt cache:

python
import hashlib
from pathlib import Path
import pandas as pd
import geopandas as gpd


def load_or_download(cache_dir: str, base_url: str, route: str, api_key: str,
                     params: dict, session: requests.Session) -> gpd.GeoDataFrame:
    """Idempotent cache: return cached parquet or download, validate, and persist."""
    key = hashlib.sha256(f"{route}:{sorted(params.items())}".encode()).hexdigest()[:16]
    cache_path = Path(cache_dir) / f"{route.replace('/', '_')}_{key}.parquet"
    if cache_path.exists():
        return gpd.read_parquet(cache_path)

    rows = fetch_eia_series(session, base_url, route, api_key, params)
    sites_df = pd.DataFrame(rows)

    # Schema check: fail loudly if the portal dropped or renamed a column.
    required = {"period", "latitude", "longitude", "capacity_mw"}
    missing = required - set(sites_df.columns)
    if missing:
        raise KeyError(f"Portal schema drift: missing columns {missing}")

    sites_df["capacity_mw"] = pd.to_numeric(sites_df["capacity_mw"], errors="coerce")
    sites_gdf = gpd.GeoDataFrame(
        sites_df,
        geometry=gpd.points_from_xy(sites_df["longitude"], sites_df["latitude"]),
        crs="EPSG:4326",  # portal lat/lon is geographic; reproject downstream
    )
    cache_path.parent.mkdir(parents=True, exist_ok=True)
    tmp = cache_path.with_suffix(".parquet.tmp")
    sites_gdf.to_parquet(tmp)
    tmp.replace(cache_path)  # atomic swap; no half-written cache on interrupt
    return sites_gdf

The resulting sites_gdf is tagged EPSG:4326 — the geographic frame the portals publish in — and must be reprojected to a projected zone before any distance or area math, exactly the coordinate reference system alignment discipline the rest of the ingestion chain depends on.

Fallback routing & performance tuning

  • Cap concurrency below the quota. NREL enforces an hourly and daily key ceiling; run serial or with a small semaphore (2–3 in flight) and read X-RateLimit-Remaining to pause before you are throttled rather than after.
  • Widen the retry budget for overnight batches. Bump total_retries to 8 and backoff_factor to 1.0 for unattended runs, where a longer stall is cheaper than a failed job.
  • Prefer bulk archives for national pulls. When you need every EIA-860 plant or an OpenEI dataset in full, download the published CSV/ZIP archive once rather than paging the API thousands of times — reserve the API for incremental refreshes.
  • Set an explicit timeout on every call. A missing timeout lets a hung portal connection block a worker indefinitely; 30s per page is a safe ceiling.
  • Key the cache on portal version. Include the dataset’s last_updated (or EIA release date) in the cache filename so a portal revision invalidates stale parquet instead of serving it silently, feeding the same auditable lineage the broader geospatial data ingestion pipelines rely on.
Exponential backoff with full jitter, and why the jitter matters A horizontal timeline of six retry attempts. For each attempt a light bar shows the base delay — 1, 2, 4, 8, 16 and 32 seconds — and a darker mark inside it shows the actual wait drawn uniformly from zero to that base. A note contrasts the jittered schedule with a fixed one, where every worker in a fleet retries at the same instant and re-triggers the same rate limit. A cap line marks 60 seconds, past which the client should stop retrying and surface the failure. Exponential backoff, full jitter — the wait is a range, not a number attempt 1 base 1s · waited 0.6s attempt 2 base 2s · waited 1.4s attempt 3 base 4s · waited 1.9s attempt 4 base 8s · waited 5.2s attempt 5 base 16s · waited 9.8s attempt 6 base 32s · waited 21.5s 60 s cap — stop and report Without jitter every worker that hit the same 429 retries at the same instant and re-triggers it.

Downstream validation

Before the cached table feeds a scoring or resource workflow, gate it with an assertion suitable for a CI/CD pipeline. This catches truncated pagination (row-count shortfall), coordinate corruption, and dtype drift introduced by an upstream portal change:

python
def assert_download_integrity(sites_gdf: gpd.GeoDataFrame, expected_min_rows: int) -> None:
    """CI/CD gate: fail the build if the downloaded table is not analysis-grade."""
    assert len(sites_gdf) >= expected_min_rows, (
        f"only {len(sites_gdf)} rows (< {expected_min_rows}); pagination likely truncated"
    )
    assert sites_gdf.crs is not None and sites_gdf.crs.to_epsg() == 4326, "expected EPSG:4326"

    expected_cols = {"period", "latitude", "longitude", "capacity_mw"}
    assert expected_cols <= set(sites_gdf.columns), "expected column contract violated"
    assert str(sites_gdf["capacity_mw"].dtype).startswith("float"), "capacity_mw not numeric"

    # Reject impossible coordinates that signal a lon/lat swap or nodata sentinel.
    lat, lon = sites_gdf.geometry.y, sites_gdf.geometry.x
    assert lat.between(-90, 90).all() and lon.between(-180, 180).all(), "coordinates out of range"
    assert not sites_gdf["capacity_mw"].isna().all(), "capacity_mw fully null after coercion"

Logging the row count and portal version alongside this assertion is what makes the download reproducible: an independent engineer can confirm the exact series, page count, and quota state behind a resource layer — the same provenance rigor applied when validating NREL solar datasets with Python. Pin requests, urllib3, and geopandas versions so a retry-semantics change cannot silently alter batch behavior between runs.

Related articles

Back to Open Energy Data Portals