14 min read 2996 words

Caching and Rate Limiting NREL API Requests in Python

The scenario: a nightly resource refresh starts failing at 03:40 with 429 Too Many Requests, and the cause turns out to be an analyst exploring the same API interactively that afternoon on the same key. The quota is per key, the pipeline had no idea the budget was already spent, and every retry made it worse. This page builds the client that shares a budget and stops asking for things it already has, and it extends open energy data portals.

Root-cause analysis

Three properties of energy-portal APIs make naive clients fail predictably.

  1. Quotas are per key, not per process. NREL enforces an hourly and a daily cap against the API key, so any limiter scoped to one process is blind to every other consumer of that key — including a notebook and a colleague.
  2. The same request is made repeatedly. Resource data for a fixed point and year does not change between runs, yet a pipeline without a cache re-fetches it on every execution, spending quota on bytes it already has on disk.
  3. Retries amplify. A 429 answered by an immediate retry consumes another unit of quota and arrives sooner than the window resets, so a naive retry loop turns a brief throttle into a sustained one.
An hourly quota, accounted by consumer A stacked bar of a 1,000-request hourly quota divided by consumer: the nightly refresh at 340 requests, an interactive analyst session at 180, retries after a failure at 220, and a reserve of 50 held back. A separate bar shows 210 requests served from the local cache that never consumed quota at all. Annotations mark the retries as recoverable through a limiter that respects the rate-limit headers, and the reserve as what keeps an interactive user from being locked out by a batch job. 1 000 requests an hour, per key — not per process 340 180 220 210 nightly refresh — 340 requests interactive session — 180 requests retries after a failure — 220 requests unused — 210 requests reserve held back — 50 requests 210 further requests were answered from the local cache and never touched the quota at all The 220 retries are recoverable: back off on the rate-limit header, not on a guess The reserve is what stops a batch job locking an analyst out of a single lookup

Pre-flight validation

Before any request, know what the budget is and how much of it is left. NREL returns the remaining allowance in response headers, so the client can carry an accurate picture rather than a guess.

python
from dataclasses import dataclass


@dataclass
class QuotaState:
    limit_hour: int
    remaining_hour: int
    limit_day: int
    remaining_day: int

    @classmethod
    def from_headers(cls, headers) -> "QuotaState":
        """NREL publishes the remaining allowance on every response."""
        return cls(
            limit_hour=int(headers.get("X-RateLimit-Limit", 0) or 0),
            remaining_hour=int(headers.get("X-RateLimit-Remaining", 0) or 0),
            limit_day=int(headers.get("X-RateLimit-Limit-Day", 0) or 0),
            remaining_day=int(headers.get("X-RateLimit-Remaining-Day", 0) or 0),
        )

    def can_spend(self, n: int, *, reserve: int = 50) -> bool:
        """Keep a reserve so an interactive user is never locked out by a batch job."""
        return self.remaining_hour - n >= reserve and self.remaining_day - n >= reserve

The reserve is the part worth arguing for: a batch job that spends the last request of the hour leaves an analyst unable to check a single site, and the resulting workaround is usually a second key that nobody tracks.

Fix implementation

The client below is content-addressed, so an identical request is answered from disk, and rate limited through a shared token bucket, so every process on the key draws from one budget.

python
import hashlib
import json
import os
import time
from pathlib import Path

import requests


class SharedTokenBucket:
    """A token bucket held in a file, so several processes share one budget."""

    def __init__(self, path: str, *, rate_per_sec: float, capacity: int):
        self.path = Path(path)
        self.rate = rate_per_sec
        self.capacity = capacity
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def acquire(self, *, timeout_s: float = 120.0) -> None:
        deadline = time.monotonic() + timeout_s
        while True:
            state = self._read()
            now = time.time()
            tokens = min(
                self.capacity,
                state["tokens"] + (now - state["updated"]) * self.rate,
            )
            if tokens >= 1.0:
                self._write({"tokens": tokens - 1.0, "updated": now})
                return
            if time.monotonic() > deadline:
                raise TimeoutError("rate limiter: no token available within the timeout")
            time.sleep(max(0.05, (1.0 - tokens) / self.rate))

    def _read(self) -> dict:
        try:
            return json.loads(self.path.read_text())
        except (OSError, json.JSONDecodeError):
            return {"tokens": float(self.capacity), "updated": time.time()}

    def _write(self, state: dict) -> None:
        tmp = self.path.with_suffix(".tmp")
        tmp.write_text(json.dumps(state))
        os.replace(tmp, self.path)          # atomic, so a crash cannot corrupt the budget


def cached_get(
    url: str,
    params: dict,
    *,
    cache_dir: str,
    bucket: SharedTokenBucket,
    session: requests.Session | None = None,
    ttl_s: float | None = None,
) -> tuple[dict, dict]:
    """Content-addressed GET: identical parameters are answered from disk."""
    key_material = json.dumps({"url": url, "params": _without_key(params)}, sort_keys=True)
    key = hashlib.sha256(key_material.encode()).hexdigest()[:32]
    blob = Path(cache_dir) / key[:2] / f"{key}.json"

    if blob.exists() and (ttl_s is None or time.time() - blob.stat().st_mtime < ttl_s):
        return json.loads(blob.read_text()), {"cache": "hit", "key": key}

    bucket.acquire()
    sess = session or requests.Session()
    response = sess.get(url, params=params, timeout=60)
    response.raise_for_status()
    payload = response.json()

    blob.parent.mkdir(parents=True, exist_ok=True)
    blob.write_text(json.dumps(payload))
    quota = QuotaState.from_headers(response.headers)
    return payload, {"cache": "miss", "key": key, "quota": quota.__dict__}


def _without_key(params: dict) -> dict:
    """The API key must never enter the cache key — it is a credential, not a parameter."""
    return {k: v for k, v in params.items() if k.lower() not in {"api_key", "apikey"}}

Excluding the API key from the cache key matters twice over: it keeps a credential out of a filename, and it means two keys share one cache instead of duplicating every payload.

From request to cache key, with the credential removed A left-to-right flow. A request with a URL, latitude and longitude at full float precision, a year, and an API key enters a normalisation step that rounds the coordinates to four decimal places, sorts the parameter keys and removes the API key entirely. The normalised JSON is hashed with SHA-256 and truncated to 32 characters, producing a path of two hex characters as a directory and the full key as a filename. Two annotations record that removing the credential keeps it out of a filename and lets two API keys share one cache, and that a volatile parameter such as a timestamp is what usually destroys the hit rate. The cache key is the question, never the caller request lat=35.22201938 lon=-101.83130221 year=2020 api_key=SECRET normalise lat=35.2220 · lon=-101.8313 keys sorted api_key removed sha256[:32] a3/a3f19c…d41.json Removing the credential keeps it out of a filename and lets two keys share one cache A volatile parameter — a timestamp, a request id — is what destroys the hit rate Round coordinates to the precision the API actually honours: four decimal places is about 11 metres.

Fallback routing and performance tuning

  • Give each job its own key. Quotas are per key, so a dedicated pipeline key means an interactive session cannot exhaust the nightly budget, and the logs identify which consumer hit the limit.
  • Cache the raw payload, not the parsed frame. The parse is cheap and the bytes are the evidence; a raw cache also survives a change to the parser.
  • Set a TTL by data type, not globally. A historical NSRDB year never changes and deserves no TTL at all; a queue endpoint that updates monthly deserves a month.
  • Use conditional requests where the endpoint supports them. An If-None-Match that returns 304 costs a request against the quota but almost no bytes, which matters when the payload is large and the change is rare.
  • Back off on the header, not on a guess. When X-RateLimit-Remaining reaches zero, sleep until the window resets rather than retrying with exponential backoff into a wall.

Downstream validation

python
def assert_quota_accounting(stats: dict, *, min_hit_rate: float = 0.6) -> None:
    """A pipeline that re-fetches what it already has is spending quota on nothing."""
    total = stats["hits"] + stats["misses"]
    assert total > 0, "no requests recorded — the accounting is not wired up"
    hit_rate = stats["hits"] / total
    assert hit_rate >= min_hit_rate, (
        f"cache hit rate {hit_rate:.0%} below {min_hit_rate:.0%} — "
        "check the cache key for a volatile parameter such as a timestamp"
    )
    assert stats["throttled"] == 0 or stats["throttled"] / total < 0.02, (
        f"{stats['throttled']} throttled responses — the limiter is set above the real quota"
    )
Cache hit rate across a month, with one regression A line chart of daily cache hit rate over 30 nightly runs. The rate starts at zero on the cold first night, climbs steeply through 71 percent on night four, and plateaus between 92 and 95 percent from night ten. On night 18 it drops abruptly to 6 percent and recovers over the following three nights, annotated as a code change that introduced a timestamp into the cache key. A dashed threshold marks the 60 percent floor below which the assertion fires. Hit rate is the metric that notices a bad cache key 0% 25% 50% 75% 100% night 1 night 10 night 20 night 30 assertion floor 60% timestamp added to the cache key Nothing else moved: the run succeeded, the row counts were normal, and the only symptom was a quota spend that quietly went back to cold-cache levels.

What to cache, and for how long

Not every response deserves the same treatment, and a single global TTL is the usual reason a cache is either stale or useless.

Immutable by construction. A historical NSRDB year for a fixed point will never change: the source reprocesses whole archives on a multi-year cadence and publishes them under a new version. These deserve no TTL at all, only a version in the cache key, and they are the bulk of a resource pipeline’s traffic.

Slow-moving. Dataset catalogues, station metadata and model coefficients change a few times a year. A TTL measured in weeks is right, and the cost of being a week stale is nil.

Revised on a schedule. Monthly series that are back-revised — most EIA data — should carry a TTL just shorter than the publication cadence, so the pipeline picks up a revision on the first run after it lands rather than a month later.

Genuinely live. Real-time or day-ahead endpoints should not be cached beyond minutes, and arguably should not be cached at all; a stale price or a stale outage is worse than a slow one.

Tagging each endpoint with its class at the client boundary — rather than deciding per call site — keeps the policy in one place and makes the cache’s behaviour explainable. The accounting then splits cleanly: hits against immutable data are pure saving, and a low hit rate on live endpoints is expected rather than a defect.

Frequently asked questions

Why does the cache hit rate fall to almost zero after a code change?

Almost always because a volatile value entered the cache key — a timestamp, a request identifier, a float formatted with full precision. The fix is to normalise the parameters before hashing: round coordinates to the precision the API actually honours, sort the keys, and exclude anything that identifies the caller rather than the request.

Should the cache live on disk or in object storage?

Disk for a single machine, object storage for a fleet — and the same content-addressed key works for both. The property that matters is that the key is derived from the request, so two workers computing the same key find the same object without coordinating.

Is a token bucket better than a simple sleep between requests?

Yes, because it allows bursts up to the capacity while holding the average rate, which matches how quotas are actually enforced. A fixed sleep either wastes the burst allowance or exceeds the sustained rate, and it cannot be shared between processes.

What happens if two workers write the limiter file at once?

The atomic replace means neither sees a corrupt file, and the worst case is that one worker’s token accounting is briefly stale — it spends a token the other also spent. With a reserve in place that is harmless. If the fleet is large enough for that to matter, the same interface backs onto Redis with a Lua script and no other change.

Should the pipeline stop when the quota is nearly gone?

It should stop cleanly rather than fail messily. Reaching the reserve is a legitimate outcome: write what has been fetched, record which partitions are outstanding, and exit with a status that the scheduler can retry after the window resets. Burning through the reserve and then failing mid-write is worse in every respect.

How do I know how much quota a run will need?

Count the cache misses in a dry run. Because the cache key is deterministic, a pass that only checks for the presence of each key gives an exact miss count without spending a single request — which is enough to decide whether tonight’s run fits in the remaining budget.

Related articles

Back to Open Energy Data Portals