Enrichment API Rate Limits and Error Handling: A Developer's Survival Guide

Your enrichment pipeline is humming along at 100 requests per minute. Company data is flowing into your CRM, technographics are populating your lead scoring model, and your sales team finally has the context they need. Then it hits a 429 Too Many Requests. Then another. Then your queue backs up, your webhook retries pile up, and your CRM sync breaks. Suddenly, you're firefighting at 2 AM instead of shipping features.

If you've built anything that talks to an external API — especially company enrichment APIs — you've been here. The difference between a fragile integration and a production-grade one isn't the happy path. It's how you handle the failures.

This guide is the playbook I wish I'd had when I first started building enrichment pipelines. It covers the full spectrum: understanding rate limits, decoding error codes, implementing retry strategies that actually work, and building API clients that survive the real world. No hand-waving, no theoretical fluff — just battle-tested patterns with code you can ship today.

Understanding API Rate Limits

Rate limits are the guardrails every API provider puts in place to keep their infrastructure healthy. They exist for two fundamental reasons: server protection (preventing any single client from overwhelming shared resources) and fair usage (ensuring every customer gets reliable access). If you're building a company enrichment pipeline, understanding these limits isn't optional — it's table stakes.

Common Rate Limit Types

Not all rate limits work the same way. Most enrichment APIs enforce one or more of these:

• Per-minute limits: The most common type. You get X requests per 60-second window. Hit the ceiling, and you're blocked until the window resets. Typical values range from 60 to 600 requests per minute depending on your plan.
• Per-hour limits: A broader throttle that catches sustained high-volume usage. Even if you stay under the per-minute cap, sustained bursts over an hour can trigger this.
• Per-day limits: Often tied to credit-based pricing. You have a daily budget of API calls, and once it's exhausted, you're done until midnight UTC. This is especially relevant for domain-based lookup APIs where each lookup consumes credits.
• Concurrent request limits: The sneaky one. You might have a generous per-minute quota but can only run 5 requests simultaneously. Fire off 20 parallel requests and 15 of them fail — not because you exceeded your overall budget, but because you overwhelmed the concurrency gate.

Most APIs communicate their limits through response headers. The de facto standard uses X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. Some APIs also include a Retry-After header when you do get throttled. Parse these. Always. They're the difference between guessing and knowing.

Sliding Window vs. Fixed Window

There's an important nuance most developers miss. A fixed window resets at predictable intervals (e.g., every minute on the minute). A sliding window tracks your requests over a rolling period. With a fixed window, you can burst 100 requests at 0:59 and another 100 at 1:01 — 200 requests in 2 seconds. Sliding windows prevent this. Know which one your API uses, because your throttling strategy depends on it.

Common HTTP Error Codes and What They Mean

Every API speaks HTTP, and every HTTP response tells you a story. Here's the field guide for enrichment API error codes — what they mean, what caused them, and what to do about each one.

Code	Name	Meaning	Common Cause	Action
200	OK	Request succeeded	Everything worked as expected	Process the response. Still validate the payload — some APIs return 200 with empty or partial data.
400	Bad Request	Malformed request	Invalid parameters, missing required fields, malformed JSON	Fix the request. Do NOT retry — the same request will fail the same way. Log the payload for debugging.
401	Unauthorized	Authentication failed	Invalid, expired, or missing API key	Check your credentials. Rotate keys if compromised. Do not retry without fixing auth.
403	Forbidden	Authorized but not permitted	Accessing an endpoint not included in your plan, IP restrictions, or insufficient permissions	Check your plan's feature access. Contact support if unexpected.
404	Not Found	Resource doesn't exist	Invalid endpoint URL, or the company/domain you're looking up doesn't exist in the database	Verify the URL. For enrichment, cache the "not found" result to avoid wasting credits on repeated lookups.
429	Too Many Requests	Rate limit exceeded	Sending requests faster than your plan allows	Back off. Read Retry-After header. Implement exponential backoff with jitter. This is the big one — see below.
500	Internal Server Error	Server-side failure	Bug or crash on the API provider's end	Retry with backoff. If persistent, check the provider's status page. Not your fault, but your problem.
502	Bad Gateway	Upstream failure	Load balancer or proxy couldn't reach the backend	Retry after a short delay. Usually transient. If sustained, the provider is likely having an outage.
503	Service Unavailable	Server temporarily unavailable	Maintenance, overload, or deliberate throttling	Respect Retry-After if present. Back off and retry. Check status page for scheduled maintenance.

The critical insight: not all errors are retryable. A 400 will never succeed no matter how many times you retry it. A 429 will succeed if you wait. A 500 might succeed on retry. Your error handling code must distinguish between these categories. For deeper HTTP status reference, check the MDN HTTP Status documentation.

The Retry Strategy Playbook

Retrying failed requests sounds simple. It isn't. A naive retry loop can turn a minor rate limit hit into a full-blown outage — both for you and for the API provider. Here are the patterns that actually work in production.

Exponential Backoff

The foundational retry strategy. Instead of retrying immediately (which hammers the server harder), you wait progressively longer between each attempt. The formula is straightforward:

wait_time = base_delay * (2 ** attempt_number)

With a base delay of 1 second, your retries happen at 1s, 2s, 4s, 8s, 16s. Each failure doubles the wait, giving the server breathing room to recover.

import time
import requests

def request_with_backoff(url, headers, max_retries=5, base_delay=1.0):
    """
    Make an API request with exponential backoff retry logic.
    Only retries on transient errors (429, 500, 502, 503).
    """
    for attempt in range(max_retries):
        response = requests.get(url, headers=headers)

        if response.status_code == 200:
            return response.json()

        # Non-retryable errors — fail immediately
        if response.status_code in (400, 401, 403):
            raise APIError(
                f"Non-retryable error {response.status_code}: {response.text}"
            )

        # Retryable errors — back off and try again
        if response.status_code in (429, 500, 502, 503):
            # Respect Retry-After header if present
            retry_after = response.headers.get('Retry-After')
            if retry_after:
                wait_time = float(retry_after)
            else:
                wait_time = base_delay * (2 ** attempt)

            print(f"Attempt {attempt + 1} failed with {response.status_code}. "
                  f"Retrying in {wait_time:.1f}s...")
            time.sleep(wait_time)

    raise APIError(f"Max retries ({max_retries}) exceeded for {url}")

Key detail: always check for a Retry-After header before calculating your own backoff. The server is telling you exactly how long to wait. Ignoring it is disrespectful and inefficient.

Jitter

Exponential backoff has a problem: if 50 of your workers hit a rate limit at the same time, they'll all retry at the exact same intervals. 50 requests at T+1s, 50 at T+2s, 50 at T+4s. This is the thundering herd problem, and it can create wave after wave of rate limit hits.

The fix is jitter — adding randomness to the wait time so retries spread out naturally:

import random

def backoff_with_jitter(attempt, base_delay=1.0, max_delay=60.0):
    """
    Full jitter strategy: randomize between 0 and the exponential ceiling.
    This provides the best spread across concurrent retriers.
    """
    exp_delay = base_delay * (2 ** attempt)
    capped_delay = min(exp_delay, max_delay)
    return random.uniform(0, capped_delay)

# Usage in retry loop:
# wait_time = backoff_with_jitter(attempt)
# time.sleep(wait_time)

There are three jitter strategies (full, equal, and decorrelated), but full jitter — where you randomize between 0 and the exponential max — consistently performs best in practice. Amazon's engineering team published extensive research on this in their Builders' Library, and it's become the industry standard.

Circuit Breaker Pattern

Sometimes, retrying is the wrong answer entirely. If an API is down, hammering it with retries wastes your resources and adds load to an already struggling service. The circuit breaker pattern solves this by tracking failure rates and "opening the circuit" when failures exceed a threshold — stopping all requests until the service recovers.

import time

class CircuitBreaker:
    """
    Three states:
    - CLOSED: Normal operation, requests flow through
    - OPEN: Too many failures, all requests blocked
    - HALF_OPEN: Testing if service recovered
    """
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

    def __init__(self, failure_threshold=5, reset_timeout=60):
        self.state = self.CLOSED
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.last_failure_time = None

    def can_execute(self):
        if self.state == self.CLOSED:
            return True
        if self.state == self.OPEN:
            # Check if enough time has passed to try again
            if time.time() - self.last_failure_time >= self.reset_timeout:
                self.state = self.HALF_OPEN
                return True
            return False
        # HALF_OPEN: allow one request to test
        return True

    def record_success(self):
        self.failure_count = 0
        self.state = self.CLOSED

    def record_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = self.OPEN
            print(f"Circuit OPEN — blocking requests for {self.reset_timeout}s")

# Usage:
breaker = CircuitBreaker(failure_threshold=5, reset_timeout=60)

def make_request(url, headers):
    if not breaker.can_execute():
        raise ServiceUnavailable("Circuit breaker is open — API is down")

    try:
        response = requests.get(url, headers=headers)
        if response.status_code < 500:
            breaker.record_success()
        else:
            breaker.record_failure()
        return response
    except requests.exceptions.ConnectionError:
        breaker.record_failure()
        raise

The circuit breaker is especially valuable in enrichment pipelines where you might be processing thousands of companies. Without it, a provider outage means thousands of failed retries clogging your queue. With it, you fail fast and can route to a fallback or buffer requests for later.

Dead Letter Queue

Some requests are never going to succeed. The domain doesn't exist. The API key is wrong. The company was delisted. After exhausting retries, you need a place for these permanently failed requests — that's your dead letter queue (DLQ).

import json
from datetime import datetime

class DeadLetterQueue:
    def __init__(self, storage_path="dead_letters.jsonl"):
        self.storage_path = storage_path

    def push(self, request_data, error_code, error_message, attempts):
        entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "request": request_data,
            "error_code": error_code,
            "error_message": error_message,
            "total_attempts": attempts,
        }
        with open(self.storage_path, "a") as f:
            f.write(json.dumps(entry) + "\n")

    def replay(self, filter_fn=None):
        """Re-process entries, optionally filtering by criteria."""
        entries = []
        with open(self.storage_path, "r") as f:
            for line in f:
                entry = json.loads(line)
                if filter_fn is None or filter_fn(entry):
                    entries.append(entry)
        return entries

A good DLQ captures the full context: the original request, the error, the number of attempts, and the timestamp. This lets you diagnose patterns (are all failures from the same domain?), fix the root cause, and replay failed requests once the issue is resolved. In production, you'd typically use Redis, RabbitMQ, or your cloud provider's managed queue service rather than a file, but the principle is identical.

Building a Robust API Client

Individual patterns are useful. Combining them into a production-ready client is where it gets real. Here's a complete enrichment API client that brings together everything we've discussed — and this is the kind of client I'd put in front of any CRM enrichment workflow.

import time
import random
import logging
import requests
from collections import deque
from threading import Lock

logger = logging.getLogger(__name__)

class EnrichmentAPIClient:
    """
    Production-grade API client with:
    - Rate limit header parsing
    - Exponential backoff with full jitter
    - Circuit breaker
    - Request queuing with concurrency control
    - Comprehensive logging
    """

    def __init__(self, api_key, base_url, max_retries=5,
                 max_concurrent=5, requests_per_minute=100):
        self.api_key = api_key
        self.base_url = base_url.rstrip("/")
        self.max_retries = max_retries
        self.max_concurrent = max_concurrent
        self.requests_per_minute = requests_per_minute

        # Rate limiting state
        self.request_timestamps = deque()
        self.lock = Lock()

        # Circuit breaker
        self.breaker = CircuitBreaker(failure_threshold=10, reset_timeout=120)

        # Dead letter queue
        self.dlq = DeadLetterQueue()

        # Session with connection pooling
        self.session = requests.Session()
        self.session.headers.update({
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json",
            "User-Agent": "EnrichmentClient/1.0"
        })

    def _wait_for_rate_limit(self):
        """Enforce local rate limiting before sending requests."""
        with self.lock:
            now = time.time()
            # Remove timestamps older than 60 seconds
            while (self.request_timestamps and
                   self.request_timestamps[0] < now - 60):
                self.request_timestamps.popleft()

            if len(self.request_timestamps) >= self.requests_per_minute:
                oldest = self.request_timestamps[0]
                sleep_time = 60 - (now - oldest) + 0.1
                logger.info(f"Local rate limit hit. Sleeping {sleep_time:.1f}s")
                time.sleep(sleep_time)

            self.request_timestamps.append(time.time())

    def _parse_rate_limit_headers(self, response):
        """Extract rate limit info from response headers."""
        remaining = response.headers.get("X-RateLimit-Remaining")
        reset_at = response.headers.get("X-RateLimit-Reset")
        retry_after = response.headers.get("Retry-After")

        if remaining is not None:
            remaining = int(remaining)
            if remaining < 10:
                logger.warning(
                    f"Rate limit nearly exhausted: {remaining} remaining"
                )

        return {
            "remaining": remaining,
            "reset_at": reset_at,
            "retry_after": float(retry_after) if retry_after else None
        }

    def enrich_company(self, domain, fields=None):
        """
        Enrich a company by domain with full error handling.

        Args:
            domain: Company domain (e.g., 'stripe.com')
            fields: Optional list of specific fields to request
                    (reduces credit usage with modular enrichment)

        Returns:
            dict: Enriched company data

        Raises:
            APIError: On non-retryable failures
            ServiceUnavailable: When circuit breaker is open
        """
        if not self.breaker.can_execute():
            logger.error("Circuit breaker OPEN — skipping request")
            self.dlq.push(
                {"domain": domain, "fields": fields},
                error_code="CIRCUIT_OPEN",
                error_message="Circuit breaker prevented request",
                attempts=0
            )
            raise ServiceUnavailable("API circuit breaker is open")

        url = f"{self.base_url}/v1/company/enrich"
        params = {"domain": domain}
        if fields:
            params["fields"] = ",".join(fields)

        for attempt in range(self.max_retries):
            self._wait_for_rate_limit()

            try:
                response = self.session.get(url, params=params, timeout=30)
                rate_info = self._parse_rate_limit_headers(response)

                # Success
                if response.status_code == 200:
                    self.breaker.record_success()
                    logger.info(
                        f"Enriched {domain} — "
                        f"{rate_info['remaining']} credits remaining"
                    )
                    return response.json()

                # Non-retryable client errors
                if response.status_code in (400, 401, 403):
                    self.breaker.record_success()  # Server is healthy
                    logger.error(
                        f"Client error {response.status_code} for {domain}: "
                        f"{response.text}"
                    )
                    self.dlq.push(
                        {"domain": domain, "fields": fields},
                        error_code=response.status_code,
                        error_message=response.text,
                        attempts=attempt + 1
                    )
                    raise APIError(
                        f"Non-retryable: {response.status_code}"
                    )

                # Not found — cache it, don't retry
                if response.status_code == 404:
                    logger.info(f"No data found for {domain}")
                    return None

                # Rate limited
                if response.status_code == 429:
                    wait = rate_info["retry_after"] or backoff_with_jitter(
                        attempt, base_delay=2.0
                    )
                    logger.warning(
                        f"Rate limited on {domain}. "
                        f"Waiting {wait:.1f}s (attempt {attempt + 1})"
                    )
                    time.sleep(wait)
                    continue

                # Server errors — retryable
                if response.status_code >= 500:
                    self.breaker.record_failure()
                    wait = backoff_with_jitter(attempt)
                    logger.warning(
                        f"Server error {response.status_code} for {domain}. "
                        f"Retry in {wait:.1f}s"
                    )
                    time.sleep(wait)
                    continue

            except requests.exceptions.Timeout:
                logger.warning(f"Timeout for {domain} (attempt {attempt + 1})")
                time.sleep(backoff_with_jitter(attempt))
                continue

            except requests.exceptions.ConnectionError:
                self.breaker.record_failure()
                logger.error(f"Connection failed for {domain}")
                time.sleep(backoff_with_jitter(attempt))
                continue

        # All retries exhausted
        self.dlq.push(
            {"domain": domain, "fields": fields},
            error_code="MAX_RETRIES",
            error_message="All retry attempts exhausted",
            attempts=self.max_retries
        )
        logger.error(f"Failed to enrich {domain} after {self.max_retries} attempts")
        return None

A few things to notice in this implementation. First, it does local rate limiting before ever hitting the server — tracking request timestamps in a deque and sleeping proactively when approaching the limit. This prevents 429s rather than just reacting to them. Second, it parses rate limit headers on every response, giving you visibility into credit consumption in real time. Third, the circuit breaker, retry logic, and DLQ work together as a layered defense system. This is the kind of architecture that runs for months without intervention.

Rate Limit Optimization Strategies

The best way to handle rate limits is to not hit them in the first place. Here are six strategies that dramatically reduce your API call volume without sacrificing data quality.

1. Caching

Company data doesn't change every minute. A company's founding year, industry, and headquarters don't shift daily. Cache enrichment results aggressively — 24 to 72 hours for most firmographic data, shorter for dynamic signals like employee count or funding status. A simple Redis cache can cut your API calls by 60–80% if you're enriching leads that share company domains. For more on keeping cached data fresh without burning credits, see our guide on keeping company data fresh with enrichment.

2. Batch Requests

If your API supports batch endpoints, use them. Instead of 100 individual requests for 100 domains, send one batch request. You reduce HTTP overhead, simplify rate limit accounting, and typically get better throughput. Even if the API doesn't offer a formal batch endpoint, you can batch on your side by queuing requests and sending them in controlled bursts.

3. Off-Peak Scheduling

Rate limits are often shared across all customers on the same infrastructure. API performance tends to be better during off-peak hours (late night UTC, weekends). Schedule your bulk enrichment jobs for these windows. You'll hit fewer rate limits and get faster response times. This is particularly effective for nightly CRM sync jobs that don't need real-time results.

4. Modular Enrichment — Request Only What You Need

This is one of the biggest credit-saving strategies available with modern enrichment APIs. If you only need a company's industry and employee count for lead scoring, don't request the full profile with technographics, social links, and executive team data. APIs that support field selection or modular enrichment let you pay only for the data points you actually use. A full profile might cost 3 credits; a partial lookup might cost 1.

5. Webhooks vs. Polling

If the API supports webhooks for async enrichment, use them. Polling means you're making repeated requests asking "is it done yet?" — each one counts against your rate limit. Webhooks deliver the result to you when it's ready, consuming zero additional API calls. For enrichment workloads that take time (bulk jobs, deep company profiles), this is a significant optimization.

6. Deduplication

Before sending a request, check if you've already enriched that domain or company. It sounds obvious, but in complex pipelines with multiple ingestion points (web forms, CRM imports, CSV uploads, API integrations), the same company can enter your pipeline multiple times. A deduplication layer — even a simple set of recently-enriched domains — prevents wasted calls.

Monitoring Your API Usage

You can't optimize what you can't measure. A robust monitoring setup turns API integration from a black box into a transparent, debuggable system. Here are the metrics that matter and how to track them.

Essential Dashboard Metrics

• Requests per minute (RPM): Your primary throughput metric. Track it against your rate limit ceiling. If you're consistently at 85%+ of your limit, you're one traffic spike away from throttling.
• Error rate by status code: Break this down by code. A spike in 429s means your throttling isn't aggressive enough. A spike in 500s means the provider has issues. A spike in 400s means you pushed bad data.
• Latency P50 / P95: The median response time (P50) tells you normal performance. The 95th percentile (P95) reveals worst-case latency. If your P95 is 10x your P50, you likely have timeout or retry cascades happening. For enrichment APIs, healthy P50 is typically 200–500ms, and P95 should stay under 2s.
• Credit consumption rate: If you're on a credit-based plan, track daily burn rate and project when you'll hit your limit. Set alerts at 70% and 90% consumption.
• Circuit breaker state changes: Log every transition between CLOSED, OPEN, and HALF_OPEN. If your circuit breaker is opening frequently, there's a systemic issue.
• Dead letter queue depth: A growing DLQ means requests are failing permanently. Review and replay regularly.

Implementation

For startups and small teams, structured logging with a log aggregator (ELK stack, Loki, or even CloudWatch) is enough. At scale, purpose-built APM tools like Datadog, New Relic, or Grafana give you real-time dashboards, anomaly detection, and alerting out of the box.

Here's a minimal structured logging approach you can start with today:

import logging
import json
from datetime import datetime

class APIMetricsLogger:
    def __init__(self):
        self.logger = logging.getLogger("api_metrics")

    def log_request(self, domain, status_code, latency_ms,
                    credits_remaining=None, attempt=1):
        self.logger.info(json.dumps({
            "event": "api_request",
            "timestamp": datetime.utcnow().isoformat(),
            "domain": domain,
            "status_code": status_code,
            "latency_ms": round(latency_ms, 2),
            "credits_remaining": credits_remaining,
            "attempt": attempt,
            "is_retry": attempt > 1,
            "is_error": status_code >= 400
        }))

Structured JSON logs let you query, aggregate, and visualize your API usage with any log analysis tool. You can answer questions like "what percentage of requests to the enrichment API required retries last week?" or "which domains consistently fail?" — questions that are impossible to answer with unstructured print statements.

API-Specific Tips for BounceWatch

If you're using BounceWatch's enrichment API, here are specific optimizations that apply to our platform.

Rate Limits by Plan

BounceWatch's API uses tiered rate limits based on your subscription. Free plans start with conservative limits to let you evaluate the API. Growth and Enterprise plans unlock higher throughput and concurrency. Check your dashboard for your current limits, and remember that rate limit headers are included in every response — your client should always parse them.

Modular Enrichment for Credit Optimization

BounceWatch's API supports modular enrichment — you can request specific data modules rather than the full company profile. This is one of the most effective ways to stretch your credits. Need just firmographics for lead qualification? Request only the firmographic module. Need technographics for your ABM campaigns? Request tech_stack alone. Each module is priced independently, so you only pay for the intelligence you actually use. For a full comparison of enrichment approaches, see our comparison of the best enrichment APIs.

Webhook Setup for Async Processing

For bulk enrichment jobs, BounceWatch supports webhook-based delivery. Instead of polling for results, register a webhook URL and receive enriched data as it's processed. This eliminates polling overhead, reduces your effective API call count, and plays nicely with rate limits. Configure your webhook endpoint to handle retries (BounceWatch retries failed webhook deliveries with exponential backoff) and return a 200 quickly to acknowledge receipt.

Smart Refresh Strategy

Not all company data ages at the same rate. Funding information and employee count can change weekly. Industry classification and founding year rarely change. BounceWatch lets you set refresh intervals per data module, so you can re-enrich dynamic fields frequently while leaving static fields cached. This approach can reduce your monthly API usage by 40–60% compared to full-profile refreshes on a fixed schedule.

Putting It All Together

Handling API rate limits and errors isn't glamorous work. It won't make your demo more impressive or your pitch deck shinier. But it's the difference between an integration that runs reliably for months and one that wakes you up at 3 AM. Let's recap the key principles:

1. Parse rate limit headers on every response. Don't guess — the server is telling you what it needs.
2. Categorize errors before retrying. 400s are your bug. 429s need patience. 500s need backoff. Know the difference.
3. Use exponential backoff with full jitter. It's the proven standard, not just a nice-to-have.
4. Implement a circuit breaker. When the API is down, stop hammering it. Fail fast, recover gracefully.
5. Build a dead letter queue. Don't lose failed requests — capture them for analysis and replay.
6. Optimize before you scale. Caching, batching, modular enrichment, and deduplication reduce your call volume by 50–80% before you need to upgrade your plan.
7. Monitor everything. RPM, error rates, latency percentiles, and credit burn rate should be visible at a glance.

For further reading on resilience patterns, Martin Fowler's writing on the circuit breaker pattern is the canonical reference. The AWS Builders' Library has excellent deep-dives on timeouts, retries, and backoff. And Stack Overflow remains the best place to debug specific error codes and library quirks when you're knee-deep in implementation.

The patterns in this guide aren't theoretical. They're what separates a weekend prototype from production infrastructure. Implement them once, and your enrichment pipeline will run quietly in the background while you focus on building the features that actually move the needle.