August 2026

How We Survived an IP Ban

Our SaaS talks to a large CRM platform all day. They send us a webhook when a lead changes. We call their API back to read the record. A few hundred calls per hour. Quiet, boring work.

On August 5th, every one of those calls died.

No error code. No warning email. Requests just hung and timed out. From any other network their servers answered in 300 milliseconds. From ours — nothing. The platform had banned our server's IP.

The bug was ours

One of our customers ran a mass update in their CRM. Thousands of records changed in an hour. Every change sent us a webhook. Every webhook made us call the API back. The platform's limit is 2 requests per second.

We had a rate limiter for exactly this. It had one flaw. When a request waited too long for a free slot, it gave up waiting — and went out anyway.

Picture a doorman told to let in 2 people per second. The line grows. He shrugs and opens the door. That was our limiter.

That "shrug" fired 6,419 times in one day. We sat above the limit for hours. The platform's protection answered the only way an automated system can: it dropped every packet from our IP. Not an HTTP 429. Silence. 

before the ban:  request → response   (~300 ms)
after the ban:   request → nothing    (timeout)
from other IPs:  request → response   (~300 ms)

And here is the cruel part. The ban only cut our outgoing traffic. Their webhooks kept arriving — 70 per minute. Work kept piling in, and we could not process any of it. In one day the queue grew to 23,000 jobs.

Day one: fix the cause

Blaming the platform would have been easy. Their protection did its job. Ours didn't. Two fixes went out the same evening:

  1. The doorman stopped shrugging. If a request can't get a slot, it goes back to the queue and tries later. Delayed, never skipped.
  2. We added a circuit breaker. A few failed connections in a row — and we stop calling that server entirely. Retrying a server that ignores you is worse than useless during a ban: every retry tells their defense system you are still knocking. Silence is what gets a ban lifted.

Then we found a leak in our own silence. The breaker re-checked each blocked server every 5 minutes, and each check burned several real connection attempts. Hundreds of packets per hour — from an IP trying to prove it had gone quiet. We stretched the interval to one hour. Traffic dropped to almost nothing.

Day two: stop waiting

We wrote to support: here is what happened, here is what we fixed, please unblock the IP. Support queues answer in days. Our customers' fresh leads were aging in ours. Waiting was not a plan.

A ban applies to one IP address. Requests from a different address go through. We had another server in another datacenter. So: tunnel. 

┌──────────────┐        SSH tunnel        ┌──────────────┐
│  app server  │ ───────────────────────▶ │ relay server │
│ (banned IP)  │                          │  (clean IP)  │
└──────────────┘                          └──────┬───────┘
                                                 │
                                                 ▼
                                          CRM platform
                                     (sees the clean IP)

The whole thing is three pieces:

The relay runs another production project, so the rules were strict. The proxy listens on localhost only — the internet cannot see it, and the firewall stayed untouched. The SSH key is caged in authorized_keys: restrict,port-forwarding,permitopen="127.0.0.1:13128" — no shell, no commands, one forwardable port. And the proxy connects only to the platform's domains. If the key leaked tomorrow, it would be almost useless.

One Docker detail ate fifteen minutes: inside a container, 127.0.0.1 is the container, not the host. The tunnel also had to listen on the Docker bridge IP. Fifteen minutes I will never get back.

Day two, evening: 23,000 jobs, carefully

The tunnel went live. 23,000 jobs woke up — aimed at a platform that had just banned us for sending too much. The 2/second cap kept us polite. But now the queue had a fairness problem: it was mostly "record updated" noise, and the few hundred new leads — the thing customers actually open the dashboard for — stood at the back of the line.

The fix was one SQL statement. Postpone every update job by two hours. New leads got the full 2/second and finished in under an hour. The updates ground through the afternoon. Same work, different order — and customers only ever see the order.

Everything that died during the ban sat in a failed-jobs table. We replayed about a thousand events a day late. Every one processed correctly, because our handlers are idempotent: they read the record's current state when they run, not the state from the stale event. Zero leads lost.

What this cost and what it bought

The ban is still not lifted — support is support. But the product ran, nobody's data was lost, and removing the whole detour will take one env variable and one systemd unit.

What I'm keeping from this week:

  1. A rate limiter with an escape hatch is a countdown timer. Ours took years to fire. It fired.
  2. When a server goes silent, go silent back. Every retry against an abuse filter is a confession that you haven't changed.
  3. Make your outbound IP replaceable before anyone makes it worthless. A proxy option in the HTTP client is ten lines. We wrote them during the fire. Write yours before.
  4. Idempotent handlers turn disasters into delays. A day of failed jobs replayed cleanly because nothing depended on being on time.
  5. Temporary infrastructure deserves permanent security. The localhost bind, the caged key, the domain allowlist — none of it was extra work. All of it lets me sleep.

The uncomfortable summary: the platform banned us, and the platform was right. The interesting work was never getting unbanned. It was building a system where a banned IP is a Tuesday, not a funeral.

← Back to salimbay.com