Timeouts, Retries & Idempotency

Learning goals

By the end of this chapter you should be able to:

• Set sensible client timeouts (connect, read, total) and explain why “no timeout” is a bug
• Describe how retries amplify load during partial outages
• Use idempotency keys so safe retries do not double-charge
• Sketch circuit breakers as a load-protection pattern (light touch)

Networks fail slowly, partially, and together. Reliability is not “avoid all errors” — it is controlling what happens when errors arrive.

Plain English: waiting forever is a strategy (a bad one)

Your HTTP client opens a TCP connection to an API. The server process is wedged. Packets go nowhere useful.

Without a timeout, your thread (or user) waits indefinitely — thread pools exhaust, checkout pages spin, cascading slowness kills healthy parts of the system.

Timeouts are contracts: “I will wait at most this long, then I fail and free resources.”

Precise terms:

• Connect timeout — how long to establish TCP (and often TLS).
• Read / idle timeout — max silence waiting for next byte after connected.
• Write timeout — max time to send request body.
• Total deadline — wall-clock cap for entire operation (strongest user-facing guarantee).

Use multiple layers — a read timeout alone does not help if DNS hangs unless connect is bounded too.

Client vs server timeouts

Both sides need limits:

• Client protects your UX and thread pools.
• Server protects workers from slow clients (read timeout on incoming request).

Mismatch causes confusion: client gives up at 30s while server still processing → client retries → duplicate work on server.

Align timeouts with SLA and upstream limits (LB idle timeout often 60s on AWS ALB — longer app work needs streaming, async job, or WebSocket pattern).

Retries: helpful until they are a DDoS from yourself

Transient failures — single packet loss, brief 503, leader election blip — often succeed on retry.

Exponential backoff with jitter: wait 100ms, 200ms, 400ms… plus randomness so 10,000 clients do not retry in sync (retry storm).

Only retry when:

• Error is likely transient (503, connection reset, timeout).
• Operation is safe to repeat (idempotent) OR you have deduplication.

Never blindly retry:

• POST /charge without idempotency — user billed twice.
• 4xx (except 429 rate limit with Retry-After) — your request is wrong; retry won’t help.
• During full outage — retries multiply traffic; healthy shards get hammered (retry amplification).

Precise term: retry amplification — when failed requests + aggressive retries increase load above normal, turning partial failure into total failure.

Idempotency: same request, same effect

An operation is idempotent if performing it once or many times leaves the system in the same state.

HTTP method folklore (useful but not absolute):

Real APIs use idempotency keys: client sends header Idempotency-Key: uuid on POST; server stores outcome keyed by that UUID — duplicate submits return same result without re-running side effects.

Payment and order APIs depend on this. Keys should expire after a window (24–72h typical).

At-least-once delivery (queues, webhooks) requires idempotent consumers — you will see duplicates.

Circuit breakers (light introduction)

A circuit breaker wraps calls to a dependency:

• Closed — normal calls through.
• Open — too many failures → fail fast without calling sick dependency (return cached default or error immediately).
• Half-open — after cooldown, allow probe requests to test recovery.

Why: gives downstream time to recover; prevents your service from burning threads on hopeless waits.

Related patterns: bulkheads (isolate thread pools per dependency), rate limiting (cap ingress), load shedding (drop low-priority work under stress).

You do not need a library on day one — fail fast + backoff + health-aware routing is the core idea.

Timeouts and the full stack

Remember the path:

1. DNS (can hang — use resolver timeout)
2. TCP connect (SYN may DROP → long wait without connect timeout)
3. TLS handshake
4. HTTP request/response
5. LB idle timeout may cut long polls

Observability: log which phase failed (connect vs read vs 503). Metrics: p50/p99 latency per hop if possible.

Designing retry policies (checklist)

• Max attempts (e.g. 3 total).
• Backoff with jitter.
• Retry only idempotent ops or with idempotency key.
• Honor Retry-After header on 429/503 when present.
• Deadline propagation — if total budget is 2s, don’t retry three 1s calls.

Common mistakes

1. Infinite or default-no timeout in HTTP clients — #1 production footgun.
2. Retrying POST /payments on any IOException.
3. Same timeout for connect and 5 GB download — use appropriate read timeout or streaming.
4. Retry storm during incident — disable retries or widen backoff when error rate spikes (feature flag).
5. Assuming TCP success means HTTP success — always handle status codes and body errors.

Practice checkpoint

1. Connect timeout vs read timeout? Answer: connect bounds TCP/TLS establishment; read bounds waiting for response bytes after connected.

2. Why jitter in backoff? Answer: desynchronizes retries from many clients to avoid synchronized spikes.

3. POST creates order without idempotency key; client retries on timeout — risk? Answer: duplicate orders/charges.

4. Circuit breaker open — what happens to new calls? Answer: fail fast locally without hitting failing dependency (optional fallback).

5. 503 from overloaded service — retry helpful? Answer: sometimes with limited backoff; unbounded retry worsens overload (amplification).

Interview angles

• Timeout layers on a typical microservice call chain.
• Idempotency key storage and replay semantics.
• At-least-once vs exactly-once — exactly-once end-to-end is hard; idempotent design + dedup is practical.
• Retry amplification during brownouts — why SREs temporarily disable retries.
• Graceful degradation vs hard fail when circuit open.

Next: /networking/learn/reliability/interview-checklist/