RPC Retry

Overview

RPC Retry is a repeated remote procedure call after a transient failure, timeout, throttling response, or unavailable endpoint. A read query can often be retried more safely than transaction submission, where the original request may have succeeded even if the response was lost.

The main risks include duplicate broadcast, retry storms, stale reads, masking permanent errors, and exceeding the user-facing latency budget. Retry behavior must account for whether the method only reads state or may create a side effect such as transaction broadcast. Configuration changes to RPC Retry should be versioned and tested under burst, timeout, dependency failure, and restart conditions.

In production, teams should define ownership and apply retryable error classification, exponential backoff with jitter, attempt caps, idempotency or transaction-hash checks, endpoint failover, and deadline budgets. The RPC Retry policy should define which failures are retryable, how long the decision remains valid, and how duplicate effects are prevented.

Useful measures include retry success rate, attempts per RPC call, duplicate broadcast count, exhausted retries, and added retry latency. RPC Retry is closely connected to Exponential Backoff, RPC Error, and RPC Failover. Metrics for RPC Retry should separate first-attempt success, recovered operations, exhausted attempts, duplicate prevention, and added latency.

For RPC Retry, a missing response should remain uncertain until status lookup or reconciliation confirms whether the original operation executed.

The RPC Retry implementation should bound attempts, preserve the original operation identity, and stop before retries amplify a dependency failure. Operational review of RPC Retry should separate successful recovery from repeated attempts that only increase latency or load.