Part VIII: Orchestration & The Second Gilded Age

Chapter 32 — Routing, Retries & Smart Fallbacks

Part VIII: Orchestration & The Second Gilded Age

Running scenario: MegaMart Online — a mid-size e-commerce merchant selling consumer electronics across Southeast Asia and Europe

A customer in Jakarta taps “Pay” on an $89 headphone purchase from MegaMart Online. The screen shows a spinner. What the customer does not see is a decision system running against the clock.

In the next 800 milliseconds, MegaMart’s orchestration layer must answer a question with no single right answer: which route gives this transaction the best chance of approval, at the lowest cost, without violating the rules of the issuer, the card network, or the acquirer?

The system checks the card’s BIN — it is a Visa card issued by Bank Central Asia. The primary route goes through PSP A, which has a direct connection to a local Indonesian acquirer. But PSP A’s error rate has been climbing for the past 20 minutes — their monitoring shows a 12% timeout rate, up from a baseline of 0.3%. The secondary route goes through PSP B, which routes through a regional acquirer in Singapore. Lower timeout risk, but higher latency, a cross-border fee, and historically 3% lower approval rates for Indonesian-issued cards.

The system picks PSP A — the timeout rate is elevated but not catastrophic, and the local acquirer connection is worth the risk. The transaction goes through. Approved, 340 milliseconds.

If PSP A had timed out, the system would have needed a plan: retry on the same route? Fail over to PSP B? Ask the customer to authenticate with 3DS? Give up and show an error? Each of those decisions has consequences — for conversion, for cost, for compliance, and for the customer’s trust.

This chapter is about how those decisions actually work.

The Optimization Problem Behind Every Payment

Every payment attempt is a decision under time pressure. The customer expects a result in seconds. The system must select a route, handle failure if it occurs, and do both without creating duplicate charges, violating network rules, or adding enough latency that the customer abandons the purchase. It is, in the language of engineering, a constrained optimization problem — and the constraints are not just technical.

Four competing objectives shape every routing decision.

The first is approval probability — what payment professionals call the authorization rate. Will the issuer approve this specific transaction? The answer depends on factors the merchant only partially controls: the card’s available balance, the issuer’s fraud models, the merchant’s MCC (merchant category code — the four-digit identifier that tells issuers what kind of business you are), the authentication state, the data quality of the request, and whether the issuer is even online. A transaction that looks identical from the merchant’s perspective — same card, same amount, same merchant — can get approved by one issuer and declined by another, or approved today and declined tomorrow. Authorization is probabilistic, not deterministic.

The second objective is latency and user experience. More routing attempts can increase the chance of eventual approval, but each attempt adds time. A customer staring at a spinner for eight seconds is a customer reconsidering the purchase. In mobile commerce, where attention spans are measured in seconds, the cost of a slow retry can exceed the cost of a decline. The system must balance persistence against patience.

The third is cost. Every route has a price: processor fees, acquirer markup, interchange, scheme fees, and foreign exchange costs if the transaction crosses borders. Retry attempts add cost even when they succeed — and when they fail, some card networks charge penalty fees for excessive reattempts. Visa charges $0.10 per excessive domestic retry. Mastercard’s fees reach $0.50 per transaction as of January 2025. These penalties are not hypothetical. They show up on your acquirer statement, and at scale, they add up fast.

The fourth objective is risk and compliance. Not every decline is a suggestion to try again. Some declines are instructions: the issuer is telling you the card is stolen, the account is closed, or the cardholder has explicitly canceled the payment arrangement. Retrying these is not just pointless — it violates network rules and triggers penalties. The system must distinguish between “try again differently” and “stop trying” — and it must do so by reading signals that are often ambiguous.

These four objectives frequently conflict. The cheapest route may have the lowest approval rate. The fastest route may not comply with authentication requirements. The route with the best historical approval rate may be experiencing an outage right now. Routing is not picking from a menu — it is navigating trade-offs in real time, with incomplete information, under a time constraint that the customer’s patience defines.

Here is the mental model shift that separates architects who struggle with payments from those who build systems that last: routing is not a feature you buy from a vendor. It is a control plane you operate. Your orchestration layer is making policy decisions — which providers to trust, when to retry, when to stop, what data to send — and those decisions need telemetry, feedback loops, and governance, just like any other control plane in your infrastructure.

Stripe describes its own routing optimization as “smart decisions in milliseconds” — and that framing is useful even if you are not a Stripe customer. The “smart” part is not magic. It is a system that observes outcomes, updates models, and adjusts decisions based on what it has learned. The “milliseconds” part is the constraint that makes it hard: you cannot run a committee meeting for every transaction. The decisions must be automated, bounded, and safe by default.

Back in Jakarta, MegaMart’s customer has their headphones confirmed. The 340-millisecond decision involved all four dimensions: the system picked the route with the highest expected approval rate (local acquirer), accepted moderate risk (elevated timeout rate), chose a cost-competitive path (domestic routing avoids cross-border fees), and confirmed the transaction did not require step-up authentication (the card had been authenticated in a previous session and was flagged for frictionless flow).

One transaction. Four trade-offs. Hundreds of milliseconds. And that was the easy case — the one where the first attempt worked. The harder cases are what the rest of this chapter is about: what happens when it doesn’t.

Routing as a Constrained Decision System

"Routing" sounds simple: pick which processor handles this transaction. In practice, routing means selecting the best feasible path based on the transaction's attributes and the system's current conditions. Depending on your architecture, that path might span a choice of processor, gateway, acquiring connection, or even the payment rail itself — cards versus A2A versus a local wallet.

What makes routing an engineering discipline rather than a configuration checkbox is that three constraints operate simultaneously. Each one is manageable alone. Together, they create a problem space that requires real system design.

You Don't Control the Issuer — So You Control the Request

The issuer decides whether to approve a transaction. You cannot change their fraud models, their risk appetite, or their system availability. What you can control is the shape and context of the authorization request you send: how complete the data is, whether authentication has been performed, what type of credential you present (raw PAN versus network token), and which network entry point you use.

This is the core insight behind Stripe's Adaptive Acceptance, which uses machine learning to adjust request parameters — formatting, timing, credential type — within milliseconds to improve acceptance. The system learned, for example, that certain Indonesian issuers approve network token transactions at higher rates than raw PAN transactions, because the token signals to the issuer that the credential is current and the merchant is known. Stripe's model recovered $6 billion in falsely declined transactions in 2024 by optimizing these request-level factors, without changing the route at all.

For MegaMart, this means their Jakarta transaction is not just sent to PSP A and hoped for the best. Before submission, the orchestration layer enriches the request with a device fingerprint, attaches the 3DS authentication result from the customer's previous session, and selects a network token over the raw PAN. Each of these decisions improves the probability of approval — not by changing the route, but by changing the request.

Every Route Has Eligibility Rules

A route is only valid if it can actually process the transaction. This sounds obvious until you map it. Each processor has constraints: supported payment methods, supported currencies, geographic licensing, risk profile requirements, and technical capabilities. A transaction that works on Route A may be ineligible on Route B — not because Route B is worse, but because it literally cannot process that combination.

MegaMart's PSP A can process Visa cards in Indonesia through a local acquirer, but it does not support GoPay, Indonesia's dominant mobile wallet. PSP B handles GoPay but charges higher fees for Visa card transactions routed through its Singapore-based acquiring connection. The orchestration layer must know these eligibility constraints before it can make a routing decision — and those constraints change as providers update their capabilities, add markets, or sunset integrations.

Orchestration platforms — whether built in-house or provided by vendors like Spreedly, Primer, or Gr4vy — earn their keep by normalizing multiple provider integrations into a single abstraction layer, then applying per-transaction routing logic on top. The value is not just "connect to more processors." It is maintaining an accurate, current map of what each processor can and cannot do.

Routing Needs Determinism Until It Must Break Determinism

For clean reconciliation, predictable settlement, and protection against duplicate charges, the first attempt on any transaction should be deterministic: this type of transaction, from this merchant, in this currency, normally goes to Route A. Determinism makes your system auditable and your finance team sane.

But determinism must yield when the route is unhealthy. If Route A's error rate spikes from 0.3% to 12%, continuing to send traffic there because "that's the policy" turns your routing logic into a single point of failure. The system must detect degradation and shift traffic — quickly, safely, and with bounds that prevent cascading failures.

Modern routing systems blend three modes: a default selection based on static policy (transaction attributes map to a preferred route), health-aware overrides that activate when monitoring detects degradation, and bounded fallbacks that limit how many alternative routes the system will try before surfacing a failure to the customer. We will unpack retries and fallbacks in the next sections. But the architecture pattern starts here: deterministic by default, adaptive when necessary, bounded always.

DimensionWhat It DeterminesExample
Payment methodWhich PSPs can process itGoPay only via PSP B; Visa via PSP A or B
CurrencyAcquirer and FX routingIDR transactions routed to local Indonesian acquirer
Issuer country / BINWhich route has best approval rate for that issuerIndonesian BINs → local acquiring path for higher approval
Authentication state3DS exemption eligibilityLow-risk transaction → attempt without 3DS first
Token typePAN vs network token pathNetwork token → higher approval probability, lower interchange
PSP healthFailover trigger thresholdPSP A error rate > 5% → route to PSP B
Cost tierFee optimizationDomestic routing saves 30 basis points on interchange

Table 1: Seven dimensions that routing logic must evaluate per transaction. The orchestration layer must know each dimension's current state to make a valid routing decision.

Retries That Don't Create Outages or Double Charges

Payments orchestration inherits the hardest problem in distributed systems: uncertainty. When you send an authorization request to a PSP and nothing comes back, you do not know what happened. Did the issuer decline the transaction? Did the request time out before reaching the issuer? Did the issuer approve but the response got lost on the way back? Each of these scenarios demands a different response — and getting it wrong means either losing a sale or charging the customer twice.

This is the uncertainty problem that makes retries dangerous in payments. In a typical web application, retrying a failed API call is low-risk — the worst case is usually a duplicate log entry or an extra database write. In payments, the worst case is taking money from someone's account twice. The stakes change everything about how you design retry logic.

Idempotency: The Money Safety Primitive

The foundational defense against double charges is idempotency — the guarantee that performing the same operation multiple times produces the same result as performing it once. In payments, this means that if your system sends the same authorization request twice (because the first attempt timed out), the PSP processes the charge exactly once and returns the same result both times.

Every major PSP implements this through request-level identifiers. Stripe uses an Idempotency-Key header on POST requests — a unique string (up to 255 characters) that your system generates per payment attempt. If Stripe receives a second request with the same key, it returns the result of the original request without processing a new charge. Keys expire after 24 hours, and Stripe's v2 API will re-execute a request if the original attempt failed, while still preventing duplicate successful charges. PayPal takes the same approach with its PayPal-Request-Id header — a UUID attached to each POST call that lets PayPal return the latest status of the original request on any replay.

The concept has been formalized beyond individual PSPs. The IETF's httpapi working group has been developing a standardized Idempotency-Key request header specification — still a draft as of late 2025 (version 07, authored by Jena and Dalal), but on the standards track. The specification defines how HTTP clients and servers should handle idempotent retries for non-idempotent methods like POST, providing a common protocol for the pattern that payment APIs have been implementing independently.

The key insight: idempotency is not developer convenience. It is a money safety primitive. Without it, every timeout becomes a potential double charge, every network glitch becomes a reconciliation nightmare, and every retry becomes a risk. MegaMart's orchestration layer generates a unique idempotency key for every payment attempt and preserves that key across retries and failovers — so when PSP A times out and the system fails over to PSP B, the key travels with the request, preventing any possibility of both PSPs successfully processing the same charge.

Diagram 1: Idempotent retry flow. The orchestration layer retries with the same key after a timeout. The PSP recognizes the key, returns the original result, and no duplicate charge is created.

Backoff, Jitter, and Retry Budgets

Idempotency prevents double charges. But it does not prevent your retry logic from making an outage worse. If PSP A is struggling under load and your system immediately retries every failed request, you are doubling the traffic hitting an already-stressed service. If every merchant using PSP A does the same thing simultaneously, the retry storm can turn a partial degradation into a complete outage.

The engineering discipline here comes from reliability engineering, not payments specifically. Google's SRE handbook (Chapter 22, "Addressing Cascading Failures") states the principle clearly: "Always use randomized exponential backoff" and "limit retries per request." Exponential backoff means each successive retry waits longer than the previous one — 100ms, then 200ms, then 400ms, then 800ms — giving the failing service time to recover. Jitter adds randomness to those wait times so that thousands of clients are not all retrying at the exact same moment. AWS's architecture guidance, particularly Marc Brooker's analysis of backoff algorithms, recommends "Full Jitter" — randomizing the wait time between zero and the exponential ceiling — as the most effective approach for reducing retry storms.

Retry budgets add a system-level constraint: cap the total retry traffic as a percentage of your baseline request rate. If your system normally sends 1,000 requests per minute to a PSP, a retry budget of 10% means retries can never exceed 100 additional requests per minute, no matter how many original requests are failing. This prevents the scenario where retries overwhelm a recovering service.

A circuit breaker takes this further — when a route's error rate exceeds a threshold, stop sending traffic entirely for a cooldown period, then gradually reintroduce requests to test whether the service has recovered. The pattern is borrowed directly from electrical engineering: when current exceeds safe levels, the circuit opens and stops the flow.

The bottom line: before you design "smart fallbacks," implement "boring reliability" first. Timeouts, bounded retries, jitter, and circuit breakers are the foundation. Without them, your sophisticated routing logic becomes a mechanism for amplifying failures instead of recovering from them.

Two Kinds of Retries People Confuse

There is a critical distinction that many teams miss when building orchestration systems, and confusing these two concepts leads to real compliance problems.

Transport retries are retries of your API call to a PSP when the call itself fails — a timeout, a dropped connection, an HTTP 5xx error. The PSP may never have received your request, or it received it but could not send a response. Transport retries use idempotency keys and backoff. This is distributed-systems engineering — the same patterns you would use for any unreliable network call.

Payment reattempts are new authorization attempts after the issuer has declined the transaction. The PSP received your request, forwarded it to the issuer, and the issuer said no. A reattempt is a new business decision: should we ask the issuer again, perhaps with different data or authentication? Payment reattempts are governed by card network rules — Visa and Mastercard have explicit limits and fee penalties for excessive reattempts, which we will cover in the next section.

Your orchestration system must track and distinguish both types — often within a single customer-visible payment attempt. A customer taps "Pay," the first attempt times out (transport failure), the system retries with an idempotency key (transport retry), the retry goes through but the issuer declines for "authentication required" (issuer decline), and the system steps up to 3DS and resubmits (payment reattempt). That is two different kinds of retries in one checkout flow, governed by two different sets of rules.

Payment Retries Under Network Rules and Issuer Intent

The fastest way to build a problematic orchestration system is to treat every issuer decline as a generic "try again later" signal. Card networks do not leave retry behavior to your judgment. They categorize declines explicitly and constrain what you may do next — with escalating financial penalties for merchants who ignore the signals.

MegaMart learned this the expensive way. In their first month with multi-PSP orchestration, their retry logic treated all declines identically: wait, retry, cascade to the next PSP. The system was technically impressive — low latency, smooth failover, high persistence. It also generated $12,000 in excessive reattempt fees from Visa in a single billing cycle, because it was retrying transactions the issuer had already said would never be approved.

Visa's Decline Categories and Reattempt Limits

Visa groups decline response codes into four categories, each with different rules about what happens next.

Category 1 is the hard stop. These are declines where the issuer will never approve the transaction for that credential — the card is reported stolen, the account is closed, the card number does not exist, or the issuer has explicitly flagged the account. When you receive a Category 1 decline, the merchant must never resubmit an authorization request for the same credential. Not "wait and try tomorrow." Not "try through a different PSP." Never. In April 2024, Visa reclassified response code 14 (invalid account number) into Category 1, tightening the rules further.

Categories 2 through 4 cover declines that may be recoverable — insufficient funds, issuer system unavailable, transaction not permitted, and similar soft signals. For these categories, Visa allows up to 15 reattempts within a 30-day window per credential. Beyond that threshold, Visa charges excessive reattempt fees: $0.10 per domestic retry, $0.15 per cross-border retry, and $0.25 for cross-border retries in Latin America. These fees appear on your acquirer statement and compound quickly at scale.

Critically, Visa tracks retry counts separately for network tokens and PANs. A retry using a network token and a retry using the raw PAN are counted independently. This means switching from a token-based attempt to a PAN-based attempt (or vice versa) does not increment the same counter — a fact that has real implications for credential fallback strategies, which we will cover in the next section.

Stripe has built automatic protections around these rules — their system blocks payments that would be subject to network penalty fees, preventing merchants from accidentally exceeding thresholds. Other PSPs are building similar guardrails, but the sophistication varies widely.

Mastercard's Resubmission Logic

Mastercard takes a different but converging approach. Their system uses Merchant Advice Codes (MACs) — signals attached to decline responses that tell the merchant what action to take.

MAC 03 means "Do not try again." The account is closed, fraudulent, or otherwise permanently unrecoverable. Like Visa's Category 1, this is a hard stop — not a suggestion.

MAC 21 means "Payment canceled" — the cardholder or issuer has canceled the payment arrangement. Also a hard stop for that specific payment attempt.

For recoverable declines, Mastercard allows up to 10 reattempts within a 24-hour window and up to 35 reattempts within a 30-day window. The fee schedule has been escalating steadily: $0.10 per excessive retry when the program launched in 2022, rising to $0.15 in January 2023, $0.30 in January 2024, and $0.50 as of January 2025. Starting in January 2026, Mastercard expanded the scope — MAC fees now apply to all declined card-not-present transactions with MAC 03 or MAC 21, not just those exceeding retry thresholds.

The message from both networks is the same: retry with discipline, or pay for recklessness.

NetworkCategory / SignalLimitConsequenceWhat to Do
VisaCat 1 (never approve)0 — never retryExcessive reattempt fees ($0.10–$0.25)Hard stop. Ask for new method.
VisaCat 2–4 (recoverable)15 attempts / 30 daysFees above thresholdRespect budget; retry with backoff
MastercardMAC 03 / MAC 210 — never retryReattempt fees ($0.50 per, Jan 2025+)Hard stop. Ask for new method.
MastercardRecoverable decline10 / 24 hrs; 35 / 30 daysEscalating fees above thresholdCap total; wait 24 hrs between attempts
BothResponse Code 65 / auth requiredRetry with 3DSN/A — legitimate retryStep-up authentication, then resubmit

Table 2: Network reattempt rules comparison. Both Visa and Mastercard enforce hard stops on unrecoverable declines and fee penalties on excessive retries. The common thread: read the decline signal before deciding to retry.

Soft Declines and the "Retry with 3DS" Pattern

Not every decline is a dead end. Some are instructions — specifically, the issuer telling you that the transaction would be approved if the cardholder authenticated.

When Adyen returns AUTHENTICATION_REQUIRED as a decline reason, it is mapping the issuer's signal — Visa response code 1A, Mastercard response code 65, or Amex 130 — into a clear directive: the cardholder needs to complete 3D Secure authentication before the issuer will consider the transaction. This is a soft decline, and it is the highest-leverage smart retry in the entire orchestration toolkit.

The correct response is to step up to 3DS (as we covered in Chapter 11), have the cardholder authenticate, and resubmit the authorization with the authentication result attached. Mastercard's guidance for response code 65 is explicit: retry using EMV 3DS with challenge indicator 04, because no funds check was performed during the initial decline — the issuer simply wants proof that the cardholder is present.

This is fundamentally different from blasting unauthenticated authorization attempts through multiple PSPs hoping one gets through. The "retry with 3DS" pattern respects the issuer's intent, adds the information the issuer requires, and converts what would have been a lost sale into a completed transaction. MegaMart implemented this pattern for their German and French card transactions — markets where SCA requirements under PSD2 mean soft declines are common — and saw their European card approval rate improve by 6%.

Adyen takes this a step further by auto-retrying with 3DS when it detects a soft decline, unless the merchant has explicitly opted out. Other PSPs leave the step-up decision to the merchant's orchestration logic. Either way, the principle is the same: a soft decline is not a failure. It is the issuer telling you exactly what to do next.

Smart Fallbacks and Cascading Without Breaking Trust

Loading interactive…

"Fallback" sounds like a safety net — something passive that catches you when things go wrong. In orchestration, fallbacks are choreography. They are deliberate sequences of actions, triggered by specific failure signals, bounded by network rules and retry budgets, designed to convert failure into completion without creating new problems. Getting this choreography right is what separates orchestration systems that recover revenue from ones that amplify incidents.

Four types of fallback cover the territory.

Route Failover for Outages and Technical Errors

When PSP A stops responding — not a decline, but a genuine technical failure like timeouts, connection resets, or HTTP 5xx errors — the orchestration layer routes the transaction to an alternative provider. Orchestration vendors describe this as "automatic failover" or "retry on backup gateway," and it is the most straightforward fallback pattern.

Two requirements are non-negotiable. First, the failover must preserve idempotency. When MegaMart's PSP A went down for 12 minutes during a flash sale, their orchestration layer routed transactions to PSP B. Because the system carried the same idempotency key across the provider switch, no customer was double-charged — even in the edge case where PSP A had actually processed a charge but failed to return a response. Second, the failover must be bounded. Attempting three alternative routes with jittered timing is reasonable. Cascading through every available PSP with no delay is a recipe for turning one provider's outage into system-wide degradation. If your failover logic can generate more outbound traffic during an incident than during normal operations, your fallback is a liability.

Decline-Informed Fallbacks

A decline from the issuer is not a generic failure — it is a message with content. The previous section covered the specific network rules, but the practical implication for fallback design is straightforward: the decline response tells you what to do next.

Authentication required (Visa 1A, Mastercard 65, Adyen's AUTHENTICATION_REQUIRED): step up to 3DS and retry on the same PSP. This is not a route change — it is adding the information the issuer needs.

System unavailable or issuer inoperative: the issuer's systems are temporarily down. Retry later with capped attempts and exponential backoff. Routing to a different PSP will not help — the issuer is the same regardless of which acquirer asks.

Do not try again (Visa Category 1, Mastercard MAC 03): hard stop. No automated retries through any route. The fallback here is a product decision, not a technical one — surface an alternative payment method to the customer, or present a clear error message explaining the card cannot be used.

Some PSPs have built their own retry prevention into their systems. Stripe, for example, automatically blocks payments that would be subject to network penalty fees. If your orchestration layer routes around that block by sending the same transaction to a different PSP, you are not being clever — you are circumventing a compliance guardrail and accepting the fee risk yourself.

Credential and Token Fallbacks

This is the subtlest and potentially most valuable fallback pattern. The customer's card is the same. The stored credential is the same. But the representation of that credential can change — and that change can improve the outcome.

When a transaction using a network token is declined, the orchestration layer can retry using the raw PAN, or vice versa. Stripe's Adaptive Acceptance system makes this decision using machine learning: the model evaluates whether a token-to-PAN switch (or PAN-to-token switch) is likely to improve acceptance for that specific issuer, based on historical patterns. Across Stripe's network, this approach — combined with other Adaptive Acceptance optimizations — recovered $6 billion in falsely declined transactions in 2024.

The strategic importance goes beyond approval rates. As we noted in the network rules section, Visa tracks retry counts separately for tokens and PANs. A credential-form switch does not increment the same retry counter as a straight reattempt. This makes credential fallback a structurally different move from a payment reattempt — it changes the request shape without burning through your reattempt budget. For merchants with token infrastructure (covered in Chapter 12), this is a meaningful lever.

Method Fallbacks

When the decline signal is unambiguous — "do not try this card again" — the only remaining fallback is to the product and UX layer. Offer the customer an alternative payment method: a different card, a bank transfer, a wallet, BNPL, or a manual invoice path. This is where the orchestration layer's breadth matters. A system that manages multiple payment methods and routes (as described in the previous chapter) can present alternatives without sending the customer back to the beginning of checkout.

Strong tokenization and credential management — the kind of multi-method vaulting we covered in Chapters 12 and 25 — become strategic assets here. If your orchestration layer already has the customer's bank account details, wallet tokens, and card credentials stored securely, a method fallback is a one-click recovery. If it does not, the customer has to re-enter payment details from scratch, and your conversion rate drops with every keystroke.

Diagram 2: Smart fallback decision tree. Each decision node reads the failure signal and routes to the appropriate response — from hard stops to authentication step-ups to bounded retries. The tree enforces a deliberate sequence: network rules first, then technical recovery, then budget constraints.

Running Transaction Optimization Like a Product

Routing and retry logic do not get better on their own. Left unattended, they calcify — the rules you set six months ago keep running long after the conditions that justified them have changed. A PSP that was your weakest performer may have upgraded its acquiring connections. An issuer that used to approve network tokens at high rates may have changed its risk models. The Indonesian Mastercard declines that were 3x higher through PSP A than PSP B? MegaMart only discovered that after adding BIN-level observability — and a routing rule fix took their Indonesian Mastercard approval rates from 78% to 86%.

Observability That Supports Decisions

If you cannot answer these questions with data, your routing layer is running on assumptions instead of evidence.

What are your approval rates broken down by issuer country, BIN range, route, currency, authentication state, and decline category? Are your failures dominated by issuer declines (you cannot fix), authentication-required soft declines (you can fix with 3DS step-up), or technical errors (you can fix with failover)? Are you approaching scheme retry limits on any credential — the kind that trigger fees or cause your PSP to block further attempts?

These are not nice-to-have dashboards. They are operational necessities. Routing needs the same rigor you would apply to any reliability-critical service, because retry logic can become invisible load (hammering a struggling PSP with retries your monitoring does not surface) and invisible cost (racking up network penalty fees that only appear on your acquirer statement weeks later).

The observability stack for routing is not exotic. It is the same telemetry you would build for any distributed system: request/response logging with structured metadata (route chosen, decline code, retry count, latency), real-time error rate monitoring per route, and alerting on threshold breaches. The payments-specific addition is tracking decline codes and retry counts against network limits — something that requires mapping raw PSP response codes to Visa categories and Mastercard MACs, which is tedious but essential.

Experimentation Under Guardrails

Modern orchestration providers — Spreedly, Primer, Gr4vy, and the PSPs themselves — position routing as something that can be continuously improved. Spreedly reports a 7.9% retry success rate and 0.76% net authorization uplift through its Recover product. Primer claims "up to 20%" recovery rates and highlights Banxa recovering $7 million. Stripe's Adaptive Acceptance uses machine learning to adjust request parameters and credential selection, achieving a 60% year-over-year increase in retry success rates and a 35% reduction in retry attempts.

These numbers deserve scrutiny. "Recovery rate" (percentage of retried declines that ultimately succeed) is a fundamentally different metric from "authorization rate improvement" (percentage-point lift on overall approval). Vendor case studies tend to highlight best results from specific merchant cohorts, and most do not prominently disclose the compliance constraints around cascading retries across acquirers. Primer is a notable exception — their documentation warns about hard decline fees and 3DS re-authentication friction. APEXX Global takes the most conservative approach with a 4-attempt hard limit referencing Visa's 15-in-30 rule. IXOPAY parses Merchant Advice Codes and enforces a doNotResubmit flag.

The pragmatic architecture pattern layers five controls, each with a clear trigger and boundary:

LayerTriggerActionWhy
Default routeEvery first attemptRoute to primary PSP per transaction attributesPredictable, easy to reconcile
Health overrideRoute error rate > threshold (e.g., 5%)Switch to backup routePrevent revenue loss during outages
Decline-aware retrySoft decline or recoverable codeRetry with correction (3DS, data enrichment, credential switch)Respect issuer signal; improve approval
Hard stopCat 1 / MAC 03 / "never retry"Stop automated retries; surface to userAvoid fees; respect issuer intent
Budget capTotal attempts approach scheme limitBlock further retries for this credentialStay below penalty thresholds

Table 3: Five-layer routing architecture pattern. Each layer has a clear trigger and action. The layers stack: default routing handles the common case, overrides handle degradation, retries handle recoverable declines, hard stops enforce network rules, and budget caps prevent penalty fees.

The pattern is not exotic. It is boring, layered, and effective — which is exactly what you want from infrastructure that handles money.

What This Means for Each Reader

If you are new to payments, routing and retries are not tricks or vendor magic. They are the practical response to two facts about payment systems. First, payments are distributed systems — requests travel through multiple intermediaries, any of which can fail, and the failure modes are more varied than success/failure binary. Second, issuers decline transactions for reasons you can sometimes fix (incomplete authentication, missing data, wrong credential format) and sometimes cannot (stolen card, closed account, permanent block). The entire skill of routing and retry engineering is knowing which is which, and responding appropriately. Start with the decline codes. Learn what Visa's categories mean. Understand the difference between a soft decline and a hard stop. That foundation is worth more than any vendor's "smart routing" marketing.

If you are a merchant or operator, treat routing as a lever for reliability and conversion — but measure it honestly. The metrics that matter are authorization rate by route and market, average retry count per successful transaction, retry penalty fees on your acquirer statement, and customer-visible latency during fallback scenarios. The most effective "smart fallback" is usually the one the issuer is already signaling: authenticate the cardholder, correct the data quality, or ask for a different payment method. Resist the temptation to optimize purely for persistence — a system that retries aggressively may show a higher recovery rate while quietly accumulating penalty fees and degrading customer experience with slow checkouts.

If you are an architect, this is where payments engineering meets site reliability engineering. Idempotency, exponential backoff, retry budgets, and circuit breakers are as fundamental to payment orchestration as decline-code taxonomies and scheme reattempt rules. The moat is not in any single optimization — it is in building a system that can make real-time routing decisions while remaining compliant with network rules, stable under failure conditions, and explainable to your finance and compliance teams. The five-layer pattern in Table 3 is a starting point. The ongoing work is telemetry, feedback loops, and continuous adjustment — treating your routing layer as a product, not a configuration.

Chapter Closing

We have now seen how orchestration works at the transaction level — the routing decisions that happen in milliseconds, the retry logic that turns uncertainty into safety, the fallback choreography that converts failure into completion, and the network rules that constrain all of it.

In the next chapter, we meet a new kind of payer entirely: software. AI agents have started initiating payments on behalf of humans and businesses, a dozen protocols are fighting to carry those payments, and the routing, trust, and fallback machinery we just built turns out to be exactly what that fight is about. Then, in Chapter 34, we zoom out to the forces reshaping the global payments landscape — fragmentation at the surface, consolidation in the plumbing, vertical integration at the control points, and regulatory intervention at the fault lines.

Sources

  • Stripe — Adaptive Acceptance documentation; idempotent requests API documentation; network token retry logic; $6B recovered in 2024; 60% YoY retry success rate improvement
  • Visa — Core Rules and Visa Product and Service Rules: decline response code categories (1–4), reattempt limits (15/30 days), excessive reattempt fees ($0.10 domestic, $0.15 cross-border, $0.25 LATAM); April 2024 reclassification of response code 14; separate token/PAN retry counting
  • Mastercard — Transaction Processing Rules: Merchant Advice Code 03 ("Do not try again"), MAC 21 ("Payment canceled"); resubmission limits (10/24 hrs, 35/30 days); fee escalation schedule ($0.10 → $0.50); January 2026 MAC fee expansion
  • Adyen — AUTHENTICATION_REQUIRED soft decline documentation; mapping to Visa 1A, Mastercard 65, Amex 130; auto-retry with 3DS behavior
  • EMVCo — EMV 3-D Secure specification v2.3.1.1; frictionless vs challenge flow
  • PayPal / Braintree — PayPal-Request-Id idempotency header documentation; excessive retry fee warnings
  • IETF — Idempotency-Key request header specification (draft-07, October 2025, httpapi working group, Jena and Dalal)
  • Google — Site Reliability Engineering handbook, Chapter 22: "Addressing Cascading Failures" — exponential backoff, retry budgets
  • AWS — Amazon Builders' Library: "Timeouts, retries, and backoff with jitter"; Marc Brooker's Full Jitter analysis
  • Spreedly — Recover product: 7.9% retry success rate, 0.76% net authorization uplift
  • Primer — Fallback documentation: "up to 20%" recovery; Banxa $7M case study; UpliftAI "as much as 5%" authorization uplift; hard decline fee and 3DS friction disclosures
  • Gr4vy — Network retry rule warnings; 10-20% subscription recovery claims
  • APEXX Global — 4-attempt hard limit; Visa 15-in-30 rule reference
  • IXOPAY — MAC parsing; doNotResubmit flag enforcement
  • Doist/Todoist — 4% authorization rate improvement with network tokens + card account updater (Stripe case study)
The Money AtlasChapter 32 — Routing, Retries & Smart Fallbacks