Chapter 37 — If You're an Architect

Part IX: How to Use This Knowledge

Maya's team ships a hotfix at 2:17 AM. The pager went off 40 minutes ago: FinFlow's retry logic has gone haywire, and 1,247 duplicate authorizations have been created against a single PSP in the past hour. The root cause, once they find it, is almost embarrassingly simple. When the PSP returned a timeout, the system treated it as "failed" and retried — without an idempotency key. The PSP had actually processed the original request. Now there are two authorizations for each of those 1,247 transactions: one the customer intended, one the system created by accident.

The hotfix kills the retry loop. But Maya knows this is a band-aid on an architectural wound. The system was designed for the happy path — request goes out, response comes back, money moves. Nobody designed for the case where the response never comes back. Nobody designed for the case where it comes back late. Nobody designed for the case where the PSP processes the request but the network drops the acknowledgment.

"Payments architecture is where distributed systems meet regulated money movement," Maya tells her team at the 9 AM post-incident. "The happy path is only a fraction of our design space. The real job is building systems that remain correct and auditable under retries, timeouts, duplicates, partial failures, reversals, and late-arriving settlement data."

This chapter is Maya's playbook — and yours.

What "Good" Looks Like in Payments Architecture

After the retry storm, Maya pins four words to FinFlow's engineering wiki: reliable, correct, auditable, operable. Not because they are novel — every engineer has heard them before. But because FinFlow's codebase violated all four, and each violation had a price tag.

Reliable means assuming every dependency will fail — and designing for it before it does. The PSP will time out. The card network will return an ambiguous response. Your database will have a connection pool exhaustion event at 3 PM on the busiest shopping day of the year. Shopify's resilience guidance captures the principle: lower your timeouts aggressively, wrap external calls in circuit breakers, and design every interaction so that "dependency is down" is a handled case, not an exception. Maya's first architectural change at FinFlow: every PSP call gets a 5-second timeout with a circuit breaker that opens after three consecutive failures. The old default was 60 seconds — meaning one slow PSP response blocked a checkout thread for a full minute.

Financially correct means modeling the full payment lifecycle as an explicit state machine, not a boolean. This is the lesson that cost FinFlow $340,000. Their original system had a single column: paid = true or paid = false. When a retry storm created duplicate authorizations, the system could not distinguish "authorized but not yet captured" from "captured but not yet settled" from "settled and reconciled." Card payments have distinct phases — authorization, capture, settlement — each on different timelines, each with different failure modes. A 200 OK from the PSP means the API call succeeded. It does not mean money moved. Treating these as the same thing is how you get phantom payments that exist in your system but never settle, or duplicate captures that take real money twice.

Auditable means every financial movement in the system is recorded as a balanced debit and credit — double-entry bookkeeping as a design pattern, not just an accounting requirement. Uber's payments platform team articulated this clearly in their engineering goals: exactly-once processing, strong consistency, and auditability via double-entry. When Maya audited FinFlow's existing system, the only "audit trail" was a JSON blob of raw PSP webhook payloads stored in an append-only log. No internal ledger. No balanced entries. No way to answer "where is this $47?" without manually tracing through three systems and a Stripe dashboard.

Operable under stress means the system is observable, recoverable, and practiced for failure. PCI DSS requires logging and monitoring as a continuous constraint — not a checkbox you satisfy during an annual assessment. But operability goes beyond compliance. It means your team can answer "what happened to this payment?" in minutes, not days. It means you have rehearsed the scenario where your primary PSP goes down at peak traffic. It means your reconciliation runs continuously, not as a monthly scramble that discovers three-month-old discrepancies.

Table 1: The four properties of a well-designed payments stack. Maya uses this as FinFlow's architectural north star — every design review asks which property a proposed change strengthens or weakens.

These four properties are not a wish list. They are a filter. Every architectural decision Maya's team makes at FinFlow — which database to use for the ledger, whether to build or buy the orchestration layer, how to structure the token vault — gets evaluated against them. The property that matters most? Correctness. You can recover from a slow system. You cannot easily recover from a system that took money it should not have, or lost track of money it did.

A Reference Architecture You Can Adapt

Maya's first week at FinFlow, she asks a simple question: "Where does our payment logic live?" The answer is everywhere. Checkout code makes direct PSP API calls. Fraud rules are embedded in the payment controller. The order service checks payment status by querying the PSP's API in real time. Token storage is split across two services with no clear owner. And the "ledger" is Stripe's dashboard.

This is not unusual. Most payment systems grow organically — you add a PSP integration, bolt on fraud checking, wire up settlement reporting — and before you know it, payment logic is scattered across a dozen services with no clear boundaries. Maya's job is to replace that with layers.

The principle: think in layers, not vendors. Vendors — PSPs, gateways, acquirers, fraud tools — will change. You will add a second PSP for redundancy, switch fraud providers when your current one's model drifts, or expand into a market that requires a local acquirer. Layering lets you swap components without rewriting business logic. Each layer owns a specific concern and exposes a clean interface to the layers above and below it.

The Six Layers

Layer 1: Payment Experience. This is the checkout — the UI where customers select a payment method, enter credentials, and authenticate. It owns the customer-facing interaction, including 3DS authentication flows (as we covered in Part III, EMVCo positions 3D Secure for both browser and app integration with step-up authentication options). The experience layer should know nothing about which PSP processes the transaction or how routing works. It submits a payment intent and receives a result.

Layer 2: Payments API and Orchestrator. This is your control plane. It receives payment intents from the experience layer, decides which processor handles the transaction, manages idempotency keys, normalizes status responses from different PSPs into a canonical format, and orchestrates retries and fallbacks. As we covered in Part VIII, the orchestrator prevents PSP-specific logic from leaking into your checkout or order management. Maya's first major project at FinFlow: building this layer, because without it, adding a second PSP means duplicating business logic everywhere.

Layer 3: Risk and Policy. This layer owns risk signals, business rules, and decisioning boundaries. "Risk" is not just fraud scoring — it includes detecting abnormal retry patterns, flagging settlement mismatches, identifying operational anomalies at scale, and enforcing business policies like velocity limits or geographic restrictions. Separating risk from the orchestrator means you can update fraud rules without touching payment routing, and vice versa.

Layer 4: Token and Secrets. This is the data minimization boundary — the layer that handles PANs and issues tokenized surrogates. PCI SSC's tokenization guidance is clear: centralizing PAN storage and minimizing PAN occurrences shrinks the Cardholder Data Environment. But the tokenization system itself remains in CDE scope — reducing blast radius does not eliminate responsibility. As we covered in Part III's tokenization chapter and in Part VI, this layer is where network tokens, vault tokens, and PCI scope boundaries intersect. Maya discovers FinFlow's PANs are stored in two different databases with no shared token mapping — her second major project.

Layer 5: Ledger and Money Movement. This is the system of record for financial obligations and transfers — what you charged, what you owe, what you paid out. The ledger must be vendor-agnostic and auditable. Uber's payments platform team built their ledger with explicit goals: strong consistency, idempotent writes, and double-entry auditability. The key insight: your PSP is an input to the ledger, not the ledger itself. When Maya told FinFlow's team "the Stripe dashboard is not your ledger," it was the architectural turning point — the moment they started treating financial truth as something they own, not something they outsource.

Layer 6: Settlement, Reconciliation, and Accounting. This layer ingests settlement files from PSPs, calculates fees, processes chargebacks and refunds, and matches everything against bank statements. Uber's settlement accounting team documented the real-world messiness: PSP files contain duplicates, missing references, and different delivery formats. This layer exists to absorb that messiness and produce clean, reconciled financial data. As we covered in Chapter 4, the canonical payment flow ends with reconciliation — this is where the architectural rubber meets the road.

Diagram 1: Six-layer payment reference architecture. The orchestrator mediates between the customer-facing experience and the external payment network. The ledger is the system of financial truth — vendors are inputs, not truth. Settlement reconciles the ledger against bank statements.

Table 2: Reference architecture layers. The top three layers change frequently as you optimize UX, routing, and risk rules. The bottom three change rarely — they are the stable foundation. PCI scope concentrates in the token layer by design.

The pragmatic scaling rule: keep the ledger and the tokenization boundary as the most carefully controlled parts of your system. Let the higher-churn layers — checkout UX, routing strategies, fraud rules, analytics — evolve faster. That separation prevents "changing the money truth" every time you optimize the funnel.

Correctness Patterns That Prevent "Money Bugs"

Architects in payments win by assuming two things: every external interaction can be duplicated, and every long-running workflow will be partially completed. The five patterns below are your defense against the class of bugs that show up not as error messages but as wrong numbers in bank accounts.

Model Payments as Explicit State Machines

Maya's $340,000 lesson started with a single database column: paid BOOLEAN. When the retry storm hit, the system created duplicate authorizations — but it had no way to distinguish "authorized but not captured" from "captured but not settled" from "settled and reconciled." Everything was either true or false, and true meant "something happened, probably."

Card payments have distinct phases, as we covered in Chapters 4 and 9: authorization (the issuer approves a hold on funds), capture (the merchant claims the held funds), and settlement (money actually moves between banks). Each phase operates on a different timeline. An authorization might last minutes or days before capture. Settlement happens in batches, often 24-48 hours later. Voids, refunds, and chargebacks create additional side paths, each with their own lifecycle.

The fix is an explicit state machine with enumerated states: created, authorized, captured, settled, plus side paths for voided, refunded, chargeback opened, chargeback won, and chargeback lost. Every transition between states has rules: you can only capture an authorized payment, you can only refund a captured or settled payment, you can only void an authorized-but-not-captured payment. The state machine makes impossible transitions visible — and impossible — at the code level.

Maya's rebuild: FinFlow's payment table now has a state column with an enum, a transitions table logging every state change with a timestamp and actor, and a constraint that prevents any write that violates the transition graph. The $340K duplicate capture? It cannot happen in the new system because the state machine rejects a second capture attempt on an already-captured payment.

Design for At-Least-Once Delivery with Idempotent Handlers

Duplicates are not edge cases in payments — they are the normal operating condition. Webhooks retry when acknowledgments time out. Clients retry when responses are slow. Message brokers redeliver after consumer crashes. Replay mechanisms re-process events after outages. Your system will see the same event more than once. The question is whether seeing it twice produces the right result.

Stripe's API documentation makes this explicit: POST operations should include an idempotency key so that retries do not create duplicate charges. Adyen takes the same approach with API-level idempotency to prevent duplicate processing. But the discipline goes deeper than a single API call. True idempotency is end-to-end: a stable internal payment_id serves as the anchor, and every side-effecting action (authorize, capture, refund, settle) gets its own dedupe key stored alongside the result.

Maya learned this the hard way when FinFlow's settlement ingestion pipeline processed the same PSP settlement file twice after a deployment restart. The pipeline was idempotent at the API call level but not at the file-processing level. The fix: every settlement event gets a composite dedupe key (PSP + file ID + line number), and the "apply event" handler checks for existence before writing.

Bounded Retries with Exponential Backoff and Jitter

Naive retries are how FinFlow created 1,247 duplicate authorizations. But even with idempotency keys, unbounded retries are dangerous because they amplify load on failing systems.

AWS's architecture guidance recommends exponential backoff with a cap: wait 100ms, then 200ms, then 400ms, up to a maximum. Google Cloud adds the critical ingredient: jitter — randomizing the wait time so that thousands of clients are not all retrying at the same instant. Pair retries with idempotency keys (so the retry is safe if the original succeeded) and circuit breakers (so you stop retrying when the dependency is clearly unhealthy, not just slow).

Maya's new retry policy at FinFlow: maximum three attempts per PSP call, exponential backoff starting at 200ms with full jitter, circuit breaker that opens after five consecutive failures and stays open for 30 seconds before a single probe request tests recovery. Simple, bounded, and combined with the idempotency layer so that every retry is safe regardless of whether the original request succeeded.

Prefer Sagas to Distributed Transactions

A single payment often touches multiple services: the order service, the inventory service, the fulfillment service, the ledger, and the notification system. The temptation is to wrap everything in a distributed transaction — if any step fails, roll back all of them. In practice, distributed transactions (two-phase commit) are fragile, slow, and poorly supported across service boundaries.

The alternative is the Saga pattern, as described in Microsoft's architecture guidance: a sequence of local transactions, each with a compensating action that undoes its effect if a later step fails. In payments, compensating actions are real financial operations: a void reverses an authorization, a refund reverses a capture, a reversal adjusts a ledger entry. Each compensation must be modeled explicitly and must itself be safe to retry — because the compensation might also time out and need to be retried.

Maya's FinFlow implementation: when a payment is captured, the saga proceeds through inventory reservation, fulfillment trigger, and ledger posting. If the fulfillment trigger fails, the saga executes compensations in reverse: release inventory, void the capture (or queue a refund if the capture window has passed), and post a reversal to the ledger. Each step writes its outcome to the saga state before proceeding to the next, so recovery after a crash can resume from where it left off.

Transactional Outbox for Reliable Event Publication

The most insidious bug in event-driven payment systems is the split-brain: the database transaction commits (the payment is captured), but the message to the fulfillment service fails to publish (the order is never shipped). Or worse: the message publishes but the database transaction rolls back (the order ships but the payment was never captured).

The transactional outbox pattern, documented in Microsoft's architecture guidance, solves this by writing the business data and the event into the same database transaction. A separate relay process reads the outbox table and publishes events to the message broker. If the relay fails, it retries from where it left off — the outbox is the durable record.

In payments, this pattern is critical because downstream actions (shipping, digital delivery, access provisioning) must not be triggered by phantom messages that correspond to rolled-back transactions, and must not be missed because the message broker was temporarily unavailable.

Diagram 2: Payment state machine. Each state is explicit and each transition has rules. The side paths — void, refund, chargeback — are first-class states, not afterthoughts. Maya's $340K duplicate capture is impossible in this model because the state machine rejects a second capture on an already-captured payment.

Table 3: Correctness patterns at a glance. These five patterns form FinFlow's architectural foundation. Each one addresses a specific failure mode that is common, costly, and preventable by design.

Designing for Scale and Multi-Region Reality

Scaling a payment system is not just "handle more transactions per second." It is scaling correctness, risk detection, and recovery under failure modes that multiply with every new region and provider.

Reliability Targets That Map to Customer Impact

Google's SRE framework defines availability as yield — the fraction of well-formed requests that succeed. For payments, Maya translates this into a target her team can reason about: 99.95% of payment attempts receive a definitive response (success or decline) within 5 seconds. That target means approximately 25 minutes of undefined-state payments per month. Not 25 minutes of downtime — 25 minutes of payments where the customer sees a spinner and has no idea whether they paid.

The SLO becomes actionable through an error budget: the allowed amount of unreliability per period. When the budget is healthy, the team ships features and migrations. When the budget is nearly consumed, the team freezes changes and focuses on reliability. This is not a management ritual — it is an objective mechanism that prevents the common pattern of launching a risky PSP migration during a month when your system is already fragile.

Multi-Region Trade-offs

AWS's Well-Architected framework describes the spectrum of multi-region architectures: pilot light (minimal standby infrastructure, activated on failure), warm standby (scaled-down secondary that can scale up), and active-active (full capacity in multiple regions simultaneously). Each increases availability and decreases failover time — at the cost of complexity and dollars.

For payments, the trade-off triangle is latency versus availability versus consistency. Active-active gives you the lowest latency for global customers and the highest availability, but it introduces conflict resolution challenges that are existential in financial systems. If two regions process the same payment simultaneously, you have a double charge. If a refund processes in one region while a settlement posts in another, your ledger diverges. FinFlow's decision: active-active for the experience and orchestrator layers (latency-sensitive, stateless, safe to duplicate), with the ledger as the consistency boundary (single-writer, strongly consistent, replicated for reads).

Three Payments-Specific Multi-Region Pitfalls

Generic multi-region guidance gets you 80% of the way. The last 20% is payments-specific, and it is where the expensive lessons live.

Pitfall 1: Global idempotency is harder than local idempotency. FinFlow learned this during a regional outage that created 847 duplicate charges. Their idempotency store was a Redis cluster — regional, not global. When Region A went down and traffic failed over to Region B, the new region had no record of Region A's in-flight requests. Retries created duplicates because the dedupe check passed in a store that had never seen the original request. The mitigation options: a global idempotency store (adds latency), deterministic ID generation that allows the ledger to deduplicate at write time (adds complexity), or accepting that failover will create some duplicates and building automated reconciliation to detect and reverse them within minutes (adds operational overhead). Uber's payments platform chose the first path — global, exactly-once idempotency as a core design goal. FinFlow, with less traffic and engineering capacity, chose the third: fast automated reconciliation with alerting.

Pitfall 2: Settlement and reconciliation scale "sideways." Adding a new market does not just increase transaction volume — it adds a new PSP, a new settlement file format, a new fee model, a new set of exception workflows, and possibly a new currency. Uber's settlement accounting team documented this: at scale, you are not processing one type of settlement file — you are processing dozens, each with their own quirks (duplicates, missing references, different delivery mechanisms). FinFlow hit this when they added their third PSP: suddenly they had three incompatible settlement report formats with different reference ID schemes, and their single-format reconciliation script broke. The mitigation: dedicated ingestion, normalization, and reconciliation services per PSP, feeding into a single internal settlement model.

Pitfall 3: Data residency and regulatory overlays. PCI DSS is a global minimum set of data protection requirements — but local laws add constraints. The EU's GDPR imposes data residency and right-to-deletion requirements that interact with PCI's data retention mandates. Southeast Asian markets have their own data localization rules. An architecture that assumes a single universal data boundary breaks at scale. FinFlow discovered this when EU data residency requirements conflicted with their centralized token vault in Singapore — they had to architect regional data boundaries that they had not planned for, at significant cost.

Table 4: Multi-region pitfalls for payments. Generic distributed systems guidance does not cover these — each is specific to the intersection of financial correctness and global operations.

The pragmatic scaling rule, restated: keep the ledger and tokenization boundary as the most carefully controlled parts. Let everything else evolve faster. That separation ensures you are not "changing the money truth" every time you optimize the checkout funnel or add a new routing strategy.

Migration Playbook for Payment Stacks

Maya's worst six months at her previous company were spent on a PSP migration that was supposed to take three months. The original estimate assumed "just point the API calls at the new provider." The reality: stored card tokens could not be ported, transaction IDs had no cross-reference mapping, settlement files used different formats and reference schemes, and 47,000 active subscriptions had to be migrated without interrupting billing cycles. Eleven months later, they finished — mostly.

Payment migrations are risky because "just send the traffic elsewhere" is never the whole story. You must preserve identity (customer tokens, subscription references), ensure idempotency (no duplicate charges during cutover), and reconcile financial truth across providers during a period where both are active.

The Strangler Fig Approach

The mental model that saved Maya: the Strangler Fig pattern, described by Martin Fowler and formalized in Microsoft's architecture guidance. Rather than a big-bang cutover, you introduce a facade or proxy that routes traffic. Initially, 100% goes to the legacy system. You gradually shift traffic to the new system — by payment method, by region, by customer cohort, by traffic percentage — until the legacy system handles nothing and can be decommissioned.

In payments, the facade is your orchestration layer. If you built one (see Part VIII), you already have the routing infrastructure to execute a Strangler Fig migration. If you did not, you will need to build at least a minimal routing proxy before you can safely migrate.

Three Migration Types

Processor swap: PSP A to PSP B. The most common migration, triggered by cost, features, or compliance requirements. The hard parts are subscription continuity (tokens issued by PSP A usually cannot be used with PSP B — you need a token migration or re-enrollment), transaction ID mapping (your internal references must map to the new provider's references for dispute handling and reconciliation), webhook and event handling (different providers send different event types in different formats at different times), and settlement reconciliation during the overlap period when both providers are active. Uber's settlement accounting team documented these challenges at scale: PSP files contain duplicates, missing references, and different delivery mechanisms.

Add redundancy: PSP A plus PSP B with routing and fallback. This is where the orchestration layer pays off — as we covered in Part VIII, multi-PSP architecture exists precisely for this scenario. But fallback must be selective. Retrying a real issuer decline across a different provider creates confused customers who see two charges (one from each provider). Retrying after timeouts or HTTP 5xx errors with end-to-end idempotency is appropriate — the first PSP may not have processed the request, and the idempotency key prevents duplicates if it did. FinFlow learned this the hard way: their first "fallback" implementation retried every decline across providers, resulting in customers receiving two charge notifications and a flood of support tickets.

Break out the ledger from the processor. This is the strategic migration — moving from "the PSP is our system of record" to "the PSP is one input to our system of record." It is the most complex migration because it requires building a vendor-agnostic ledger while the existing system continues to run, then gradually shifting financial truth from the PSP dashboard to the internal ledger. Uber's platform team articulated the endgame clearly: exactly-once processing, strong consistency, double-entry auditability — all properties that require owning your own ledger rather than relying on a vendor's. Maya's realization: "The PSP dashboard is not your ledger" was the moment FinFlow committed to this migration.

Five Steps for Any Payment Migration

Maya distills her experience into a sequence that applies to all three migration types.

Step 1: Define invariants and acceptance tests. Before you write a line of migration code, write the tests that prove the migration is correct. One checkout produces at most one captured payment. Every capture exists in the ledger. Every settlement line maps to an internal payment record. Differences are surfaced as reconciliation exceptions, not silently absorbed. These invariants are your rollback safety net — if any invariant is violated, you stop the migration and investigate.

Step 2: Build adapters, not forks. Maintain a canonical internal schema for payments, and translate provider-specific fields at the edges. When you add PSP B, you write an adapter for PSP B — you do not fork your payment logic. This reduces rework when you eventually add PSP C or replace PSP A entirely.

Step 3: Run in parallel where it matters. Shadow reads (fetch status from both providers and compare), or dual writes with a single settlement truth. Use the saga and outbox patterns from earlier in this chapter for predictable behavior under partial failures. Maya runs shadow mode for two weeks before cutting any live traffic to the new provider — comparing authorization responses, settlement data, and webhook timing between old and new.

Step 4: Cut over by cohorts, not by date. Cohorts can be regions, payment methods, customer segments, or traffic percentages. The Strangler Fig facade makes this feasible without proliferating feature flags. FinFlow migrates their Singapore Visa transactions first (small volume, well-understood issuer landscape), then Malaysia, then the full APAC portfolio, then EU — each cohort validated before the next begins.

Step 5: Reconcile continuously during migration. Assume settlement formats and reference IDs differ between providers. Compare at the ledger level (do both providers agree on which transactions were captured?) and at the bank statement level (does the money that arrived match what both providers reported?). Continuous reconciliation during migration catches discrepancies in hours, not months.

Diagram 3: Strangler Fig migration pattern. The routing facade gradually shifts traffic from legacy to new PSP. Both feed into a single ledger. Settlement and reconciliation run continuously against both providers and bank statements, catching discrepancies during the migration rather than after.

Security, Compliance, and Data Governance by Design

Security architecture in payments is largely scope architecture — deciding what touches sensitive data, where that data lives, and how small you can make the blast radius when something goes wrong.

PCI DSS v4.0.1 provides the baseline: 12 principal requirements spanning network security, data protection, vulnerability management, access control, logging and monitoring, testing, and governance. As we covered in Chapter 24, these requirements are not a checklist you satisfy once a year — they define your system boundaries and shape your Cardholder Data Environment. Every architectural decision that affects where cardholder data flows also affects your PCI scope.

Four design principles translate PCI's requirements into architectural decisions.

Minimize what you store and where. The PCI Security Standards Council's tokenization guidance states the core principle directly: the key for merchants wishing to reduce PCI DSS scope is to not store, process, or transmit cardholder data. If business, legal, or regulatory requirements demand retention, store the minimum necessary in the minimum number of locations. Maya's first security audit at FinFlow found PANs in two databases, three log files, and a customer support tool's local cache. Her remediation: route all PAN handling through the token vault layer, eliminate every other PAN touchpoint, and instrument alerts for any PAN pattern detected outside the vault boundary.

Tokenization reduces blast radius, not responsibility. Tokenization centralizes PAN storage and reduces the number of systems that handle sensitive data — shrinking the CDE and simplifying compliance. But all elements of the tokenization system itself — including de-tokenization services and PAN storage — remain in scope for PCI DSS. Architects who assume "we use tokens, so we're out of scope" are building on a misconception that auditors will correct, expensively. As we covered in Part III's tokenization chapter and in Part VI, the token vault is inside the scope boundary, and the interfaces between the vault and the rest of the system must be secured accordingly.

Segmentation is useful but not a silver bullet. PCI SSC's scoping and segmentation guidance warns that attackers frequently compromise systems designated as "out of scope" and then pivot into the CDE. Network segmentation reduces the attack surface, but it does not replace defense in depth. Maya's approach: treat segmentation as one layer of a defense-in-depth strategy, not as a moat that makes everything inside safe.

Be explicit about authentication flows as part of the payment state machine. EMV 3DS outcomes influence not just the customer experience but also downstream authorization behavior and liability allocation. A successful 3DS authentication shifts fraud chargeback liability to the issuer — which changes the risk calculus for authorization. Treating 3DS as a front-end concern disconnected from the payment state machine means your system cannot reason about liability state, cannot make informed retry decisions on authentication failures, and cannot properly represent the payment's security posture in dispute evidence. Build 3DS into the state machine: "authentication requested," "authentication completed," "authentication failed" are states that affect what happens next.

Design logging, monitoring, and audit trails as data products, not compliance artifacts. PCI DSS requires logging and monitoring of access to system components and cardholder data. Architecturally, this means your incident responders and auditors need consistent evidence trails across services — not scattered log files in different formats with different retention policies. Maya builds FinFlow's audit infrastructure as a first-class data pipeline: structured events, centralized storage, standardized retention, and query interfaces that let anyone answer "what happened to this payment?" within minutes.

Operability, Testing, and Incident Readiness

Payment systems do not "stay correct" automatically. They stay correct because you make them observable and you practice failure.

Build operational guardrails into the design. Timeouts and circuit breakers are not operational afterthoughts — they are architectural choices that influence queue sizes, retry behavior, and user-visible failure modes. Shopify's resilience guidance frames this well: if your system does not have explicit timeouts, your system's timeout is "whatever the slowest dependency decides." Maya sets FinFlow's defaults: 5-second timeout on all PSP calls, circuit breaker with a 30-second cooldown, and queue depth limits that shed load rather than accumulate latency.

Turn reliability targets into actionable policies. The SLO and error budget framework from Google's SRE workbook gives you an objective mechanism for making trade-offs. When your error budget is healthy, ship features and run migrations. When it is nearly consumed, freeze changes and focus on stability. This prevents the pattern Maya saw at her previous company: launching a risky PSP migration during a month when the system was already fragile, turning a recoverable situation into a crisis.

Practice resilience. Game days where you simulate PSP outages, region failures, and settlement file corruption are not optional extras — they are how your team builds the muscle memory to respond without panic. Rehearsed runbooks mean the on-call engineer at 2 AM knows the first three steps before they are fully awake. AWS's reliability guidance emphasizes that failover is operational work, not magic: if you have not practiced it, it will not work when you need it.

Design reconciliation as always-on. Stable identifiers and durable event histories mean your operations team can answer "what happened to this payment?" at any time, not just during monthly close. As we discussed in Chapter 36, the operator who discovers a discrepancy three months late has a very different problem than the operator who discovers it three hours late. Treat reconciliation as a continuous monitoring capability — an always-on health check for your financial data — not as a periodic accounting exercise.

The chapter's through-line comes down to a single principle: treat payments correctness as a product you operate. Your best architecture work does not show up as flashy features or impressive throughput numbers. It shows up as fewer manual interventions, faster incident recovery, and cleaner audits — because the system was built to be explainable under failure, not just fast when everything works.

The architecture patterns in this chapter will evolve. New payment rails will emerge. Regulations will shift. Failure modes will surprise you. And here's the thing: the judgment you've built by reading this far — knowing where the failure modes hide, what a statement actually says, which vendor claims are real — is scarce enough that people will pay for it. In the next chapter, we look at exactly that: how to turn payments knowledge into a practice of your own.

Sources

• Shopify — Resilience engineering guidance (lowering timeouts, circuit breakers, graceful degradation)
• Uber Engineering — Payments platform design goals (exactly-once processing, strong consistency, double-entry auditability); settlement accounting architecture (PSP file ingestion, duplicates, missing references, format normalization)
• Google — Site Reliability Engineering (availability as yield, SLO/error budget framework, Chapter 22 on cascading failures); SRE Workbook (error budget policies)
• AWS — Well-Architected Framework reliability pillar (multi-region disaster recovery strategies: pilot light, warm standby, active/active); exponential backoff guidance; Amazon Builders' Library (timeouts, retries, and backoff with jitter)
• Google Cloud — Exponential backoff and jitter guidance for idempotent requests
• PCI Security Standards Council — PCI DSS v4.0.1 (12 principal requirements); tokenization guidance (scope reduction, CDE implications); scoping and segmentation guidance (pivot risk from "out of scope" systems); third-party security assurance
• EMVCo — EMV 3-D Secure specification (browser and app integration, step-up authentication); EMV Payment Tokenisation specification (PAN replacement with usage constraints)
• Visa — Visa Token Service (token provisioning, lifecycle management, authorization uplift)
• Stripe — Idempotent requests documentation (POST operations, idempotency keys, 24-hour key expiry)
• Adyen — API idempotency documentation (safe retries, duplicate charge prevention)
• Martin Fowler — Strangler Fig Application pattern
• Microsoft — Saga pattern (local transactions with compensating actions); Transactional Outbox pattern (reliable event publication); Strangler Fig pattern guidance