01
Multi-Vendor Payment System and Arrears Recovery
Owned the service from initial design to implementation
Led the end-to-end architecture and implementation of VoltUp's payment platform, evolving a single-provider system into a multi-provider module featuring DLQ-based recovery and atomic database approval locks to eliminate duplicate charges. Connected stranded READY and AUTHORIZED states to automated recovery and operator-review paths so indeterminate outcomes remain traceable.
Design Context
Beyond supporting multiple providers through one extension point, the service needed a final safety boundary that prevents reapproval across Redis-lock expiry, lost payment-gateway responses, database rollbacks, and provider changes, with clear recovery ownership.
Key Point
A project that demonstrates multi-provider extensibility, payment state machines, atomic approval ownership, and recovery boundaries for indeterminate outcomes.
Core Implementation
- Built a payment-gateway integration architecture using an abstract-class-based provider strategy.
- Implemented a GCP Pub/Sub dead-letter-queue pattern that isolates failed events and keeps failed-payment recovery traceable.
- Added database-backed approval ownership that conditionally transitions `payments.status` to `AUTHORIZED` immediately before the payment-gateway call, letting the database atomically claim the single approver for an order.
- Separated unpaid-balance bookkeeping from payment confirmation so ledger-lock contention cannot roll back an approval, while correcting duplicate point deductions and preventing current settings from being applied retroactively during retries.
- Added state sync/event republishing for READY orphans and resolution batches/manual repair APIs for AUTHORIZED holds, separating automatic recovery from operator review.
Engineering Lens
- Centered request orchestration, point hold/confirm, and success/failure transitions around PaymentProcessor so state changes remain traceable in one place.
- Kept Redis locks as a first-line concurrency control, while making a conditional DB update the final serialization boundary immediately before money moves.
- Used a fail-closed model: transition to PAID/FAILED only for definitive outcomes and retain AUTHORIZED when uncertain, prioritizing duplicate-charge prevention over automatic availability.
- Verified with a real-MySQL 8-thread concurrency test that exactly one contender wins approval ownership, locking the conditional UPDATE serialization guarantee into a regression test rather than relying on mocks.
Architecture Snapshot
Mermaid View
Multi-vendor orchestration: lock, hold, and state transitions
Shows request data, lock keys, point holds, READY-state creation, provider approval, and success or failure transitions in one integrated diagram.
flowchart TD
Req["pay / rePay<br/>order=ORD-240915-001<br/>user=421 method=17 point=2000"] --> Lock["distributed lock<br/>payment-user-process:421"]
Lock --> Hold["PointUpdater.hold()<br/>wallet -> HOLD 2000P"]
Hold --> Ready["createWithReady()<br/>payment READY"]
Ready --> Sub["resolve subscription<br/>methodId=17 or primary"]
Sub --> Vendor["Payment Provider Router<br/>Kakao Pay / Toss Payments / Kakao T<br/>selected: Kakao T"]
subgraph PGV["Payment Provider Layer"]
direction TD
Vendor --> Keys["read vendor keys<br/>pgPayKey + token"]
Keys --> Api["vendor client.pay(...)"]
Api --> Tx["save pgTransactionId<br/>paymentId / tid / paymentKey"]
end
Tx --> Result{"approval result"}
Result -->|success| Done["updateSuccess<br/>payment PAID<br/>point HOLD->CONFIRM"]
Result -->|fail| Fail["updateFailed<br/>releaseHold(order)"]
Fail --> Recovery["repair / retry / failover"]
classDef vendor fill:#dff2ff,stroke:#0f4c81,stroke-width:2px,color:#0f172a;
class Vendor,Keys,Api,Tx vendor;
style PGV fill:#eef7fb,stroke:#0f4c81,stroke-width:2px,color:#0f172a; Mermaid View
Multi-vendor system: mandatory contracts and optional extensions
Places `VendorChecker.select()` on top of the `VendorType` extension point, separates contracts required for every vendor from features needed by only some vendors, and uses `@RequiredVendor` plus `VendorRequirementsValidator` to catch missing mandatory implementations at startup.
flowchart TD
Vendors["VendorType<br/>Kakao Pay / Toss Payments / Kakao T"] --> Select["VendorChecker.select(vendorType)"]
Select --> Required["Required on all vendors<br/>VendorPaymentProcessor<br/>VendorMethodProcessor"]
Select --> Partial["Required on some vendors<br/>VendorPaymentOnceProcessor<br/>(KakaoPay only)"]
Select --> Optional["Optional extensions<br/>RepairService / vendor hooks"]
Required --> Validate["@RequiredVendor<br/>+ VendorRequirementsValidator"]
Partial --> Validate
Validate --> Boot{"startup validation"}
Boot -->|missing| Error["application start fail"]
Boot -->|ok| Route["route to concrete impl"]
classDef core fill:#dff2ff,stroke:#0f4c81,stroke-width:2px,color:#0f172a;
classDef optional fill:#edf9f3,stroke:#2f6f57,stroke-width:2px,color:#0f172a;
classDef error fill:#fff1f2,stroke:#be123c,stroke-width:2px,color:#0f172a;
class Vendors,Select,Required,Partial,Validate,Route core;
class Optional optional;
class Error error; Operational Outcomes
- Built a payment architecture that integrates multiple payment providers behind one interface and remains extensible as providers are added.
- Applied dead-letter-queue handling and state-specific recovery paths for READY and AUTHORIZED payments, making each failure either automatically recoverable or explicitly reviewable by an operator.
- Blocked paths where post-approval database contention, lost responses, or retries through a different payment provider could lead to duplicate charges.