Skip to main content

Phase 3: Multi-EVM Rollout and Keeper Redundancy

· 5 min read
OrcaRail
OrcaRail
Crypto payment rails for web2 apps

April 20, 2026 — Phase 3 takes the contract surface from Phase 1 and Phase 2 and puts it on every EVM chain we support. It also adds a second keeper, so a NestJS outage cannot skip a subscription charge.

TL;DR​

  • Deploy PaymentLinkFactory, PaymentLinkReceiver implementation, FeeSplitter, and SubscriptionHub to Ethereum mainnet, Arbitrum, Optimism, and Polygon.
  • Same CREATE2 salt scheme everywhere — the factory address is the same on every chain (clean for docs, clean for integrations, clean for computeAddress from the client).
  • Register Chainlink Automation (preferred) and/or Gelato Web3 Functions as a second keeper calling the same charge(ids[]).
  • Publish SLOs: 99.5% of charges within 10 minutes of due time, measured.

What this phase changes​

Multiple keepers calling the same public charge() is the liveness story. None of them has custody; any of them can save a due charge.

Deployment plan​

Deploy order (one chain per week, canary per chain):

  1. Ethereum mainnet (chainId 1)
  2. Arbitrum One (42161)
  3. Optimism (10)
  4. Polygon PoS (137)

Addresses on each chain are published in /docs/non-custodial/contract-reference. The deployer key is the same 3-of-5 Safe multisig defined in Phase 0, redeployed per chain from the same ceremony.

All four chains share the same useOnChainReceiver feature flag scoped by chainId inside api/src/payments/wallet.service.ts. Merchants opt in chain-by-chain; defaults flip to on once the canary week is clean.

Gas strategy for charge()​

The keeper's charge() transactions need gas. We reuse the existing paymaster plumbing in api/src/withdrawals/services/gas-strategy:

  • Alchemy paymaster enabled (Base, Arbitrum, Optimism, Polygon): keeper calls SubscriptionHub.charge() as a UserOperation with policy sponsorship. Zero gas cost for the keeper EOA.
  • Ethereum mainnet (no paymaster): keeper self-funds from its hot-gas balance. Funded daily by the platform wallet; monitored with a Datadog alert for low balance.
  • Merchant-subsidized gas (optional): merchants can set a gas_refund_address and the keeper pulls reimbursement from there, matching the existing support/commission pattern. Out of scope for initial rollout.

The second keeper is a redundancy play, not a replacement. It runs alongside the NestJS scheduler.

  • Register an Upkeep per SubscriptionHub.
  • checkUpkeep() returns performData = abi.encode(ids[]) for all id with block.timestamp >= nextChargeAt (read from a view function).
  • performUpkeep(performData) calls charge(ids) in a batch.
  • Funded with LINK from the platform treasury; balance monitored.

Gelato Web3 Functions (alternative)​

  • A JS Web3 Function polls the hub, finds due IDs, and submits a charge(ids[]) transaction.
  • Pricing is post-paid from Gelato's 1Balance.

Either one is sufficient on its own; we will pick based on operational experience during Phase 3. Running both is fine — charge() is idempotent within a period thanks to lastChargedAt.

SLOs and monitoring​

Published SLO targets:

MetricTarget
charge() within 10 minutes of nextChargeAt99.5%
charge() within 1 hour of nextChargeAt99.9%
Keeper EOA hot-gas balance above alert threshold99.99%
Events indexed within 2 minutes of block finality99.5%

Monitoring hooks into the existing Sentry setup documented in the repo's Sentry workflow skill. Alerts:

  • charge() reverts per hour > threshold → PagerDuty
  • Keeper EOA native balance below N gwei × gasLimit × 100 → PagerDuty
  • Chainlink/Gelato upkeep unhealthy → warning
  • Paid event indexer lag > 5 min → warning

Deprecations​

Same as Phase 1 and 2, but on more chains. Nothing is deleted yet — deletion waits for Phase 7.

Migration plan​

  • New payment links: use contract receivers on any chain with the flag on.
  • New subscriptions: use SubscriptionHub on any chain with the flag on.
  • Existing customers: opt-in migration at renewal, same as Phase 2.
  • Cross-chain settlement (Relay): unchanged surface; the keeper still calls Relay executors in api/src/withdrawals/services/bridge, just from an EOA that has no custody.

Risks and mitigations​

  • Multi-chain address drift. Mitigation: deterministic CREATE2 deployer (e.g. 0x4e59b44847b379578588920cA78FbF26c0B4956C) with a locked init-code hash; factories share an address across all four chains.
  • Per-chain paymaster policy limits. Mitigation: per-chain monitoring and policy quota alerts at 70% usage.
  • Chainlink upkeep registration churn. Mitigation: one upkeep per chain; the list of IDs is read on-chain, so merchant onboarding does not require Chainlink config changes.
  • Double-charge across keepers. Mitigation: lastChargedAt monotonic plus ChargeSkipped for duplicates; both keepers are idempotent by design.
  • Mainnet gas spikes. Mitigation: per-chain batch size tuning; a gas-price ceiling above which charge() is deferred one block.

Status checklist​

  • Deploys completed on Ethereum, Arbitrum, Optimism, Polygon
  • Factory + hub addresses published in contract reference
  • Paymaster policies validated per chain
  • Chainlink Automation upkeep live on at least 2 chains
  • SLO dashboard in Datadog
  • Runbook for keeper outage (manual charge() from merchant CLI)
  • Canary merchants rolled per chain

What is next​

Solana is the other big chain we must make non-custodial: Phase 4.

Reference docs: Phase 3 status · Keeper strategy · Contract reference