Skip to main content

Keeper strategy

SubscriptionHub.charge(id) (EVM) and the Anchor charge instruction (Solana) are public and permissionless. Any EOA can call them. A "keeper" is just a scheduler that does.

This page explains how OrcaRail triggers charges, what happens when a keeper is unavailable, and what SLOs we publish.

Primary keeper: NestJS scheduler​

The primary keeper lives in api/src/scheduler/processors/subscription-auto-charge.processor.ts. It is a BullMQ processor that:

  1. Reads due subscriptions from the database (mirror of on-chain state maintained by the indexer).
  2. Groups by chain id.
  3. Signs and submits a charge(ids[]) transaction per chain using a per-chain keeper EOA.
  4. On success, waits for confirmation and updates the database via indexer events.
  5. On revert, parses the revert selector and maps it to a SubscriptionAutoChargeEntity state:
    • ChargeSkipped(InsufficientAllowance) → past_due + allowance email
    • ChargeSkipped(InsufficientBalance) → past_due + balance email
    • ChargeSkipped(CapExceeded) → canceled (payer must re-approve)
    • ChargeSkipped(NotDue) → benign (indexer rounding); retry in 1 cycle

From Phase 3 onward, each SubscriptionHub deployment registers an Upkeep with Chainlink Automation:

  • checkUpkeep(bytes) returns performData = abi.encode(ids[]) for IDs where isDue(id) && !chargedThisBlock(id).
  • performUpkeep(bytes) calls charge(ids[]) in a batch.
  • Funded with LINK from the platform treasury.
  • Configured for "conditional" mode with a gas limit suitable for batch size.

Fallback keeper: Gelato Web3 Functions (alternative)​

If Chainlink is not available on a chain (or the pricing doesn't fit), a Gelato Web3 Function polls the hub's isDue() view and submits charge(ids[]) when due. Paid post-pay from Gelato's 1Balance.

Solana keeper​

Solana's NestJS branch uses the same scheduler pattern with a SOL-signing keeper. Solana has no Chainlink Automation equivalent with equivalent coverage today; fallback options:

  • Clockwork (if its maintenance status is acceptable at rollout time) — Solana-native cron.
  • A second NestJS instance in a different region as a simpler redundancy.

Solana fees are low enough that gas-balance monitoring is less critical than EVM.

Gas management​

EVM​

  • Chains with Alchemy paymaster policy: keeper calls charge() as a sponsored UserOperation via the existing logic in api/src/withdrawals/services/gas-strategy.
  • Chains without paymaster: keeper self-funds from its hot-gas balance. Funded daily by the platform wallet.
  • Gas-price ceiling: charges are deferred one block if gas price exceeds a per-chain threshold.
  • Merchant-subsidized gas (optional): merchants can set a gas_refund_address; reimbursement is pulled through the fee-split logic.

Solana​

  • Keeper runs with a stable SOL float per chain.
  • Priority fee is set dynamically based on recent blocks; retries bump priority one step.

SLOs​

MetricTargetApplies to
charge() within 10 minutes of nextChargeAt99.5%All supported chains
charge() within 1 hour of nextChargeAt99.9%All supported chains
Keeper EOA hot-gas balance above alert threshold99.99%All supported chains
Paid / Charged events indexed within 2 minutes99.5%All supported chains
charge() revert rate (non-benign reverts only)< 0.1%All supported chains

Non-benign reverts exclude ChargeSkipped(InsufficientAllowance), ChargeSkipped(InsufficientBalance), and ChargeSkipped(CapExceeded) — those are expected payer-side conditions.

Monitoring and alerts​

Using the existing Sentry + Datadog pipeline:

  • charge() revert rate per hour > threshold → PagerDuty
  • Keeper EOA native balance below gasPrice × gasLimit × 100 → PagerDuty
  • Chainlink upkeep balance below threshold → warning
  • Gelato 1Balance below threshold → warning
  • Indexer event lag > 5 minutes → warning

Dashboards: one per chain, plus a global roll-up. Runbooks for outages are linked from the on-call rotation.

Reorg protection​

  • SubscriptionHub.charge(id) enforces block.timestamp > lastChargedAt monotonically. A reorg that would replay a charge is reverted with AlreadyChargedThisPeriod.
  • The Solana program uses last_charged_at the same way; Solana has much rarer reorgs, but the same invariant holds.

Manual intervention​

Merchants can trigger charge(id) manually via a dashboard action in emergencies (e.g. NestJS + Chainlink simultaneously offline). The transaction signs from the merchant's own wallet; it is not custodial.

What the keeper cannot do​

Explicit list, to make the scope obvious:

  • Create subscriptions without a payer signature.
  • Raise cap, amount, or interval on an existing subscription.
  • Move funds to an address not in the subscription's Split.
  • Bypass nextChargeAt or lastChargedAt.
  • Disable pause, cancel, or refund flows.

The keeper holds gas and nothing else.

See also​