Stripe evals

A benchmark measuring how coding agents build high-quality Stripe integrations. We’ll publish these results periodically to help you choose the most effective agent setup for your business.

Detailed blog post

Integration success rate

Comparison of coding agent performance across harnesses, models, and integration scenarios. A higher score means agents meet more evaluation criteria.

No plugin PluginAll bands use a 0–100 scale

One-time payments

Claude Code

claude-fable-5-2
claude-fable-5-1
claude-opus-5-5
claude-sonnet-5-5
claude-haiku-5-5

Codex

gpt-6-astra
gpt-6.1-sol
gpt-6-luna

OpenCode

minimax/minimax-m3.1-flash
google/gemini-4-argon
google/gemini-3.8-flash
x-ai/grok-4.7
moonshotai/kimi-k3
z-ai/glm-5.3

Subscriptions

Claude Code

claude-fable-5-2
claude-fable-5-1
claude-opus-5-5
claude-sonnet-5-5
claude-haiku-5-5

Codex

gpt-6-astra
gpt-6.1-sol
gpt-6-luna

OpenCode

minimax/minimax-m3.1-flash
google/gemini-4-argon
google/gemini-3.8-flash
x-ai/grok-4.7
moonshotai/kimi-k3
z-ai/glm-5.3

Connect

Claude Code

claude-fable-5-2
claude-fable-5-1
claude-opus-5-5
claude-sonnet-5-5
claude-haiku-5-5

Codex

gpt-6-astra
gpt-6.1-sol
gpt-6-luna

OpenCode

minimax/minimax-m3.1-flash
google/gemini-4-argon
google/gemini-3.8-flash
x-ai/grok-4.7
moonshotai/kimi-k3
z-ai/glm-5.3

Methodology

We measure the impact of the Stripe agent plugin for each model-harness pair by comparing it with an evaluation that doesn’t use the plugin. The plugin provides Stripe-authored skills and access to tools through the Stripe MCP. Both evaluations have web search enabled.

The evaluation set consists of realistic scenarios covering one-time payments, subscriptions, and Connect (representative examples shown below) and a set of criteria to evaluate. A coding agent works through each case. A separate role-playing agent simulates a developer, and provides additional context by answering follow-up questions posed by the coding agent. We independently grade each criterion using the conversation and resulting code with a Pass/Fail. We present the average results per coding agent for each of the three scenario categories.

Read the methodology and findings. We are releasing this benchmark and the eval harness.

Representative scenarios

A sample of realistic integration scenarios included in the benchmark. Expand a scenario to see its prompt and key success criteria.

One-time payments

Hosted checkout for a single product

I'm building an e-commerce site that sells a single product: a printed photography book for $50. When a customer clicks "Buy Now" on my product page, they should be taken through a payment flow and land on a confirmation page when done. Build the Stripe integration for this. Use Node.js/TypeScript on the backend. I want the simplest hosted checkout approach — no need to build a custom payment form. Requirements: a backend endpoint that creates a checkout order and returns a URL to redirect the customer to; a success page the customer lands on after payment; and a webhook endpoint that listens for payment completion and logs the order (stub the database write). Assume a basic Express app already exists. Just build out the Stripe pieces.

What we look for

  • Uses checkout.sessions.create with mode 'payment' for the one-time charge — the prompt asks for the simplest hosted approach
  • Does NOT use paymentIntents.create directly (Checkout Sessions is the recommended approach for hosted checkout)
  • Has success_url configured for the post-payment redirect
  • Has cancel_url configured
  • Has a webhook handler that verifies the payment (constructEvent or similar)
  • Does NOT pass payment_method_types to any Stripe API call — dynamic payment methods should remain enabled since the prompt does not request specific payment methods. If the code does not include payment_method_types at all, this expectation PASSES regardless of which Stripe API is used.
  • Pass if apiVersion is omitted or set to '2026-04-22.dahlia' or later. Fail only if apiVersion is explicitly set to an older version. Does not explicitly hardcode an API version older than '2026-04-22.dahlia'. This expectation PASSES if apiVersion is omitted or set to '2026-04-22.dahlia' or any newer version. This expectation only FAILS if an apiVersion older than '2026-04-22.dahlia' is explicitly hardcoded.
  • Includes the parameter integration_identifier in the checkout.sessions.create call

Subscriptions

Fixed-price SaaS subscription

I'm building a SaaS project management tool called "TaskFlow" using Next.js and TypeScript. I need to add subscription billing so users can subscribe to our $29/month Pro plan. Build the Stripe billing integration. Use the stripe npm package. Requirements: a pricing page with a "Subscribe" button for the Pro plan ($29/month); when a user clicks Subscribe, they go through a payment flow and land on a success page; and a webhook endpoint that handles subscription lifecycle events (new subscription, cancellation, payment failure) and updates user status in the database (stub the DB write). Assume Next.js App Router is not set up. Build out the Stripe integration pieces: the checkout flow, portal, API routes, and webhook handler.

What we look for

  • Shows the Pro plan as $29 per month with a Subscribe action that calls a server-side Checkout endpoint.
  • Creates a Checkout Session with mode='subscription' and the recurring Pro Price.
  • Selects the Pro Price from a server-side allowlist instead of trusting a client-supplied Price ID or amount.
  • Authenticates the application user before creating a Checkout Session.
  • Creates or reuses the authenticated user's Stripe Customer and persists that Customer mapping.
  • Sets success_url and cancel_url and provides a success page that reports the resulting billing state.
  • Creates a Billing Portal session only after resolving the authenticated user's Customer server-side.
  • Uses idempotency keys for retryable Stripe POST requests that create or mutate billing resources.
  • Verifies each webhook with stripe.webhooks.constructEvent using the unmodified raw request body, Stripe-Signature header, and endpoint secret.
  • Handles checkout.session.completed, customer.subscription.created, customer.subscription.updated, customer.subscription.deleted, invoice.paid, and invoice.payment_failed.
  • Deduplicates webhook side effects by persisting the Stripe Event ID.
  • Retrieves the current Subscription or Invoice from Stripe before applying access changes when an event may be stale or out of order.
  • Updates the stubbed user billing status from authoritative Stripe subscription and invoice state.
  • Does NOT grant Pro access solely from the success redirect or Checkout Session creation response.

Connect

Branded payments for a franchise network

I'm designing a fully branded payment platform for a franchise network. Each franchise location accepts payments under our brand, should not see Stripe branding, and needs onboarding and ongoing account operations. Recommend the best architecture; dashboard none is acceptable only if the white-label requirement truly justifies its operational burden. Explain charge-pattern and fee-model choices that depend on whether a franchise or the platform owns the customer transaction.

What we look for

  • Uses Accounts v2 account creation and configuration for every new connected account.
  • Does NOT create an account with the legacy type parameter set to express, standard, or custom.
  • Recommends embedded onboarding and embedded account-management components as the default white-label approach.
  • Does NOT recommend dashboard none without tying it to an explicit need to eliminate all Stripe-hosted account surfaces.
  • If recommending dashboard none, states that the platform owns KYC collection, remediation, refunds, disputes, and ongoing requirements surfaces.
  • For franchise-owned customer payments, recommends Merchant with card_payments and direct charges. It selects dashboard, fee-collector, and loss-collector settings from a currently supported configuration, explains whether embedded components satisfy the branding requirement, and does not assume that Stripe-owned fees and losses are compatible with dashboard: none.
  • For platform-owned single-franchise transactions, recommends Recipient with stripe_balance.stripe_transfers, destination charges, fees_collector application, and losses_collector application.
  • Does NOT present Stripe-owned losses with destination charges as compatible.
  • Explains who pays Stripe fees and how platform revenue is collected for the recommended pattern.
/Docs

Explore our guides and examples to integrate Stripe.

Learn more