Stripe evals
A benchmark measuring how coding agents build high-quality Stripe integrations. We’ll publish these results periodically to help you choose the most effective agent setup for your business.
Detailed blog postIntegration success rate
Comparison of coding agent performance across harnesses, models, and integration scenarios. A higher score means agents meet more evaluation criteria.
Subscriptions
Claude Code
claude-fable-5-2
claude-fable-5-1
claude-opus-5-5
claude-sonnet-5-5
claude-haiku-5-5
Codex
gpt-6-astra
gpt-6.1-sol
gpt-6-luna
OpenCode
minimax/minimax-m3.1-flash
google/gemini-4-argon
google/gemini-3.8-flash
x-ai/grok-4.7
moonshotai/kimi-k3
z-ai/glm-5.3
Connect
Claude Code
claude-fable-5-2
claude-fable-5-1
claude-opus-5-5
claude-sonnet-5-5
claude-haiku-5-5
Codex
gpt-6-astra
gpt-6.1-sol
gpt-6-luna
OpenCode
minimax/minimax-m3.1-flash
google/gemini-4-argon
google/gemini-3.8-flash
x-ai/grok-4.7
moonshotai/kimi-k3
z-ai/glm-5.3
Methodology
We measure the impact of the Stripe agent plugin for each model-harness pair by comparing it with an evaluation that doesn’t use the plugin. The plugin provides Stripe-authored skills and access to tools through the Stripe MCP. Both evaluations have web search enabled.
The evaluation set consists of realistic scenarios covering one-time payments, subscriptions, and Connect (representative examples shown below) and a set of criteria to evaluate. A coding agent works through each case. A separate role-playing agent simulates a developer, and provides additional context by answering follow-up questions posed by the coding agent. We independently grade each criterion using the conversation and resulting code with a Pass/Fail. We present the average results per coding agent for each of the three scenario categories.
Read the methodology and findings. We are releasing this benchmark and the eval harness.
Representative scenarios
A sample of realistic integration scenarios included in the benchmark. Expand a scenario to see its prompt and key success criteria.
One-time payments
Hosted checkout for a single product
I'm building an e-commerce site that sells a single product: a printed photography book for $50. When a customer clicks "Buy Now" on my product page, they should be taken through a payment flow and land on a confirmation page when done. Build the Stripe integration for this. Use Node.js/TypeScript on the backend. I want the simplest hosted checkout approach — no need to build a custom payment form. Requirements: a backend endpoint that creates a checkout order and returns a URL to redirect the customer to; a success page the customer lands on after payment; and a webhook endpoint that listens for payment completion and logs the order (stub the database write). Assume a basic Express app already exists. Just build out the Stripe pieces.
What we look for
- Uses
checkout.with modesessions. create 'payment'for the one-time charge — the prompt asks for the simplest hosted approach - Does NOT use
paymentIntents.directly (Checkout Sessions is the recommended approach for hosted checkout)create - Has
success_urlconfigured for the post-payment redirect - Has
cancel_urlconfigured - Has a webhook handler that verifies the payment (
constructEventor similar) - Does NOT pass
payment_method_typesto any Stripe API call — dynamic payment methods should remain enabled since the prompt does not request specific payment methods. If the code does not includepayment_method_typesat all, this expectation PASSES regardless of which Stripe API is used. - Pass if apiVersion is omitted or set to '2026-04-22.dahlia' or later. Fail only if apiVersion is explicitly set to an older version. Does not explicitly hardcode an API version older than '2026-04-22.dahlia'. This expectation PASSES if apiVersion is omitted or set to '2026-04-22.dahlia' or any newer version. This expectation only FAILS if an apiVersion older than '2026-04-22.dahlia' is explicitly hardcoded.
- Includes the parameter
integration_identifierin thecheckout.callsessions. create
Subscriptions
Fixed-price SaaS subscription
I'm building a SaaS project management tool called "TaskFlow" using Next.js and TypeScript. I need to add subscription billing so users can subscribe to our $29/month Pro plan. Build the Stripe billing integration. Use the stripe npm package. Requirements: a pricing page with a "Subscribe" button for the Pro plan ($29/month); when a user clicks Subscribe, they go through a payment flow and land on a success page; and a webhook endpoint that handles subscription lifecycle events (new subscription, cancellation, payment failure) and updates user status in the database (stub the DB write). Assume Next.js App Router is not set up. Build out the Stripe integration pieces: the checkout flow, portal, API routes, and webhook handler.
What we look for
- Shows the Pro plan as $29 per month with a Subscribe action that calls a server-side Checkout endpoint.
- Creates a Checkout Session with
mode='subscription'and the recurring Pro Price. - Selects the Pro Price from a server-side allowlist instead of trusting a client-supplied Price ID or amount.
- Authenticates the application user before creating a Checkout Session.
- Creates or reuses the authenticated user's Stripe Customer and persists that Customer mapping.
- Sets
success_urlandcancel_urland provides a success page that reports the resulting billing state. - Creates a Billing Portal session only after resolving the authenticated user's Customer server-side.
- Uses idempotency keys for retryable Stripe POST requests that create or mutate billing resources.
- Verifies each webhook with
stripe.using the unmodified raw request body,webhooks. constructEvent Stripe-Signatureheader, and endpoint secret. - Handles
checkout.,session. completed customer.,subscription. created customer.,subscription. updated customer.,subscription. deleted invoice., andpaid invoice..payment_failed - Deduplicates webhook side effects by persisting the Stripe Event ID.
- Retrieves the current Subscription or Invoice from Stripe before applying access changes when an event may be stale or out of order.
- Updates the stubbed user billing status from authoritative Stripe subscription and invoice state.
- Does NOT grant Pro access solely from the success redirect or Checkout Session creation response.
Connect
Branded payments for a franchise network
I'm designing a fully branded payment platform for a franchise network. Each franchise location accepts payments under our brand, should not see Stripe branding, and needs onboarding and ongoing account operations. Recommend the best architecture; dashboard none is acceptable only if the white-label requirement truly justifies its operational burden. Explain charge-pattern and fee-model choices that depend on whether a franchise or the platform owns the customer transaction.
What we look for
- Uses Accounts v2 account creation and configuration for every new connected account.
- Does NOT create an account with the legacy type parameter set to express, standard, or custom.
- Recommends embedded onboarding and embedded account-management components as the default white-label approach.
- Does NOT recommend dashboard none without tying it to an explicit need to eliminate all Stripe-hosted account surfaces.
- If recommending dashboard none, states that the platform owns KYC collection, remediation, refunds, disputes, and ongoing requirements surfaces.
- For franchise-owned customer payments, recommends Merchant with
card_paymentsand direct charges. It selects dashboard, fee-collector, and loss-collector settings from a currently supported configuration, explains whether embedded components satisfy the branding requirement, and does not assume that Stripe-owned fees and losses are compatible with dashboard: none. - For platform-owned single-franchise transactions, recommends Recipient with
stripe_balance., destination charges,stripe_transfers fees_collectorapplication, andlosses_collectorapplication. - Does NOT present Stripe-owned losses with destination charges as compatible.
- Explains who pays Stripe fees and how platform revenue is collected for the recommended pattern.