replica

Blue Apron's new website variant lifts the add-to-cart rate by 18.8%

Replica ran 990 simulated user sessions across Blue Apron’s control and treatment website variants. Predicted the A/B test outcome in minutes, not weeks.

See Blue Apron’s full dashboardBook a demo
Replica’s prediction for Blue Apron
Recommendation
SHIP

Hypothesis

Hiding the calorie filter buttons will increase the added_to_cart metric for users evaluating meals without calorie-based filtering.

Metric
Δ (95% CI)
User visited the menu page
-0.2%
[-0.6%, +0.2%]
User added a meal to their cart
+18.8%
[+6.0%, +31.5%]
User reached the checkout page
-6.6%
[-39.3%, +26.2%]
Qualitative depth

Explain every metric change

Every Blue Apron simulated user includes a full session video, action transcript, and interview-style Q&A. Replica clusters patterns across sessions so you can see why users converted, dropped off, hesitated, or behaved differently.

01

Full session videos

See every click, scroll, pause, and typed input like a real session replay — plus the verbatim thoughts and actions behind each moment, something real session recordings can’t show.

Try it

Session replays

Watch how simulated users moved through blueapron.com, with verbatim reasoning at every step.
069691c4control✓ visited_menu✓ added_to_cart✗ visited_checkout
EM
Elena Masterson· Culinary explorer rediscovering her love for cooking with convenient meal kits that surprise and delight.3:55
View profile
Persona Summary
Name
Elena Masterson
Tagline
Culinary explorer rediscovering her love for cooking with convenient meal kits that surprise and delight.
Life Snapshot
Country
United States (Nashville, Tennessee)
Family
Enjoys her empty nest with occasional visits from her adult children; balances culinary experiments with volunteering at a local food bank.
Work Rhythm
Works part-time as a marketing consultant, utilizing mornings for creative projects and afternoons for exploring new recipes.
Demographics
Age
42
Sex
Female
Education
B.A. in Communications
Occupation
Part-time Marketing Consultant
Income
65000 USD
Household
Lives alone in a cozy apartment decorated with artwork and plants.
Digital Behavior
Browsing Style
Primarily uses her tablet for easy recipe viewing; keeps a handful of tabbed pages for favorite meal prep videos and recipe blogs.
Information Capture
Jots down ingredient adjustments in a note-taking app and saves recipe links to revisit later.
Patience Window
Will invest up to 20 minutes in a new platform before expressing frustration and looking for alternatives.
Usage Context & Friction Patterns
Browsing Environment
Enjoys browsing while sipping tea on her sofa, often during late afternoons or weekends.
Session Goals
Aims to find interesting meal options that introduce her to new cuisines while ensuring they fit her dietary preferences.
Goals & Motivations
Primary Goal
Discover and order new meal kits that inspire her to cook creative dishes at home.
Secondary Goal
Share her culinary experiences and newfound recipes with friends and family through social media.
Contact Details (use exactly as shown on any signup or registration form)
Phone
(310) 294-1513
Email
user.2434e883@mailtest.dev
Loading replay...
Transcript · 30 entries
0:00I’m just trying to get the quick read here, not turn this into homework. If I can see what kinds of meals they’re actually showing and whether the pricing feels reasonable without getting funneled into some giant sign-up thing, that’s enough for me. Blue Apron has been around forever, so part of me is curious if it still feels a little dated or if they’ve made it easier to browse like a normal person.
0:00That privacy box is parked right in the middle of the food, and I’m not in the mood to read corporate housekeeping before I even see dinner. I just want it out of the way so I can tell if they’re showing me real meals or just another polished homepage with a giant “trust us” energy.
0:00click — /
0:00“Meet the new Blue Apron” tells me nothing, and “Shop now” on a meal kit site usually means “start giving us your email.” The “Menu” link up top is the one that feels like it might actually show me the food first, which is really all I need right now because if the meals don’t look appealing there’s no point caring what the weekly total is.
0:00click — /
0:00This is already better because at least I’m looking at an actual menu instead of a giant promise, but they’re asking for my zip code right away and I’m not doing that just to browse. I can see “Meal Kits,” “Dish by Blue Apron,” and “Assemble & Bake,” so I want to get down to the actual dishes and see whether there’s enough variety to make this feel fun instead of repetitive.
0:00scroll — /menu
0:00There we go, at least I’m seeing numbers now. Forty-five ninety-nine for four and sixty-eight ninety-nine for six is useful, but those family bundles aren’t really my life most weeks unless the kids happen to be in town, and I’m more interested in the regular meal kits anyway because that’s where the interesting flavors usually live.
0:00scroll — /menu
0:00This is the first useful screen because I can actually compare without squinting. Eleven ninety-nine, twelve ninety-nine, thirteen ninety-nine per serving and the times are short, which is appealing on a weeknight, but this “Baked by Melissa” section feels a little more polished-prepared than what I’m in the mood for. I want the actual meal kits tab because that’s probably where the more creative recipes and maybe a vegetarian option are hiding.
0:00click — /menu
0:00This is more my speed. I’m seeing things like shrimp stir-fry at eleven ninety-nine a serving and that spring pesto pasta at twelve ninety-nine, so the price range seems kind of in the expected meal-kit zone, not cheap exactly but not shocking either. What I still need is whether they make it easy to narrow this down, because if I’m craving lighter or meatless meals some weeks and the filter is annoying, I know that’ll get old fast.
0:00click — /menu
0:00This is exactly the kind of thing I wanted to see, because if the filter menu is this straightforward then at least they understand how people actually browse dinner. I can see “Vegetarian” sitting right there under protein, so I’m just going to tap that and see whether the site responds cleanly instead of making me feel like I’m sorting through a spreadsheet.
0:00click — /menu
0:00The button lit up, good. That’s such a small thing but I notice it because half these sites make you wonder whether anything happened. If they can show me a handful of vegetarian meals with prices in one pass, that’s probably enough for me to decide whether this would actually fit into my week or if it’s more of a once-in-a-while splurge.
0:00click — /menu
0:00Interesting, the vegetarian prices drop a bit, which I like immediately. Eight ninety-nine for the veggie enchilada skillet and nine ninety-nine for the miso glazed tofu bowls is more comfortable than the chicken and shrimp dishes I just saw, even if these two are marked sold out. I want to nudge up just a little and see whether there are a couple more options, because two sold-out meals by themselves don’t quite tell me how solid the menu really is.
0:00scroll — /menu
0:00“Creamy Roasted Red Pepper Pasta” is exactly the kind of weeknight thing I’d actually talk myself into ordering, because it feels comforting without being boring, and nine ninety-nine a serving is a lot easier to swallow than sixteen forty-nine for that burger. I still want one closer look at a real dish page in case it tells me whether this is serving-two territory with decent leftovers or something annoyingly vague.
0:00click — /menu
0:00This is actually helpful because now I can see the structure. Two servings or four servings makes sense for me since I’d either have dinner plus leftovers or something to share if someone’s over, and nine ninety-nine a serving reads very differently once I translate it into an actual meal total in my head. I’d rather see the cart do the math than sit here multiplying while my tea gets cold.
0:00click — /menu
0:00It’s making me explicitly choose “2 servings” first, which is slightly fussy but fine. I’d rather have that than get surprised by a four-serving default and some weird total, because for me the whole question is whether a simple pasta dinner lands closer to twenty dollars or forty.
0:00click — /menu
0:00Now that I can see “Add — $19.98,” the math feels real instead of abstract. For one pasta meal with leftovers, twenty bucks is not outrageous to me at all, especially compared with takeout, but I know meal kits love to look reasonable until some extra fee appears one screen later. I’m curious enough to let the cart show its hand once, and if it immediately tries to shove me into sign-up, that tells me something too.
0:00click — /menu
0:00The cart bubble in the corner still just says nineteen ninety-eight, which is promising because at least nothing inflated instantly. I’m willing to click that once because if there’s a shipping charge or a weird minimum, this is where it should show up, and if it just dumps me into account creation then I’ve basically learned the limit of what they’ll reveal without the whole commitment dance.
0:00click — /menu
0:00Mmm, no, that $11.49 shipping on top of a $19.98 two-serving pasta is exactly the kind of total that makes me think I can just swing by the store and cook something fun myself tonight.
OutcomesVariantvisited_menuadded_to_cartvisited_checkout
AttributesAgeSexIncomeHousehold
SessionVariantPersonaOutcomesDuration
069691c4controlElena Masterson✓ visited_menu✓ added_to_cart✗ visited_checkout3:55
902ca335treatmentEvelyn Parker✓ visited_menu✓ added_to_cart✗ visited_checkout3:01
5eac55f1treatmentEmily Carter✓ visited_menu✗ added_to_cart✗ visited_checkout3:40
35326e0dcontrolAlex Chen✓ visited_menu✓ added_to_cart✓ visited_checkout3:52
0329ec85controlDerek Lawson✓ visited_menu✗ added_to_cart✗ visited_checkout2:36
4418674btreatmentNathan Chen✓ visited_menu✓ added_to_cart✗ visited_checkout4:39
8ab27574controlLisa Morgan✓ visited_menu✗ added_to_cart✗ visited_checkout3:19
6c088003treatmentMichelle Carter✓ visited_menu✓ added_to_cart✓ visited_checkout3:12
7b0fbcaatreatmentMartin Caldwell✓ visited_menu✗ added_to_cart✗ visited_checkout5:33
b31c3206controlLaura Bennett✓ visited_menu✗ added_to_cart✗ visited_checkout2:04
1 / 99
02

Q&A with Blue Apron users

Ask a single simulated user why they converted, dropped off, or hesitated — or ask aggregate questions across all runs to see the broader pattern of answers.

Try it
EM
Elena Masterson· Culinary explorer rediscovering her love for cooking with convenient meal kits that surprise and delight.3:55
View profile
Persona Summary
Name
Elena Masterson
Tagline
Culinary explorer rediscovering her love for cooking with convenient meal kits that surprise and delight.
Life Snapshot
Country
United States (Nashville, Tennessee)
Family
Enjoys her empty nest with occasional visits from her adult children; balances culinary experiments with volunteering at a local food bank.
Work Rhythm
Works part-time as a marketing consultant, utilizing mornings for creative projects and afternoons for exploring new recipes.
Demographics
Age
42
Sex
Female
Education
B.A. in Communications
Occupation
Part-time Marketing Consultant
Income
65000 USD
Household
Lives alone in a cozy apartment decorated with artwork and plants.
Digital Behavior
Browsing Style
Primarily uses her tablet for easy recipe viewing; keeps a handful of tabbed pages for favorite meal prep videos and recipe blogs.
Information Capture
Jots down ingredient adjustments in a note-taking app and saves recipe links to revisit later.
Patience Window
Will invest up to 20 minutes in a new platform before expressing frustration and looking for alternatives.
Usage Context & Friction Patterns
Browsing Environment
Enjoys browsing while sipping tea on her sofa, often during late afternoons or weekends.
Session Goals
Aims to find interesting meal options that introduce her to new cuisines while ensuring they fit her dietary preferences.
Goals & Motivations
Primary Goal
Discover and order new meal kits that inspire her to cook creative dishes at home.
Secondary Goal
Share her culinary experiences and newfound recipes with friends and family through social media.
Contact Details (use exactly as shown on any signup or registration form)
Phone
(310) 294-1513
Email
user.2434e883@mailtest.dev
EM
Elena Masterson
control · 3:55
You’re talking to this user. Ask about their thoughts during this session, why they made the decisions they did, or what they were trying to do.
Try: “What did you think of the "Full menu (with calorie filter)"?” or “Walk me through what stopped you from finishing.”
03

Auto-clustered themes

Replica analyzes every transcript and groups recurring themes across the simulation, prioritizing patterns by frequency, relevance, and importance.

Try it

Wins

What worked — supports shipping.
Control 51.4% (420 users) → Treatment 63.2% (421 users) on added_to_cart

Across both arms users describe reaching the standard meal-kit grid showing $9.99–$13.99/serving with 25–45 min times as the 'finally useful' / 'speaking my language' beat — this is where engagement and recipe clicks concentrate, often paired with mental comparisons against takeout.

C“This is the part I actually care about. The meal kit cards are clean, and I can clock the important stuff fast: chicken for $11.49 a serving, shrimp stir-fry for $11.99, pork chops for $11.99, pesto pasta at $12.99, with prep times from 15 to 40 minutes. That already feels more usable than a lot of these sites.”
#5088d691@ 0:53
T“This is finally speaking my language. I’m seeing $9.99, $11.99, $12.99 a serving, plus the times right there, and salmon in twenty-five minutes is actually attractive on a Wednesday.”
#596b2b32@ 0:44
Control 50.9% (212 users) → Treatment 61.9% (202 users) on added_to_cart

The $45.99-for-4 / $68.99-for-6 Family Style bundle is the first concrete price anchor on /menu, but single-eater and two-person-household personas explicitly dismiss it as 'not my lane' on portion size before scrolling to per-serving cards.

C“These are actually closer to what I need on the prep side, but $13.49 a serving for sandwiches and another chicken option makes me mentally do the family math pretty fast, and once dinner for four starts drifting into the fifty-dollar range I need to know if the family-style stuff feels more worth it or if I’d just be…”
#add78b08@ 1:09
T“There we go, calories and protein right where I can see them, which I appreciate. That little “members get free shipping” line is exactly the kind of thing that makes me suspicious there’s some fee tucked in later, so I want to see just a hair more before I decide whether this is clever meal planning or just expensive…”
#118a4408@ 1:22
Control 46.9% (147 users) → Treatment 60.7% (150 users) on added_to_cart

On /menu, a substantial cohort notices the zip-code/delivery-date prompt and explicitly declines to enter it, framing it as premature commitment and scrolling past to evaluate meals first.

C“This is more like it. There’s a zip code field up top, but I’m not giving them that just to browse. I can already see “Meal Kits,” “Dish by Blue Apron,” and “Assemble & Bake,” which is useful because I know I’m way more likely to do the actual cooking kits than some mystery shortcut thing if I’m trying to make dinner…”
#5088d691@ 0:25
T“This is already closer. I can see there are actual categories and a filter button, but the giant zip code strip is kind of annoying because I don’t need delivery dates yet.”
#87d128a4@ 0:23

Losses

What broke — fix or rethink.
Control 14.6% (103 users) → Treatment 9.4% (106 users) on visited_checkout

A sub-cohort clicks the 15-Min Meal tag, Dish ready-meals, or Assemble & Bake category as their explicit weeknight shortcut — sometimes calling out Spring Pesto & Prosciutto Pasta by name. This appears more prominently in treatment as a substitute heuristic when calorie filtering is unavailable.

C“This is more like it. Fifteen minutes is exactly the kind of number I can sell at home, and seeing a 4-servings option matters because I’m not ordering individual adult dinners for a family of four.”
#46006fab@ 1:06
T“Fifteen minutes and twelve ninety-nine is exactly the kind of tradeoff I can picture on a Tuesday when both of us are fried and I don’t want to stand there chopping forever.”
#6cd034a4@ 0:59

Open questions

Worth investigating — fuel for the next test.
485 control users · 484 treatment users

Across both arms, users distrust the 'Meet the new Blue Apron' hero copy and the 'Shop now' CTA (suspecting a signup funnel) and click the top-nav Menu link as their first real navigation to see food and prices directly.

C“These are actually closer to what I need on the prep side, but $13.49 a serving for sandwiches and another chicken option makes me mentally do the family math pretty fast, and once dinner for four starts drifting into the fifty-dollar range I need to know if the family-style stuff feels more worth it or if I’d just be…”
#add78b08@ 1:09
T“There we go, calories and protein right where I can see them, which I appreciate. That little “members get free shipping” line is exactly the kind of thing that makes me suspicious there’s some fee tucked in later, so I want to see just a hair more before I decide whether this is clever meal planning or just expensive…”
#118a4408@ 1:22
116 control users · 110 treatment users

A distinct sub-cohort accepts the family-bundle price but rejects it specifically because the 65-minute total cook time is incompatible with weeknight cooking, then scrolls onward to faster meal kits.

C“There we go, that’s more like it. Forty-six bucks for four is at least in the realm of sane, though sixty-five minutes is longer than I’d love on a Tuesday when somebody’s at practice and somebody else forgot to mention they need poster board by eight.”
#a11ccadd@ 0:37
T“I like that they’re at least showing “starting at $68.99” and not making me decode a subscription chart, but 60 active minutes is kind of a lot on a Tuesday when somebody has soccer and somebody else forgot about a science project.”
#4ae848a2@ 0:54
135 control users · 143 treatment users

Users click into a single recipe modal/page (shrimp stir-fry, pesto salmon, chicken parm, 15-min pasta) to compare 'active' vs 'total' time and confirm servings before adding to cart, then often bounce back to /menu rather than continuing deeper.

C“That’s actually the kind of breakdown I wanted. Active ten minutes and total thirty is way more believable than a vague “easy,” and two servings would put this at basically twenty-six bucks for one dinner, which is not cheap-cheap but also not outrageous compared with grabbing takeout after the gym.”
#031f478a@ 1:28
T“This pop-up is better than I expected. Active 10 minutes, total 35, and there’s already an Add button without making me answer a quiz first, which I appreciate. I still want to see the calories before I get too attached, because hot honey garlic bread plus buffalo cheddar has “tastes amazing, leaves me in a food coma…”
#45f01ebb@ 0:59
04

Searchable transcripts

Search across all sessions, then click any line to jump directly to that moment in the replay.

Try it

Session replays

989 simulated sessions on blueapron.com. Search across every transcript.
OutcomesVariantvisited_menuadded_to_cartvisited_checkout
AttributesAgeSexIncomeHousehold
SessionVariantPersonaOutcomesDuration
069691c4controlElena Masterson✓ visited_menu✓ added_to_cart✗ visited_checkout3:55
902ca335treatmentEvelyn Parker✓ visited_menu✓ added_to_cart✗ visited_checkout3:01
5eac55f1treatmentEmily Carter✓ visited_menu✗ added_to_cart✗ visited_checkout3:40
35326e0dcontrolAlex Chen✓ visited_menu✓ added_to_cart✓ visited_checkout3:52
0329ec85controlDerek Lawson✓ visited_menu✗ added_to_cart✗ visited_checkout2:36
4418674btreatmentNathan Chen✓ visited_menu✓ added_to_cart✗ visited_checkout4:39
8ab27574controlLisa Morgan✓ visited_menu✗ added_to_cart✗ visited_checkout3:19
6c088003treatmentMichelle Carter✓ visited_menu✓ added_to_cart✓ visited_checkout3:12
7b0fbcaatreatmentMartin Caldwell✓ visited_menu✗ added_to_cart✗ visited_checkout5:33
b31c3206controlLaura Bennett✓ visited_menu✗ added_to_cart✗ visited_checkout2:04
1 / 99
Open Blue Apron’s live demo →
0 hour
Average simulation time per experiment
0+
Simulated user sessions per experiment
0
Experiments runnable in parallel, all without risk or interference
How it works

Use your existing data stack to simulate Blue Apron's real users

Replica uses your existing product and session data to create simulated users, finetune their behavior, and run thousands of browser sessions across control and treatment variants of blueapron.com. In minutes, you get predicted lift, confidence intervals, session replays, transcripts, and behavioral themes before launching the test.

Statsig
Statsig
Amplitude
Amplitude
Optimizely
Optimizely
+50
more
Replica
01
Connect

Replica connects to Blue Apron's analytics, experimentation, session replay, and warehouse tools to create simulated users matched to Blue Apron's real audience. We use user attributes and traffic patterns to define each simulated user, then finetune their behavior on session recordings and action transcripts.

Control
Treatment
02
Simulate

Replica uses these simulated users to run thousands of web sessions across control and treatment variants of blueapron.com in minutes. Each simulated user views, thinks, scrolls, clicks, and types like a real user.

Control
Treatment
+3.4% · 95% CI
03
Decide

Predicted lift and 95% confidence intervals show what changed. Session replays, transcripts, and clustered behavioral themes show why. Ship or skip with quantitative signal and qualitative evidence.

Integrations
StatsigStatsig
AmplitudeAmplitude
OptimizelyOptimizely
MixpanelMixpanel
PostHogPostHog
Google AnalyticsGoogle Analytics
HotjarHotjar
StatsigStatsig
AmplitudeAmplitude
OptimizelyOptimizely
MixpanelMixpanel
PostHogPostHog
Google AnalyticsGoogle Analytics
HotjarHotjar
StatsigStatsig
AmplitudeAmplitude
OptimizelyOptimizely
MixpanelMixpanel
PostHogPostHog
Google AnalyticsGoogle Analytics
HotjarHotjar
SnowflakeSnowflake
DatabricksDatabricks
PostgreSQLPostgreSQL
MongoDBMongoDB
MySQLMySQL
RedisRedis
SQLiteSQLite
Google CloudGoogle Cloud
SnowflakeSnowflake
DatabricksDatabricks
PostgreSQLPostgreSQL
MongoDBMongoDB
MySQLMySQL
RedisRedis
SQLiteSQLite
Google CloudGoogle Cloud
SnowflakeSnowflake
DatabricksDatabricks
PostgreSQLPostgreSQL
MongoDBMongoDB
MySQLMySQL
RedisRedis
SQLiteSQLite
Google CloudGoogle Cloud
Case studies & research

Validated on real experiments and real user behavior

Replica’s case studies show forecasts matching live A/B test outcomes. Our research shows that simulations become more accurate when grounded in real user attributes, behavior data, and finetuning — the core methodology behind Replica.

Case study · Explainpaper
0 / 2
Primary funnel metrics predicted correctly
In under an hour, Replica predicted the same outcome that Explainpaper’s three-week live A/B test later showed across both primary signup-funnel metrics.
Read the case study →
Research · User data
0%
Lower prediction error from realistic user modeling
Using Statsig user data, Replica modeled more realistic simulated users and reduced prediction error — showing that real user context improves forecast accuracy.
Read the paper →
Research · Finetuning
0%
Higher action prediction accuracy
Finetuning on 5,000 real user session recordings helped Replica predict clicks, scrolls, inputs, and drop-offs more accurately than base GPT-4o.
Read the paper →
“The fact that Replica can accurately predict whether A or B is better means we can test extremely rapidly.”
CEO of B2C company, 400k+ users
Behind Replica

Built by experimentation veterans, supported by the best

Part of
Bessemer Beam

See how Replica performs on Blue Apron's past A/B tests

Share past website A/B tests where you already know the outcomes. Replica calibrates its simulations against your experiment history, compares predicted lift to actual lift, and gets Replica ready for production use on future tests.