Running your first A/B experiment: A founder's guide

Every fast-growing SaaS company you admire runs experiments constantly. Duolingo runs hundreds per week. Notion runs them on every major feature launch. Even small teams at companies like Linear experiment relentlessly with onboarding flows.

Experimentation is how you replace opinions with evidence. Here's how to run your first one properly.

Step 1: Start With a Clear Hypothesis

A bad hypothesis: "Let's try a different headline."

A good hypothesis: "We believe that changing the CTA from 'Start Free Trial' to 'See It In Action' will increase click-through rate by 15% because our user interviews suggest people are skeptical about setup complexity."

The format: We believe [change] will cause [outcome] because [reasoning].

Step 2: Define Your Success Metric Before You Start

Decide what you're measuring before you run the test. Changing the metric mid-experiment is one of the most common ways founders fool themselves into seeing results that aren't there.

Step 3: Calculate Your Required Sample Size

This is where most first-time experimenters go wrong. You need enough traffic to reach statistical significance. Use a sample size calculator — for most SaaS conversion experiments, you'll need at least 500 users per variant.

Step 4: Run the Experiment — Don't Peek

Checking results daily and stopping when you see something you like is called p-hacking. Run your experiment until you hit your pre-defined sample size, then read the results.

Step 5: Document Everything

Whether the experiment wins or loses, write it up. Your experiment log is one of the most valuable assets your company has — it prevents you from running the same failed test twice.

Common Mistakes to Avoid

• Testing too many variables at once — change one thing at a time

• Running tests during unusual periods — avoid holidays, product launches, or PR spikes

• Ignoring segment differences — a test that wins overall might lose badly for your best customers

GrowthLab has a built-in experiment runner that handles traffic splitting, significance calculations, and result logging automatically. Launch your first experiment in minutes — no engineering required.