A/B Testing Copy

Data Beats Opinions

🔄 Quick Recall: In the previous lesson, we adapted copy for different social media platforms—LinkedIn, Twitter/X, Instagram, and TikTok, each with unique constraints and audiences. Now we’ll learn to test which copy actually performs best, because the audience always has the final vote.

You think your headline is great. Your colleague prefers a different one. Your boss wants something else entirely.

Who’s right? None of you. The audience decides. And A/B testing is how you let them.

By the end of this lesson, you’ll design, run, and interpret A/B tests that systematically improve your copy performance.

What Is A/B Testing?

A/B testing (split testing) means showing two versions of copy to similar audiences and measuring which version performs better.

Version A: Your current copy (the “control”) Version B: A variation with one change (the “challenger”)

Traffic is split randomly between the two versions. After enough data, you compare the results.

If B wins: B becomes the new control. Now test a new variation against it. If A wins: Try a different approach for the next test.

What to Test (Priority Order)

Not all tests are equal. Test the highest-leverage elements first:

Tier 1: Highest Impact

1. Headlines — Changes the first-impression experience
2. Value proposition — Changes what visitors think they’ll get
3. Core offer — Changes the fundamental exchange (pricing, trial length, etc.)
4. CTA text — Changes the conversion moment

Tier 2: Significant Impact

5. Social proof — Type, placement, specificity
6. Email subject lines — Determines open rates
7. Page structure — Section order, content length
8. Images/visuals — Supporting imagery

Tier 3: Incremental Impact

9. CTA button color/size — Visual prominence
10. Testimonial selection — Which quotes to feature
11. Microcopy — Form labels, error messages, help text
12. Font and formatting — Readability improvements

✅ Quick Check: Why would testing a headline produce a bigger impact than testing a button color?

Start with Tier 1. A 20% headline improvement affects everything downstream. A 2% button color improvement affects only the click moment.

How to Run a Test

Step 1: Form a Hypothesis

Template: “If I change [element] from [current] to [variation], then [metric] will improve because [reasoning].”

Example: “If I change the headline from ‘Project Management Software’ to ‘Finish Projects 2x Faster,’ then signups will increase because the benefit is more compelling than the category description.”

Step 2: Create Your Variation

Change only one element. Everything else stays identical.

Testing a headline? Same page, same CTA, same images. Only the headline differs.

Testing a CTA? Same headline, same body copy, same page. Only the CTA button text differs.

Step 3: Split Traffic

Most testing tools (Google Optimize, Optimizely, VWO, simple email platform split tests) handle this automatically. Traffic is randomly divided between versions.

Requirements:

• Random assignment (not “morning gets A, afternoon gets B”)
• Equal split (50/50 is standard)
• Same time period (don’t compare Monday to Sunday)

Step 4: Wait for Significance

This is where most people fail. They check results after 24 hours and declare a winner.

Statistical significance means the difference between A and B is unlikely to be due to random chance.

Rules of thumb:

• Run tests for at least 1-2 full weeks (captures day-of-week variation)
• Need at least 100 conversions per variation (more for smaller differences)
• Use a significance calculator (most tools have built-in ones)
• 95% confidence is the standard threshold

Step 5: Analyze and Document

Record in your testing log:

Common Testing Mistakes

Testing too many things at once. You changed the headline, CTA, and hero image. Conversion went up 15%. What caused it? You’ll never know.

Ending tests too early. After 50 visitors, version B is up 40%. You declare victory. Next week, with 500 visitors, it’s a statistical tie. Wait for significance.

Testing trivial changes. The difference between “Get Started” and “Get Started Now” is rarely worth testing. Focus on substantial variations.

Not documenting results. Running the same losing test three months later because nobody recorded the first result.

Ignoring segments. Version A might win overall but lose with your most valuable segment. Look at results by audience when possible.

What to Test by Channel

Landing Pages

Emails

Social Media

Building a Testing Culture

The best copywriters test continuously. Build the habit:

1. Always be testing. There should always be one test running.
2. Start with big swings. Test radically different approaches, not tiny tweaks.
3. Let data win. When your gut disagrees with the data, trust the data.
4. Share results. Testing insights benefit the entire team.
5. Compound improvements. A 10% improvement per month is a 3x improvement per year.

Try It Yourself

Design an A/B test for a piece of copy:

1. Choose an element (headline, CTA, subject line)
2. Write your hypothesis
3. Write the control and variation
4. Define what metric you’ll measure
5. Determine how long you’d run the test

Key Takeaways

• A/B testing lets the audience decide which copy works—not opinions
• Change only one element per test to isolate the cause of any improvement
• Test high-impact elements first: headlines, value propositions, offers, CTAs
• Wait for statistical significance before declaring a winner (minimum 1-2 weeks)
• Document every test in a log to build institutional knowledge
• Continuous testing compounds: small improvements add up to transformative results

Up Next

In Lesson 8: Capstone, you’ll bring everything together by creating a complete copy campaign—from headlines to landing page to email sequence—for a real or hypothetical product.