You're running a cold email campaign. The open rate is 32%. The reply rate is 4.2%. So what?
That's the question most people skip. They look at their metrics, feel good or bad about them, and move on. But they never ask - is this actually working? Should I change something? If so, what?
The "so what test" is the framework that bridges the gap between data and action. It takes your metrics and turns them into decisions.
What the So What Test Actually Is
The so what test is simple: for every metric you're tracking, you answer three questions.
- Is this metric good, bad, or average for my context?
- If it's not where I need it to be, what's the root cause?
- What specific change will move it?
Most people stop after question one. They see a 4% reply rate and think "that's okay." They don't dig into whether it's okay for their specific situation, and they definitely don't connect it to an actual fix.
Establishing Your Baseline: What Normal Actually Means
Before you can test anything, you need a benchmark for your industry and audience.
Here's what we've seen work at scale across service businesses and agencies:
- Open rate: 30-45% is normal for warm cold email. If you're at 15-25%, your subject lines or sender reputation are the problem. If you're below 15%, check your deliverability first - you might not be landing in inboxes at all.
- Reply rate: 3-7% is standard for most B2B service businesses. Below 2%? Your email copy or targeting is off. Above 8%? You're probably hitting warmer leads or have a strong hook.
- Conversion rate (calls booked to actual clients signed): 15-25% of replies that turn into meetings typically become clients. This varies wildly by your close rate and offer.
- Cost per lead: If you're sending 500 emails per month and getting 15 replies, your cost per lead (if you value your time at $50/hour for 2 hours of campaign setup and management) is roughly $7 per reply.
These aren't universal rules - they're anchors. Your baseline depends on your niche, your list quality, and how warm your outreach is. But now you have something to compare against.
The So What Test in Practice
Scenario 1: Open Rate Problem
You're getting 22% open rate. Your benchmark is 35%.
So what? People aren't opening your emails. Why? Two root causes: either your subject line isn't compelling enough, or your emails aren't reaching inboxes. Test deliverability first - check if you're landing in spam. If deliverability is fine, then you test subject lines.
Here's what a subject line test looks like - not vague tweaks, but specific variations based on different hooks:
Subject A: Quick thought on [Company Name]'s content strategy Subject B: Saw your post on LinkedIn about [specific topic] Subject C: 3-minute read: how [competitor] is getting 2x the engagement
Send 100 emails with each. Measure opens. The winner typically beats losers by 5-15 percentage points. Now you have direction.
Scenario 2: Reply Rate Problem
You're getting 40% open rate (good). But only 2% reply rate (bad). The problem isn't getting people to open - it's getting them to respond.
So what? Your email body isn't compelling enough, your ask is unclear, or you're not hitting a real problem they care about. This is usually a copy problem, not a targeting problem.
Test a different email structure. Instead of asking for a meeting, try opening with a specific insight about their business:
Hey [Name], I was looking at your site and noticed you're getting traffic but your demo signup flow has 3 friction points that are probably costing conversions. We helped [similar company] redesign theirs and added 40 more qualified demos per month. Want to see what's actually happening on yours? [Your name]
This works better than generic openers because it does three things: shows you looked at their specific situation, makes a claim tied to a metric, and asks a yes/no question instead of "let's hop on a call."
Scenario 3: Wrong Audience
Your open rate and reply rate look fine. But the replies you're getting aren't qualified. People are responding but they're not the decision makers, or they're not your ICP.
So what? Your list is the problem. You can have perfect copy with the wrong audience and nothing happens. Go back and audit your lead generation process - are you targeting the right titles? The right company size? The right pain point?
Building Your Testing Matrix
Here's how to structure testing so you actually learn something:
Week 1-2: Establish baseline across 200-300 emails
Run your current campaign as-is. Collect open rates, reply rates, and the quality of replies. This is your control.
Week 3-4: Test one variable
Change only one thing. If you're testing subject lines, keep your email body identical. If you're testing copy, keep subject lines and audience the same. Send another 200-300 emails.
Week 5: Compare and decide
Did the new variable beat your baseline by at least 15-20%? If yes, keep it. If no, revert and test something else next cycle.
This matters: don't test three things at once. You won't know which variable moved the needle, and you'll waste time guessing.
The Metrics That Actually Matter
Not all metrics deserve your attention. Focus on these:
- Qualified reply rate: Replies from people who fit your ICP and have budget. This is 10x more useful than total reply rate.
- Meeting-to-client conversion rate: Of the people who say yes to a call, how many become clients? This tells you if your offer actually solves their problem.
- Time to reply: If replies are coming back in 4 hours, they're engaged. If it takes 2 weeks, they might be interested but not urgent. This affects your follow-up strategy.
- Unsubscribe rate: If this is above 0.5%, your messaging is off-target or your list quality is poor.
Common So What Test Mistakes
Testing without a control. You change your subject line but don't track what the old one was doing. Now you can't measure improvement. Always keep one thing constant.
Testing too small. You send 20 emails with a new subject line, get 1 open, and declare victory. That's noise, not data. Send at least 100-150 per variation.
Testing the wrong variable. Your reply rate is bad, so you change your subject line. But subject lines affect opens, not replies. You tested the wrong thing. Diagnose first, then test.
Not implementing winners. You discover a subject line that performs 25% better, nod, and never use it again. Test is only valuable if you actually deploy what wins.
When to Use This Framework
Run a so what test every 2-4 weeks once you have baseline data. Each cycle should improve one metric by 15-20%. Over 3 months, that compounds - a 20% improvement in reply rate compounds into 73% better performance by month 3.
Stop testing when you hit your target metrics. For most service businesses, that's a 4-6% qualified reply rate and 20%+ meeting-to-client conversion.
Where This Breaks Down
The so what test works great if you're already set up for cold email success - your infrastructure is solid, your list quality is high, and you have basic copy that works. If your infrastructure isn't right, no amount of copy testing will fix it. If your list is full of bad emails, testing subject lines is pointless.
The testing framework also assumes you have volume - you need at least 100-150 emails per test to see real patterns. If you're sending 20 emails a week total, you won't get clean data.
And frankly, running this testing cycle month after month while also managing your business is a slog. It's the difference between knowing what to do and actually having a system that does it consistently. That's where most people get stuck - they understand the framework but don't have the infrastructure, the lead source, the copy variations, or the discipline to run it properly. That's also where a team that specializes in cold email at scale becomes valuable.