You're looking at cold email agencies right now because you're tired of your pipeline looking like a ghost town. Either you've tried building this in-house and it fell apart, or you know it's not your core competency and you want someone else to handle it.
Here's what I need to tell you first: most "cold email agencies" aren't actually cold email agencies. They're email blast shops. They'll send 10,000 emails with a 1% reply rate and call it a win. Or they'll "optimize" your existing list and deliver basically nothing.
The real problem is evaluating them. You don't have time to run a six-month test with every agency that pitches you. So here's how to separate the ones that actually know what they're doing from the ones that will waste your money.
What a Real Cold Email Agency Actually Does (Not What They Say They Do)
First, filter for what the work actually includes. A cold email agency that's worth your time doesn't just send emails and ghost you.
Real cold email work includes:
- Building infrastructure (warm-up, authentication, sending accounts) - this takes 2-4 weeks and most agencies skip it because it's boring
- Research and lead list building specific to your business model - not buying a list, actually vetting who you're sending to
- Copy that's written for YOUR offer, YOUR market, YOUR conversion path - not templates they've used 50 times
- Managing replies and qualification - this is where the actual work happens, and most agencies treat it like a checkbox
- Campaign iteration based on real response data - not just firing and forgetting
When an agency pitches you, ask them straight: "Who handles the replies?" If they say "you do" or "we have a portal where you manage it," they're not running a full service operation. They're running a sending operation.
The Numbers That Actually Matter
Ignore response rate percentages as a selling point. They're meaningless without context. Instead, ask for these:
Cost per qualified conversation: This is what you actually care about. If an agency gets you 10 qualified meetings per month and charges $2,000, your cost per meeting is $200. That's the number that matters to your P&L.
Time to first reply: A good agency should have replies coming in by week 2-3 of a campaign. If they're promising results in week 1, they're either cherry-picking results or they're lying. If they're not showing replies until week 4-5, their setup is bad.
Reply-to-meeting conversion: This depends entirely on your service, but if an agency is converting 5-10% of replies into qualified meetings, that's solid. Below 2% means their qualification framework is weak.
Clients retained past month 4: Ask directly. If an agency won't tell you their client retention rate past the first four months, it's because it's low. The best agencies will be confident sharing this number because it's usually 60%+.
Red Flags in How They Present Their Work
Some specific things that tell you an agency doesn't actually understand cold email:
Heavy emphasis on personalization mechanics. If they spend most of their pitch talking about merge tags and "how personalized our emails are," they're missing the core insight. Personalization matters only if the offer matters. A perfectly personalized email about the wrong thing still gets deleted.
Guaranteed response rates. Real work has variation. Market conditions, seasonality, offer fit - these change results by 30-50% month to month. Anyone guaranteeing you a specific response rate is overselling.
No mention of list quality. If their pitch is all about "scale" and "volume," they don't understand cold email. The best cold email agencies are actually selective about who they mail. Sending 5,000 emails to the right people beats 50,000 to the wrong people.
Vague copy examples. Ask them for actual subject lines and opening lines they've used. Not templates. Actual ones they sent. If they give you corporate-sounding examples, that's a signal they don't test enough or they don't have real results to show you.
How to Structure a Real Evaluation
If you're comparing agencies, run this test:
Ask each one to write 5 subject lines for your specific offer. Not frameworks. Actual subject lines. Don't tell them your target audience first - see if they even ask. The ones that dig into who your buyer is before writing copy are the ones that understand the work.
Here's an example of what you're looking for. If you're a staffing agency selling managed hiring to mid-market SaaS companies, a real subject line sounds like this:
Pipeline dried up overnight? We placed 8 engineers at [Company] last quarter
That's specific. It references an outcome, not a feature. It acknowledges a real problem. Compare that to generic subject lines like "Quick question about your hiring" and you'll immediately see the difference.
Ask them to explain why each line works. If they explain it in terms of open rate psychology or curiosity gaps, they're thinking about metrics. If they explain it in terms of what actually matters to your buyer, they understand the market.
The Qualification Question That Separates Them
Here's the question to ask every agency: "What's your framework for deciding which replies to pursue and which to kill?"
A good answer sounds like this: "We look at budget indicators in their responses, whether they mentioned current vendors, and whether the problem they're having actually fits your service. We disqualify anyone asking for a proposal immediately, anyone who says 'send me info,' and anyone who's clearly just collecting vendors."
A bad answer is vague: "We qualify based on fit" or "We focus on engagement."
The ones with clear qualification criteria are running real sales operations. The ones that are vague are just forwarding replies to you and hoping you figure it out.
What to Ask About Their Process, Specifically
Walk through their actual process for your account. Ask:
- "How many people will be working on this, and what does each role do?" (If it's one person doing everything, you're getting commodity service.)
- "How do you decide which 5,000 people to target in month 1?" (The answer should involve research, not just demographic filters.)
- "What happens in week 1 before any emails go out?" (This is where they set up infrastructure. This matters.)
- "How often do you change the copy, and based on what data?" (Weekly is good. Monthly is okay. If they say "we don't change it unless you ask," that's lazy.)
- "What's the average deal size of clients you work with?" (If it's wildly different from yours, they may not understand your market.)
The agencies that have clear answers to these questions have run enough campaigns that they've built actual systems. The ones that fumble are winging it.
One More Practical Filter
Ask for a reference from a client in your exact industry. Not a reference from any client - from your industry. If they can't provide one, that's not a deal-breaker, but it means you're taking more risk.
When you call the reference, ask: "What percentage of replies did you actually want to take meetings with? Did the quality match what they promised?"
That answer will tell you everything.
Understanding the Gap Between Knowing This and Running It
Reading this, you now know what to look for and how to evaluate an agency. The gap between knowing this and actually having a cold email operation running at scale is mostly logistics - infrastructure setup, ongoing list research, daily reply management, copy testing cycles, and holding the entire operation accountable to real metrics week after week.
If you have the bandwidth to manage an agency well and the infrastructure to handle everything they hand off, you can do this with a contractor or a smaller shop. Most service businesses and agencies don't. They need a partner who handles the entire operation end-to-end and takes ownership of the results, not just the activity.
That's the actual differentiation between agencies worth your time and the rest.