SaaStr's AI agent ran a 500,000-email vendor bake-off, and caught a failure the dashboard missed

SaaStr's in-house AI marketing agent spent a week running a head-to-head test of three email vendors across more than 500,000 messages, and in the process caught a delivery failure that its incumbent provider's own status page never flagged. The company's founder, Jason Lemkin, published the results on November 5, crediting the entire test, from design to diagnostics to the migration recommendation, to "10K," the AI system SaaStr built to do the work of its VP of marketing.

What changed

SaaStr ran a matched split test between its incumbent email service provider and a challenger, sending identical content to roughly 2,300 recipients per group at the same time. The challenger pulled 1,168 unique opens (50.5%) and 281 unique clickers (12.2%). The incumbent managed 485 unique opens (21.0%) and 96 unique clickers (4.2%) on a nearly identical number of recorded deliveries. That is roughly 2.4 times the opens and 2.9 times the clicks for the challenger on the same list, same message, same minute.

The more consequential finding showed up on a separate daily send. The incumbent's workflow dashboard marked the job "Completed." Step-level logs told a different story: 80,708 send attempts, 80,708 errors, zero emails delivered. Four follow-up diagnostic sends, each changing a single variable, all failed the same way, pointing to a problem in the vendor's transport layer rather than SaaStr's configuration. The vendor's own status page stayed green throughout.

Who this affects

Most B2B software companies run their newsletter, product updates and nurture sequences through exactly this kind of vendor, and most of them take "Completed" at face value. SaaStr's test is a reminder that a dashboard status is a claim, not a receipt. The company scored vendors on seven criteria rather than one: confirmed delivery at the step level, inbox placement, spam complaints, unsubscribes, human clicks with scanner traffic filtered out, how accurately the vendor's own reporting matched reality, and how much human time the whole thing took to operate.

That last criterion is the part aimed squarely at marketing leaders evaluating AI tools rather than email vendors. Lemkin has documented 10K's build-out in a running series on SaaStr since earlier this year: the system plans campaigns, works SaaStr's Salesforce pipeline, and analyzes five-plus years of historical data, alongside a human VP of marketing whose output it is explicitly measured against. He has put the fully loaded cost of a comparable human VP of marketing at $300,000 to $400,000 or more a year. This bake-off is presented as evidence of what that system can now do unsupervised: design a test, run it at production volume, catch a vendor's silent outage, and write the switching plan, in what Lemkin estimates would otherwise take a consultant roughly a quarter to complete.

What a software marketer should do differently

The specific vendors in SaaStr's test are not named, so this isn't a buying recommendation for any particular ESP. The transferable part is the method, and the posture behind it.

• Stop trusting vendor status pages as proof of delivery. Ask for step-level send logs, or pull your own, before accepting that a campaign actually went out.
• Run new and incumbent vendors in parallel on matched, real traffic before migrating fully, rather than switching on a sales pitch or a single bad week.
• If an ESP has ever shown "Completed" on a send you can't verify independently, treat that as a trust problem with the vendor's reporting, not a one-off.
• Score vendor performance on operator time, not just open and click rates. A vendor that is 10% cheaper but eats a day a week of your team's attention is not actually cheaper.

80,708 send attempts, 80,708 errors, zero emails delivered — and the vendor's status page stayed green the entire time.

SaaStr's own ramp plan is instructive for anyone moving a list of meaningful size: shift roughly 20,000 contacts to a new vendor first, then 40,000, then full volume, advancing only if bounce rates, spam complaints and unsubscribes hold steady at each step. The company is also requiring a documented root-cause analysis and a confirmed step-level test from its outgoing vendor before it will let that vendor touch another sponsor-backed send.

The underlying argument Lemkin makes is less about email than about where AI agents are already doing real operational work in marketing: not writing copy, but auditing vendors, reconciling logs and building the case for a switch, jobs that used to sit with a consultant or get skipped because nobody had the week to spare. For a marketing team deciding whether an AI agent belongs anywhere near its martech stack, a deliverability audit that catches a six-figure-impact outage missed by the vendor's own monitoring is a concrete place to start, whether or not the agent doing it was built in-house.