Costs
Part of Email marketing analytics: what to keep and what to drop
Email marketing analytics: why the numbers move and what they hide
This article explains how email analytics can mislead, covering Apple open rates, bot clicks, attribution windows, GA4 joins, and checks that keep reports honest.
What to take away
- Each email marketing analytics case study below names a real company or platform, the published result, and the condition that produced it.
- A metric that moves without a matching business outcome is a definition problem before it is a performance problem.
- Apple Mail Privacy Protection inflates opens by prefetching remote content, so open rate cannot carry engagement decisions alone.
- Attribution windows and join cardinality change reported revenue without changing collected revenue.
- Keep raw events, version every definition, and restate affected reports in writing.
Apple Mail and the open rate that stopped meaning anything
Litmus publishes an email client market share report built from opens across its own user base. Apple Mail sits at or near the top of that report, ahead of Gmail, and the climb starts in September 2021, when iOS 15 shipped Mail Privacy Protection.
Apple Mail open rate shift
- September 2021iOS 15 ships Mail Privacy Protection
- AfterApple Mail climbs to top of Litmus report
- EffectOpens register at delivery, not reading
- ResultOpen rate rises while clicks stay flat
The condition behind that rise: the feature downloads remote content whether or not a person reads the message, so an open registers at delivery. A list can post a rising open rate while clicks, replies and purchases stay flat.
Fix the definition before you touch the campaign. Stop using opens to prune unengaged subscribers. Segment on clicks, site visits or purchase recency instead. For the wider measurement picture, see this guide to email marketing analytics.
Twilio SendGrid and the clicks that arrive in seconds
Twilio's Event Webhook reference documents the event types SendGrid emits, including a machine-open indicator, and warns that event payloads can contain personal data. It does not label every scanner that requests your links.
Why instant clicks are not people
- Message delivered to inbox
- Mail security fetches every URL in seconds
- Fixed order, destination page never opened
- Real HTTP requests logged as clicks
- Filter raw events, validate against controlled send
The requests usually come from enterprise mail security. Microsoft Defender for Office 365, Proofpoint URL Defense, Mimecast Targeted Threat Protection, Barracuda and Cisco Secure Email all fetch links at delivery.
The condition that produces a click spike: those tools pull every URL in the message within seconds, in a fixed order, and never open the destination page. The clicks are real HTTP requests. They are not people.
Keep the raw event stream. Build a separate filtered view, define the filter in writing, and validate it against a controlled send to known human and known automated addresses. Report the filter's uncertainty rather than presenting the filtered number as truth.
Klaviyo attribution windows and the revenue that doubled
Klaviyo's attribution documentation sets a default window for email and lets accounts change it. Extend that window during a seasonal promotion and attributed revenue rises while the order count does not move.
Attributed revenue is a model output. Finance-reconciled payments are the observed number. The gap between the two is the incrementality question, and only a holdout answers it. Klaviyo supports holdout groups on flows, which lets you compare a group that receives the flow against one that does not.
Version the attribution setting, restore comparable history before you compare periods, and report attributed, observed and incremental value as three separate lines. Treat the copy in those messages as a separate discipline; email copywriting trends covers what changes response.
GA4 joins and the order count that multiplied
Google's GA4 Measurement Protocol reference documents these areas:
- HTTPS requests, identifiers, timestamps, events and parameters
- consent fields, and validation behavior
A valid request confirms the event was accepted. It says nothing about how that event joins to your order table.
The condition: several messages and several order items joined on account key produce a many-to-many table, and each order value repeats once per message. The dashboard shows more orders than the warehouse.
Aggregate to one row per order before joining, test cardinality on a sample, and reconcile the combined metric against the source system before publishing it. Label every restated report and list the decisions that were made from the faulty view.
The Obama 2008 list test and the reported $2.5 million gain
The 2008 Obama campaign split-tested subject lines on a slice of a 13-million-name list, then sent the winner to everyone else. Campaign staff have put the result of that one test at about $2.5 million in donations.
Three conditions made the number readable: a random slice, one ask, and a control group large enough for the difference to show. A test run on a hand-picked segment measures the segment, not the list.
Ronny Kohavi, who led experimentation at Microsoft and at Airbnb, has published the estimate that roughly one in three tested ideas improves the metric it targets. Plan for the two thirds that do not.
Google and Yahoo sender rules and the 0.10% complaint line
Google's Email Sender Guidelines and Yahoo's matching bulk sender rules took effect in February 2024. Gmail asks bulk senders to keep the spam complaint rate shown in Postmaster Tools under 0.10%, and never at 0.3% or above.
Those rules also require one-click unsubscribe and DMARC authentication from bulk senders. Free reputation data sits in Google Postmaster Tools and Microsoft SNDS, so a complaint spike is visible before it costs you the inbox.
Case investigation record
| Signal | First test | Decision response |
|---|---|---|
| Open jump after iOS 15 | Client mix in the Litmus share report, privacy settings | Relabel or suspend the trend |
| Instant SendGrid clicks | Timing, sequence, user agent, scanner list | Separate raw from filtered |
| Klaviyo revenue jump | Attribution setting and window | Restate model output |
| GA4 order multiplication | Join cardinality and keys | Repair and rerun |
| Gmail complaints near 0.3% | Postmaster Tools and SNDS reputation pages | Pause, clean the list, resume |
Audit the published evidence
The FTC advertising substantiation policy requires a reasonable basis for objective advertising claims before they are disseminated. That applies to any performance statement you publish about a named client or platform, and the standard depends on the claim. Ask qualified counsel about your own facts.
The FTC guidance for marketers using reviews covers fake feedback, selective requests, conditioned incentives and undisclosed relationships. Apply the same integrity checks when a case study names a provider. More worked examples sit in these email automation case studies.
Vendor benchmark studies from Mailchimp, Klaviyo and Omnisend publish median open, click and conversion rates by sector. A single campaign's headline lift means little next to those medians.
Common questions
Why did my open rate rise while sales stayed flat?
Apple Mail Privacy Protection downloads remote content on delivery, so opens register without a person reading anything. Check your client mix and clicks before treating a rise as engagement.
How do I tell a bot click from a human click?
Look at timing and sequence. Automated fetches arrive seconds after delivery, in a consistent order, and never reach the destination page. Validate your filter with a controlled send.
Why does my dashboard show more orders than my store?
A many-to-many join between messages and order items repeats each order value. Aggregate to one row per order before joining, then reconcile against the source system. For more of these recurring measurement traps, work through these email marketing analytics questions before you trust the next dashboard.
What should I keep when a metric turns out to be wrong?
Keep raw events, query versions and definitions. Also keep dashboard snapshots, settings and change history. Record which decisions were made from the faulty view and who was told about the correction.
