Submission + - Klaviyo's Built-In Test of What Email Really Earns Needs 400,000 Profiles (liftsignals.com)
Opens make the problem worse. Apple Mail Privacy Protection can register opens that no human made. Klaviyo's attribution settings include an option to remove those opens from attribution, though Klaviyo notes they still appear in reporting. Even for accounts that qualify, a global holdout measures the whole email and SMS program, and Klaviyo's guide doesn't describe a way to hold out a single flow. In the article's worked example, which uses an illustrative store, attributed revenue comes to about 3.5 times what the flow actually caused. The article also argues that the 10% holdout many marketers pick "because it feels safe" is often too small to detect anything. From the article:
"So if a customer opens your abandoned checkout email and buys within five days, the email gets the sale. It gets the sale if they were always going to come back after payday. It gets the sale if they only opened it to delete it."
The guide points out that this is how any last-touch model hands out credit, not something peculiar to Klaviyo. Has anyone here run a true holdout on their marketing email, and how far was the result from the dashboard number?