Four times this changed what a company believed
These are published, peer-reviewed and checkable. We include them because they show what the difference between reporting and analysis is actually worth — and because the first one is the reason we take measurement seriously rather than taking a dashboard at face value.
| Company | The question | What the analysis found |
|---|---|---|
| eBayLarge-scale field experiment | Is our paid search advertising actually generating sales, or is it buying customers who were coming anyway? | Brand-keyword ads had no measurable short-term benefit. Across the wider program, returns were a fraction of what the non-experimental reporting showed, and average returns on paid search came out negative.1 |
| NetflixMatrix factorization | How do we recommend accurately when most customers have rated almost nothing? | Latent-factor models outperformed the nearest-neighbor methods the industry had been using, and could absorb implicit behavior and changes in taste over time.2 |
| AmazonItem-to-item collaborative filtering | How do we recommend across an enormous catalog without the computation growing with the customer base? | Restructuring the problem around item similarity rather than user similarity made recommendation scale with the catalog instead of with the number of shoppers.3 |
| eBay, againControlled shutoff | Why did the reporting say something different? | Because clicks and purchase intent are correlated. People who were already going to buy click the ad on the way. Attribution credits the ad; the experiment shows it changed nothing.1 |
The eBay result is the one to sit with. A company with excellent analysts, complete data and a mature attribution system was measuring a return that the experiment showed was not there. Nothing was broken and nobody was careless. The reporting was answering “what happened” and the question was “what did we cause” — and those need different methods.
1. Blake, T., Nosko, C., & Tadelis, S. (2015). Consumer Heterogeneity and Paid Search
Effectiveness: A Large-Scale Field Experiment. Econometrica, 83(1), 155–174.
doi:10.3982/ECTA12423
2. Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix Factorization Techniques for
Recommender Systems. Computer, 42(8), 30–37. doi:10.1109/MC.2009.263
3. Linden, G., Smith, B., & York, J. (2003). Amazon.com Recommendations: Item-to-Item
Collaborative Filtering. IEEE Internet Computing, 7(1), 76–80.
Send us a web address.
We’ll give you a free AI analysis and tell you the three things we’d fix first — in writing, for nothing.