11 Algolia A/B Testing Gotchas, Tips, and Lessons!!

harlan@harris.name (Harlan Harris) — Mon, 09 Feb 2026 00:00:00 +0000

I've supported A/B testing of Algolia search systems at three companies now, and have dived deep into A/B testing generally as well as specifically for search. The Algolia documentation on search A/B testing is technically adequate for getting started, and the dashboard has improved, but there are still many ways that you can go wrong when A/B testing Algolia search results. In the style of a 2014-era Buzzfeed listicle, here are 11 Algolia A/B Testing Gotchas, Tips, and Lessons!! All of the horrible illustrations are generated by AI, the rest are from earlier posts.

Ab-Testing on Harlan D. Harris

11 Algolia A/B Testing Gotchas, Tips, and Lessons!!