A Big Win from a Small Search UI Change and a Predictive Model

One of my biggest commercial wins was a seemingly small UI tweak that led to a roughly 1.5% marketplace revenue bump. Teachers Pay Teachers (TPT, think "Etsy for Teachers" -- educational resources created by and sold to other educators) was a mature marketplace, where revenue wins from search system changes were typically under 0.5%. How we got this win is, I think, instructive, reinforcing the value of a search scientist, and of having a holistic view of search UI/UX and algorithms.

Ratings and reviews are important proof of value to users searching on e-commerce sites. In 2020, the resource cards on the search results page looked like this -- the average historical rating, shown as partial stars, followed by the number of ratings in bold text.

The Problem

We knew from user interviews that buyers anchored on the number of ratings as a key signal of value, and that the partial star format made it hard to compare resources by the actual rating average. The number of ratings also was a component of search ranking -- everything else being equal, resources with more ratings were ranked higher.

At some level this is reasonable. Historically popular resources were generally good. But this pair of design features -- a UI design that focused on rating count, and an algorithm that reinforced that -- had serious down-sides. First, it clearly penalized new resources, which may have been excellent, simply for being new. Second, it meant that resources whose first few ratings were poor, for whatever reason, were basically doomed. Nobody will purchase an item with "⭐️⭐️ (2)" next to it.

The Approach

So, we on the Search team set about trying to solve these problems. The goal was multi-part:

• Buyers should anchor on how good the resource likely is, not how historically popular it has been.
• Sellers should not be penalized for bad luck on their initial ratings.
• The UI should be clear and informative.

The solution we landed on had several components. Critically, all of those components had to be deployed together. Let me walk you through them:

• On the back end, a statistical model estimated the likely long-term average rating for this resource, based on the data accumulated so far. (More on this below.)
• On the front end, resource ratings were shown numerically, with the historical average rating in bold and the less-important count now shown in grey.
• In the search engine, the likely next rating replaced the rating count in the scoring algorithm.

Now, the page looks like this:

Note that, as a buyer, you're now likely to focus on two signals of quality -- the highlighted average rating, and the search ranking, which reflects the estimated long-term average rating. This is a better signal of quality. Users now see more items with high ratings, even if they're newer and haven't had the opportunity to accumulate hundreds of ratings yet.

The Algorithm

Consider two resources. Resource 1 has 200 ratings, averaging 4.7 stars, and is from a seller whose resources average 4.6 stars. Resource 2 only has two ratings, averaging 4.0 stars. But this resource is from a seller whose many resources average 4.8 stars.

How should this information be put together to construct an estimate of the long-term rating average, assuming many people will eventually purchase and rate the item?

The solution is Bayesian statistics. The intuition is not too complicated -- if you don't have much (or any) data, "borrow" some data from other similar cases until additional data comes in. So, for Resource 2, you can supplement the existing two ratings with ratings of other resources by the same seller, or if necessary, from other sellers, to get a conservative estimate of the eventual rating.

Here's the core calculation. Each resource has an actual number of ratings and an actual average rating. Each is also linked to a seller whose other resources have their own average rating -- this is the "prior" belief about a typical rating for this seller, before we've seen much data on this particular resource.

I'm papering over a number of technical details, but assume that we've established that we want to borrow 50 ratings from the seller's other resources. The fewer real ratings a resource has, the more its estimate will be pulled toward the seller's average. This number is tuned as part of the model-fitting process, but is static in production.

The expected value of the eventual rating average is just a weighted average of the resource's actual ratings and the borrowed, seller-level ratings, weighted by how many ratings each group contributes:

1Resource 1:  (200 × 4.7 + 50 × 4.6) / (200 + 50) = 4.68
2Resource 2:  (  2 × 4.0 + 50 × 4.8) / (  2 + 50) = 4.77

Resource 2 has an expected value that's quite different from its 4.0 average, while Resource 1, with many more actual ratings, changes little.

 1resources = [
 2    {"name": "Resource 1", "n_ratings": 200, "actual_rating": 4.7, "seller_rating": 4.6},
 3    {"name": "Resource 2", "n_ratings": 2,   "actual_rating": 4.0, "seller_rating": 4.8},
 4]
 5
 6# Borrow this many imaginary ratings from the seller's other resources.
 7borrowed_n = 50
 8
 9for r in resources:
10    # Posterior (Bayesian) estimate: a weighted average of the
11    # resource's actual ratings and the borrowed seller-level ratings.
12    r["posterior_rating"] = (
13        r["n_ratings"] * r["actual_rating"] + borrowed_n * r["seller_rating"]
14    ) / (r["n_ratings"] + borrowed_n)
15
16for r in resources:
17    print(f'{r["name"]}: {r["posterior_rating"]:.2f}')

Tuning this algorithm to work properly with 1-5 star ratings, handling sellers with few other ratings, and other edge cases, requires careful work by a statistically-savvy search scientist. Once the algorithm is determined and tuned, computing these values nightly and incorporating them into the search algorithm requires some data pipelines to be set up but is not particularly difficult.

The Impact

We ran an A/B test of these changes, and saw a roughly 1.5% increase in sales during the rollout, with neutral or positive impact on other metrics. The rollout required careful communication from the product marketing team, as some sellers saw shifts in how often their historically popular resources appeared, but we were able to clarify the situation and complete the rollout.

This case study shows how critical it is to treat search design and algorithms holistically. Changing the algorithm without the UI would have confused users -- items with large bold rating counts would seem wrongly ranked for no visible reason. Changing the UI alone would have been equally problematic, highlighting a noisy metric that penalized bad luck. Together, the changes directed users to a useful signal, encouraging them to click through and make the qualitative evaluation critical for purchase.

Notes

• It's been a few years since these changes were made. I may have mis-recalled some details. Other aspects of the page design have subsequently evolved, and I have no information as to the current algorithm.
• Statistical note -- The algorithm was an empirical Bayes shrinkage model with a normal approximation to the ordinal data. Offline back-testing allowed computation of variances and the shrinkage parameter. Production code was only slightly more complex than what's shown here.
• This article was primarily authored by me, with secondary contributions from AI systems. I wrote almost all of the text. The AI generated the R code (which I reviewed) and the initial version of the diagram, and made a number of suggestions and copy-edits. Thanks to (human, and cross-functional expert in user-facing systems) Rito for very helpful comments on a draft of this post.
• Are you looking for support with search and discovery on your company's website? Does this post make you think I could help your company make great design or technology decisions? I'm a freelance consultant with extensive experience -- please reach out!