Testing search relevance and bias for an e-commerce platform
The challenge
The platform had added AI-powered search and recommendations. Conversion improved on average, but the team had no way to see where it was failing. Some queries returned irrelevant items, and results skewed for certain customer segments, which is both a revenue and a fairness problem.
Our approach
- Built golden query sets covering head, torso, and long-tail searches.
- Scored relevance against known-good results and measured differences across customer segments.
- Added regression gates so a model or ranking change could not quietly degrade quality.
- Set up ongoing monitoring to catch drift as the catalog and behavior changed.
Search and recommendation models are easy to improve on average and hard to keep honest at the edges. A lift in overall conversion can hide poor results for specific queries or specific segments, and neither shows up without deliberate measurement.
What we did
We built golden query sets across head, torso, and long-tail searches, scored relevance against known-good results, and measured how quality differed across customer segments. Regression gates stop a ranking change from quietly degrading results, and ongoing monitoring catches drift as the catalog and shopper behavior change.
Result
Relevance became a number the team could track rather than a hunch, the quality gaps between segments narrowed, and the evaluation runs on every change so improvements do not come at the cost of a hidden regression.
Representative engagement. Client details are anonymized and figures are illustrative ranges shown to convey the type and breadth of work, not a specific named result.
