Background

Quince is an online retail company that democratizes luxury goods. It sells them directly to consumers at affordable prices. They recently raised a $50 million series A investment to accelerate their growth.

Problem

As Quince's business was scaling, they wanted to be aware of any structural changes in revenue and conversion rates. It was a challenge to look at all cuts of data since there were millions of possible ways to slice and dice. Revenue and conversion rates could be broken down by any of the following dimensions:

Business_department: e.g., men's, women's

Marketing_product: e.g., apparel, bedding

Product_category: e.g., apparel, bedding.

Product_franchise: e.g., silk apparel.

Product_gender: e.g., male, female.

Product_title: e.g., washable stretch silk tie.

Traffic_medium. e.g.,: google, facebook, email, affiliate.

Quince wanted to be:

Alerted when conversion rate across traffic changes (conversion is defined as the ratio of checkouts to views, expressed as a percentage). They wanted these alerts not just on the total but across all significant traffic sources.

Alerted when any segment sees a big change in checkouts.

Solution

The team at Quince used BoostKPI to solve their problem. First, they connected their data warehouse to BoostKPI. Doing so enabled BoostKPI to pull daily checkouts and product_views, broken down by the (above) dimensions. Next, they added conversion rate as a derived KPI as the ratio of checkouts to views.

Alerting on Conversion Rate

With BoostKPI's smart alerting, it was straightforward to solve the first scenario. The conversion_rate was monitored as a function of the traffic_medium. Here is an alert that was sent when the conversion rates on the email traffic_medium dipped. Note that the alert configuration remains the same even as new traffic_mediums are added or old traffic_mediums are removed.

Alerting on Checkout Changes

With 7 possible dimensions to choose from, and hundreds of thousands of segments across the possible values of these dimensions, setting up an alert to look at all possible segments was challenging. For example, even enumerating all possible pairs (of dimensions) leads to a permutational explosion – with 7 dimensions, 42 alerts need to be configured.

To solve this problem, the BoostKPI team turbo-charged their smart alerting so that the dimension list did not need to be specified. It automatically analyzed the data to figure out the dimensions along which the data should be sliced-and-diced.

Here is an example alert that was fired, with all the sensitive numbers redacted. This alert was sent over both email as well as slack.

Other use cases:

In addition to the above use cases, there were other situations where BoostKPI was able to help.

Highlighting a data quality issue. For an earlier dataset, BoostKPI quickly alerted that user_state field (the state where the user lived) started seeing a lot of null values.

Google bot issue. BoostKPI quickly alerted on a sudden conversion drop in the Web channel. The root-cause then turned out to be Google bot crawlers that were driving up product views but as expected, not doing any checkouts.