Advertising Experiments at RestaurantGrades
RestaurantGrades (RG) is a restaurant review platform (similar to Yelp or TripAdvisor) with an
impressive stock of online reviews written by ordinary restaurant-goers. RG has compelling data
demonstrating that these reviews have an economically important influence on the restaurant choices
people make, which is exciting for the executive team and has caught a lot of attention within the
industry. However, doubts have been raised about the efficacy of their main source of revenue –
selling ads to restaurants. To better understand this issue, they have run a randomized controlled trial
with a control group and two treatment groups: one treatment to test the impact of their current
ads on restaurant sales, and the other treatment to test the impact of an alternative ad design that
they are considering switching to. They have given you the attached data, and asked you to help
them interpret and act on the results. In particular, they want to understand whether their ads
really work, and whether they should stick with their current design or switch to the alternative
design.
Background on Advertisements
On RG, each restaurant has a profile page with operational information including its hours, phone
number, and location, where RG users who have visited the restaurant can leave reviews for other
users. Users can also discover and search for restaurants on the platform using filters, and make
reservations and order food through a restaurant’s profile page.
The majority of RG’s revenues stem from selling ads through its sales team, which cold-calls
restaurants to try to convince them to advertise on the platform. Advertisements, labeled as sponsored
search results, are placed in a separate section above the organic results for searches that users conduct.
Packages of advertisements are purchased for about $300 per month, and advertisers are required to
sign up for a one-year contract. While RG uses a search algorithm much like Google’s that determines
when and which ads are shown given a user’s search for restaurants on the platform, restaurants have
little say in what search terms will trigger their ads. However, they are guaranteed that their ads will
be shown a minimum of 1,000 times per month to users.
RG’s current search algorithm shows ads for restaurants triggered by type of cuisine within a 0.5-
mile radius of a user’s search. For example, if a user searches for Italian restaurants in Harvard Square,
the algorithm will choose two Italian restaurants within Harvard Square to advertise. The engineering
team has run a variety of tests looking at how users respond to different types of ads in different
Advertising Experiments at RestaurantGrades
2
searches, and designed an alternative search algorithm. Rather than choosing two restaurants by
cuisine, the alternative algorithm shows advertisements when a user searches for a specific restaurant
and selects two restaurants with similar ratings and hours. While they are reasonably satisfied with
how the current algorithm shows ads, they are open to the possibility that the alternative design may
be significantly better (or significantly worse) in providing benefit for their advertisers.
Experiment
For the experiment, RG randomly selected 30,000 restaurants that were active on their platform but
were not currently advertising, yielding a sample that is representative of their population of
restaurants in the US.
For the one-month duration of the experiment, 10,000 restaurants were randomly selected to receive
free ads using the current advertising approach, and another 10,000 restaurants were randomly
selected to receive the alternatively designed ads The main difference between these two treatment
groups is that the alternative design used a very different algorithm to decide when to deliver ads and
which ads to pair with each search, as described above. The rest of the 10,000 restaurants received no
advertisements. None of the restaurants were informed about the experiment or the advertisements.
The spreadsheet supplement for this exercise contains a variety of outcomes observed for these
30,000 restaurants during the one month of the experiment.
Spreadsheet Data
The spreadsheet supplement contains data for each restaurant in the experiment observed in the
month during the experiment. The unit of observation is a restaurant-month, so data of each restaurant
are in a single row. For example, in a row, pageviews means the number of unique visits on restaurant’s
RG page during that month. The variables included are as follows:
Variables
Variable Name Definition
treatment =0: in the control group
=1: in the first treatment group (ads of current design)
=2: in the second treatment group (ads of alternative design)
pageviews # of visits to the restaurant’s RG page per month
calls # of phone calls made from the restaurant’s mobile RG page per month
reservations # of reservations made from the restaurant’s RG page per month