Use Case: Ad Experimentation
We need to add the ability to systematically experiment with the ad portion of our flow, so that we can deterministically show how experiments related to ads affect downstream user behavior. This will
We need to add the ability to systematically experiment with the ad portion of our flow, so that we can deterministically show how experiments related to ads affect downstream user behavior. This will also be the basis for how we implement other experimentation related to email, and more.
## Overview
Similar to how we have one main branch of the platform we are building, we want to have one main mechanism for ads. This allows us to focus our messaging and direction, and add experiments for key pieces of the ad as needed.
So, how will we create the experiments and variants? How will the variants propagate to newly minted units? Should the ad campaign be its own product, or should it be part of the socra product on the lab?
## Thoughts
If the ad campaign is its own product, we lose traceability on a unit-level for experiments run on the ad campaign, but gain separation between the ad campaign product and socra product. Functionally, though, we need to show how variation in ad elements impact downstream user behavior. For this reason, I believe ads should be part of the same core product as socra.
Let's use a basic example to cement this reasoning. Some common elements we will want to experiment with for ads are:
- The max CPC bid
- Ad content (headlines/descriptions)
- Keywords
Each of these variables could be experimented with together, but more than likely, we'll want to experiment with each independently. This means that each will be its own experiment (ads.google.max_cpc_bid, ads.google.content, ads.google.keyword), with its own set of variants. This gives us the most amount of flexibility in terms of experimentation while also providing a common "branch" to grow from.
So, how many ads should be created on Google? How will we tie specific experiments to downstream effects? The aim is to see how each variable affects the baseline. So, we have two main options:
1. Have a single, overall baseline ad that represents all experiment control groups. Each non-control experiment variant has a unique ad created for it. Thus each ad is either the overall control or created to explicitly test a single variant. This is functionally no different than having a single experiment representing all variables. We don't get to see how each variable affects others, so we would only be able to see how each variable affects the control group. The amount of data required is (number of CPC bid variants) + (number of ad content variants) + (number of keyword variants).
2. Have a unique ad for each unique combination of variants across experiments. For the example above, this would be (number of CPC bid variants) * (number of ad content variants) * (number of keyword variants). Assuming 2 variants for each, this yields 8 unique ads running. In order to collect a statistical sample of data, we'd effectively need 8x the amount of sample data, and we would be able to see each variable's unique effect, and combined effects of all possible combinations of variables.
Since our ad represents a continuous living entity, (2) is the optimal approach, since it doesn't require stop/apply/restart. We can have one configuration constantly adapting and improving over time.
## Assignment
Each unique combination of variants across experiments will have a single ad campaign, ad group, and ad group ad created for it.
When we receive a visitor with a unique GCLID, we need to be able to extract the assigned variants of the ad, and pass these assignments to the unit created for the user.
So, how do we uniquely identify an ad, and tie it to specific variants? The best way is to create a database table to help us tie unique Google resource(s) to the experiment variants we are running. This will allow us to create an assignment table directly from ad to variant.
What will the process look like to create and manage ads?
- Each ad will be uniquely identified by its unique combination of assignments.
## Stage 1
- [x] Add Ad model
- [x] Add AdVariantAssignment model to tieBy Mike Morton