1. What is a Change Log for Experimentation?

A change log for experimentation is essentially a detailed and structured records of all the experiments a team or organization runs. Think of it as a diary of every hypothesis you test, timeline, experimentation design document, result you achieve and learnings you unlock. It begins from the idea backlog stage and continues through to the final results and beyond.

2. Why is it Essential?

• Knowledge Base Creation:

  • A Change Log acts as a comprehensive knowledge base, cataloging all the experiments, metrics, cohorts, growth area, documentation including valuable insights from completed experiments and the evolution of ideas. Having a central record ensures that every team member, both present and future, can quickly understand what was tried, what worked, and what failed. Onboarding new team members, as it provides them with a wealth of knowledge and insights at their fingertips.

• Measure Experimentation Program Success:

  • Measuring success in experimentation often hinges on more than just the number of experiments conducted. Relying solely on this metric can lead to less ambitious optimization experiments and miss out on the complete picture. To truly gauge success, it's essential to consider multiple factors in addition, like impact estimations, time to build, and comparing expected outcomes with actual results. This fosters an environment of perpetual improvement and responsibility. This not only assesses the precision in forecasting impacts and required engineering efforts but also refines future estimates, ultimately boosting the experimentation program’s success rate.

• Control Tower Functionality:

  • The change log can also serve as a control tower, helping teams avoid collisions by providing real-time information on upcoming and ongoing experiments. It enables teams to coordinate better and streamline the experimentation process.

• Enhanced Insights Sharing:

  • With a centralized change log, knowledge sharing among teams becomes streamlined and efficient. Team members can quickly understand the state and outcome of ongoing or completed experiments across all the growth areas without sifting through scattered data. This is vital when there are multiple growth teams, it helps leverage user learnings derived from experiments widely across the company.

• Timeline Construction & Key Metrics Correlation:

  • A change log aids in building a timeline of past changes as necessary, allowing teams to correlate these changes with impacts on key metrics. This is essential for understanding the long-term impacts experiment launches have made and making informed decisions for future initiatives.

3. Getting Started

To facilitate a smooth transition to using change logs, here are two templates:

• Simple Change Log Template:

  • Ideal for a team new to experimentation, this template is straightforward and easy to use. It includes basic fields that cover the essential aspects. Use along with experimentation design document.

• Airtable EVELYN (Experiment Velocity Engine Lifting Your Numbers) Template:

  • This comprehensive template built by Darius is for teams looking for a more sophisticated and detailed approach to documenting their experiments with better filtering capabilities and formulas.

4. Leveraging Technology

While templates are beneficial, a custom internal tool can elevate the effectiveness and avoid manual creation and maintenance of spreadsheets. Depending on resources available, size and maturity of the company, this may be valuable. Such a tool, for instance, can:

• Enable Advanced Search: Easily find past experiments based on various criteria.

• Highlight Key Learnings: Quickly see the main takeaways from each test to subscribers and leaders.

• Visualize with Timelines: Understand the sequence of experiments and their impacts over time.

• Provide a Control Tower View: Get a holistic view of all ongoing and completed experiments.

5. Conclusion

Maintaining a change log for experimentation isn't just about record-keeping; it's about harnessing the collective intelligence of your team. It's a testament to the value of every test, every insight, and every innovation. Whether you're just starting or scaling rapidly, having a structured way to capture and share learnings can make all the difference.

An experimentation design document is crucial in helping an experiment owner think through all cases—to frame, provide clarity, and democratize knowledge cross functionally, improving future strategies and optimizing resources. It also enhances collaboration and decision-making within the team.

Scroll down for the template you can use immediately, it should take ~10mins to leverage it, enjoy!

1. Problem Statement 🚧

What is the business outcome?
Start with the outcome we want to improve. What metric or area shows there's an opportunity to drive meaningful impact?

What is the customer problem?
What problem are we solving, and for whom? Why does this problem exist? Focus on the user's experience and pain point.

What observation, data, or insight led us to identify the problem or opportunity?
What research, insight, or data surfaced this as something worth exploring?

How does this connect to the current growth model and strategy?
Which part of the growth model (e.g., acquisition, activation, retention) does this address? How does solving this problem tie back to the company’s broader strategy and mission?

You can also provide links to related artifacts e.g. user studies, research that’s been done, and previous experiments in this area.

2. Hypothesis 🧪

We believe that by changing the [independent variable] we expect [dependent variable] (not) to [increase/decrease] because [some reason].

Example:

We believe that by changing product suggestions at checkout from static to personalized we expect average revenue per user to increase because users are more likely to buy products that are relevant to them.

A hypothesis is something that we believe to be true based on what we know about users. Hypothesis should be testable and falsifiable — that means there’s something out there you can practically observe that would lead you to reconsider the hypothesis.  

• Start with the words “we believe”

• Use the word “because”

• Don’t use “if” or “then” (that’s a prediction)

Resources:

• Clear learnings come only from clear hypotheses.

3. Experiment Design 📊

How are we designing this experiment to test our hypothesis? Which group of people are in the experiment? How does the Control vs Variant experience look like?

How long will we run the experiment? Has effort and time to build the test been identified? Ensure test does not have a collision.

Assets (add links)

• Jira issue

• Design mockups

• Report that shows the current state of things

• Figma or Miro comparing control vs treatment experience.

3.1. Experiment Cohort/Target Audience 🎯

Describe the user base we will target to run this experiment to, e.g.: 

• Markets: e.g. "US only", "All English", etc

• Customer type: New, Existing, Paid, Free, etc

• Signup pathways: All, Front of Site, Email, etc

• Tiers: Basic, etc.

• Other: (for example, users who did x action or users who have specific characteristics - whatever it may be)
  

3.2. Test Length ⏳

Given how we are measuring the test, and the size of the audience, how long will it take us to get to significance? 

Use a test duration estimator to state how long it will take for the experiment to reach statistical significance.

Example:Test should take 2 weeks to get to significance

4. Metrics 📈

Describe exactly which metrics we are measuring to determine results for the test. List primary and secondary metrics if applicable. Please include baseline metrics (e.g. the control). Describe how we will calculate the metric if it is not a standard metric or if we are inferring from behavior. For example, if the goal is conversion improvement but we are using clicks on the plan page options as a leading indicator, spell that out. Any other metrics that could be cannibalized should be considered and documented

Example: 

Primary test metric: Conversion rate for New Users. Current baseline is x%

Secondary metrics: Plan mix, Term mix, etc. Current baselines are... 

Other metrics to consider or be aware of for this test that might be impacted for analysis.

You can also include links to relevant reports, or other artifacts that may help whoever is performing or interpreting the analysis. 

4.1. Instrumentation 📡

Do we have all the event logging in place to gather these metrics? What events and properties are missing?

Provide a list of tracking which will be used for the experiment so it can be verified both during development and post release by examining data reported. It can be specific guidelines and let developers format the name or it can be explicit names required. 

Please provide when these should fire.

5. Definition of Success 🏆

We will determine that the variant is a winner if it… [increase/decrease the primary metric by ≥ x%] higher than the control group at statistical significance.

6. Pre-Mortem ⚰️

This is to help increase test preparation. Based on the potential results, what actions will we take?

• Implementation plan: How will we take the experiment variation and turn it into permanent product experience if it wins?

• Iteration plan: What about our initial assumption has changed? Will we form a new hypothesis?

• Expansion plan: How will we apply the learnings from this experiment widely and double down?

• Holdout groups: Does it have potential for actual win to be different when made permanent product experience and should you use holdout groups or follow up data checks?

7. Analysis/Results 📊

Our hypothesis turned out to be [correct / incorrect].

You can post a link to a spreadsheet or other document if results are captured elsewhere and link to funnels demonstrating user behavior, etc. 

8. Learnings 🧠

We learned that…  What did we learn from running this experiment? How do these learnings impact the next steps? Tie them back to the pre-mortem test preparation work and share out the learnings widely.

When sharing out the learnings widely, it may help to categorize them i.e, high value vs informational.

Additional Notes:

• Feel free to copy and customize the Experimentation design template.

• Experimentation is still expensive so ensure that one is needed (This is a good post that talks about when you might not need one).

• You might also want to evaluate if the test is complex enough that you need to run an A/A test. (Test set ups in new surfaces or more than one surface often require both preplanning and an a/a test run to ensure data collection is set up for analysis)

• You can create an experiment writeup document using the template and complete steps 1-6 before development begins.

• Depending on the documentation tools you use, eg. Confluence or Notion, you can enable creating this documentation using this template with the click of a button.

This template is continually evolving, and your input is appreciated. Feel free to share any suggestions or comments you may have, and enjoy using it!

Leave a comment

🌟 What to Expect 🌟

🚀 Growth Engineering: Dive deep into innovative engineering systems and strategies that fuel growth.

🛠️ Growth Processes: Uncover the proven methodologies, frameworks & tools that empower cross-functional growth teams.

🌐 Growth Teams: Uncover the secrets of building and nurturing experiment driven growth-focused teams.

💡 Actionable Tips: Receive practical & actionable tips that you can implement immediately.

📆 Updates: Expect our newsletter right in your inbox, keeping you informed and inspired.

If you've already subscribed, THANK YOU. That means the world to me. Subscribing to the newsletter is free and incredibly simple

Let's grow together! 🌱🚀