A short guide on rapid experimentation

Image Source: http://study.com/academy/lesson/experimental-design-in-science-definition-method.html 

In the last couple of years, I have started to appreciate the power of small, quick experiments. As product managers, we often face the challenge of bringing together the right resources (especially, design / tech expertise) to validate certain ideas and to make progress.

Thanks to the lean methodology, many of us have started to embrace the experimentation mindset, which involves

• Taking a complex problem and breaking down into small, executable chunks
• Seeking progress, irrespective of the results
• Incorporating key learnings/insights back into the experimentation process

It's easy to read about a framework or a methodology from a book/blog and get excited about the benefits it can bring to your role. But unless we practically experience/execute it in our realm of work, we will not be able to appreciate the benefits or realize the impact.

Experiments are akin to prototyping in the design world. They are not expected to provide the complete solution but give you enough information to make the "next" decision. The reasons why experiments are a handy tool for a product manager are the following:

• Helps to explore the problem space and the context
• Can be driven independently
• No dependency on tech/UX resources
• Quick way to capture baseline metrics

An experiment is slightly different from a hypothesis. In the case of an experiment, you are seeking the unknown whereas for a hypothesis, you have a point of view and want to validate whether it is true or not.

The scope of an experiment could fall under any of these categories:

• to identify the hypothesis to go after
• to validate/invalidate a hypothesis
• to gather sample data, using which you can derive meaningful insights
• to understand a process, potential flow, dependencies, pitfalls and bottlenecks
• to explore multiple alternatives, different means to achieve a certain goal
• to simulate a workflow before embarking on multi-level approvals and coordinating with various stakeholders

For an experiment to be successful, I recommend a 6-step process:

1. Be very clear of the scope
2. Define a goal
3. Identify the target audience
4. Fix the timeline during which the experiment will be run
5. Make a clear plan of action. List down the steps involved
6. Define the metrics to be captured

To give you an example, I'm sharing an experiment that I ran for a SaaS product.

Scope - to identify a goal-driven communication plan for new user onboarding.

Goal - to design a series of contextual lifecycle emails that result in higher conversion rate (the number of users who completed the expected onboarding flow)

Target Audience - users who signed up for free trial of the SaaS product.

Steps involved / Plan of action - As a new user completes certain tasks/activities, an email will be sent, nudging the user towards the next task. The end outcome of these emails is to get the user to experience the complete flow and achieve success with the product.

As part of the experiment, we tried out various subject headers, email content, personalized contextual success messages and supporting collateral/documents. This was a completely manual effort without any dev/UX involvement. Once we finalized the flow, we then operationalized it using a customer communications platform.

Metrics to be captured - % of users who complete steps 1 to N after they signup for free trial. Measured cumulative as well as for every change in variables (subject, email content etc)

Do share your thoughts on this topic and how an experimentation mindset has worked for you.

Originally published in LinkedIn Pulse - https://www.linkedin.com/pulse/short-guide-rapid-experimentation-anuradha-sridharan