Experimentation Practices

Running experiments to validate assumptions around potential new products and features is crucial because it provides empirical evidence about their viability before committing significant resources.

Experiments allow for testing hypotheses in a controlled manner, helping to identify whether a feature genuinely addresses user needs and delivers value. This approach reduces the risk of investing in features that may not resonate with users or achieve the desired outcomes. By validating assumptions early, product teams can make data-driven decisions, optimise resource allocation, and increase the likelihood of successful product outcomes.

Experiments don’t have to be products. They are low-investment opportunities to test hypotheses. Here are some examples of experiment types that can be used.

Observation

Observation helps gather evidence for interest and demand for a feature by allowing you to directly witness how users interact with current products or processes in their natural environment. This method provides insights into user behaviours, pain points, and unmet needs that might not be articulated in interviews or surveys. By closely observing users’ actions we can identify gaps and opportunities for new features.

Interview

Interviews are a vital tool for gathering evidence of interest and demand for a new product or feature by providing direct insights from the target audience. Through these interviews, we can understand customers’ needs, pain points, and preferences, which helps in validating whether a proposed feature addresses real user problems. By asking open-ended questions, observing customer reactions, and delving into their experiences, we can collect qualitative data that highlights the relevance and potential value of the feature.

Explainer Video

An explainer video is a short, engaging video designed to communicate products features in a clear and concise manner. Typically ranging from one to three minutes in length, these videos use a combination of animated visuals, voiceovers, and text to break down information into easily understandable segments. They are an effective tool for capturing audience attention, simplifying complex concepts, and highlighting the unique value propositions of a product or service.

Explainer videos can be used to gauge interest levels by in product features that may not have been built yet.

Pre-Order Page

A pre-order page is a web page that enables potential customers to place orders for a product before it is officially released or available for sale. This strategy helps companies gauge market interest, validate demand, and secure initial revenue before committing to full-scale production or development. It typically includes detailed product descriptions, high-quality visuals, and special incentives such as discounts or exclusive access to attract early buyers. By collecting pre-orders, businesses can make informed decisions about product viability, necessary adjustments, and optimal inventory levels.

Landing Page

Create a simple landing page on a website to test a product’s potential interest level and viability. The goal is to gauge user interest and gather data to inform their future product decisions.

The landing page highlights the product’s fundamental value proposition and benefits (which do not yet exist). It will include tracking tools to measure user behaviour on the page. We want to know how many users visit the page, how long they stay, and if they take the desired action, such as signing up for more information or clicking a call-to-action button.

We will utilise various channels, such as targeted online ads, social media posts, and email campaigns, to drive traffic to the landing page. As users start to visit the landing page, we monitor their behaviour. They analyse user interactions, looking for patterns in how they engage with the content. They pay attention to any drop-offs or areas where users seem confused or disinterested.

After running the experiment for a defined period, the Scrum Team gathers and analyses the data. They compare the results against their initial hypothesis to determine if it was validated to give them the confidence to proceed with further investment in developing the product. If the hypothesis is invalid, they use the learnings to refine the idea and design a new experiment to test an alternative approach.

Concierge MVP

Concierge MVP (Minimum Viable Product) is a product development strategy where a manual, high-touch service is provided to the initial set of users to validate the product idea without building a fully automated solution. This approach involves personalising the service for each user to gain deep insights into their needs, preferences, and behaviours. The goal is to test and validate the core assumptions of the product concept with minimal development effort.

Feature Fake

Feature fake experiments give the illusion of a fully functional feature with a button or a link on a website or in an app. When the user clicks the button or link, they receive a message informing them that the feature does not yet exist and advising them how to complete the desired action without it.

This experiment aims to learn what to call a feature, where to put it in your workflow and see if people can find it, understand it, and think it is valuable enough to click it.

This can be a powerful way to test the interest level in a new feature without paying the cost to build it. It must be used sparingly and for limited amounts of time so as not to upset users by setting false expectations.

Wizard of Oz

The Wizard of Oz experiment is a technique used to simulate the functionality of a product or feature without fully implementing it. It looks and feels like a functioning product to the user, but it’s primarily humans doing the work behind the scenes. The goal is to gather user feedback and validate assumptions before investing significant resources in further development.

Imagine a team working on a voice-activated virtual assistant for home automation. They want to test how users interact with the assistant and assess its usefulness. Instead of building a fully functional assistant, the team conducted a Wizard of Oz experiment. They create a facade that gives the appearance of a working voice-activated assistant. Behind the scenes, a team member acts as the “wizard” and manually responds to user commands.

The team invites users to interact with the virtual assistant to start the experiment. Users may believe the assistant operates through advanced voice recognition and artificial intelligence, unaware that a human is orchestrating the responses.

As users issue voice commands, the “wizard” listens and quickly formulates appropriate responses, relaying them as if the assistant generated them autonomously. The team member behind the scenes may even simulate delays to mimic processing time.

The team observes how users interact with the assistant. They monitor which commands users try, how they phrase their requests, and whether they appear satisfied with the responses.

The team gains valuable insights into user behaviour and preferences through this experiment. They may discover potential issues with the interaction flow, identify popular commands, or notice areas where users struggle to communicate effectively with the virtual assistant.

Indeed, Amazon used the Wizard of Oz approach extensively to develop its Smart Speaker technology.

Paper Prototype

A Paper Prototype experiment is a hands-on, low-fidelity technique used to quickly and iteratively test product concepts with users. It involves creating simple, hand-drawn sketches of user interfaces on paper, simulating the flow of interactions and user journeys.

For example, imagine a team developing a new mobile app for a recipe-sharing platform. Instead of investing time and resources in building a digital prototype or coding a functional app, they conducted a Paper Prototype experiment.

The team gathers in a room with a whiteboard or large table and starts sketching the key screens of the app on paper. They create drawings of the home screen, search function, recipe detail page, and user profile, among others. These sketches are basic representations of the app’s UI elements, allowing the team to visualise the overall design.

Once the paper prototypes are ready, they role-play as users and simulate interactions with the app. One team member acts as the user, while others move the paper screens accordingly based on the user’s interactions. The facilitator observes the user’s actions and asks questions to understand their thoughts.

For instance, the facilitator might say, “You want to find a vegan dessert recipe. How would you do that?” The user then navigates the paper prototype, pointing at different screens and explaining what they expect to see or do next.

Through this hands-on exercise, the team gains valuable insights into how users interact with the app and identifies potential usability issues or design flaws early in the process. They can quickly adjust the paper prototype based on user feedback and iterate on the design until they achieve a more intuitive and user-friendly interface.

The advantage of the Paper Prototype experiment is its rapid and low-cost nature. Since it involves minimal effort to create and modify sketches on paper, the team can test multiple design iterations quickly. The insights gained from users’ reactions and feedback inform the following stages of the product’s development.

Service Prototype

A Service Prototype experiment is a technique used to test and refine service-based experiences. It involves creating a simulated version of the service to understand how customers interact with it and identify areas for improvement.

For example, let’s consider a team working on improving the customer service experience for an e-commerce website. They conducted a Service Prototype experiment to gather insights and make data-driven decisions.

To begin, the team identifies critical touch points in the customer service journey, such as website chat support, email inquiries, and phone support. They create a script and scenario for each touchpoint to simulate customer interactions.

Next, the team gathers in a controlled setting, such as a conference room, where they role-play both the customer and customer service representative (CSR) roles. They might use props or tools to create a more immersive experience.

As the experiment begins, one team member acts as the customer, while another plays the CSR. They follow the predefined scenarios, with the customer making inquiries or raising issues and the CSR responding accordingly.

During the simulation, the team observes the interactions and takes notes on customer responses, CSR performance, and pain points in the service delivery. They pay attention to the effectiveness of the responses, the clarity of communication, and the customer’s overall satisfaction.

After the simulation, the team gathers to discuss their observations and reflections. They share feedback on what worked well and what needs improvement. The team considers possible solutions to address the identified issues.

Based on the insights gained from the Service Prototype experiment, the team can iterate on the service design and make necessary adjustments before implementing the changes on a larger scale.

The advantage of the Service Prototype experiment is that it provides a safe and controlled environment to observe customer interactions without impacting real customers. It allows the team to gain valuable insights into the service experience and make informed decisions to enhance customer satisfaction and loyalty.