A guide to experimentation and testing your product decisions

We all want to make good decisions. We can’t yet bounce back in time to fix our mistakes. Make informed decisions by testing your genius before going all in. A few of my thoughts on approaching testing.

Image for post
Image for post
Credit to Lauren Okura https://twitter.com/OKyouRA for these awesome little vector humans.
  1. Lead to spending too many resources building the wrong thing
  2. Drive up support interactions
  3. Lose your business good will in the industry
  4. Trigger a larger system breakdown
  1. Apply a structured approach to deciding how you’ll test
  2. Settle on what success or failure might look like and how you’ll know either way
  3. Choose your toolset carefully, set them up properly, follow conventions

Step 1: The anatomy of a good test — Locking down the basics

Testing is important. Testing the right thing is vital.

  1. Increase the accuracy of your results
  2. Have more business impact with well thought out tests
  3. Get to testing faster
  4. Avoid adding additional work and cost to your experimentation
  1. Can stand on their own — You need to be able to trust the results of your test. Don’t test two similar things at the same time. If you change your pricing model and an important page about premium features at the same time, for example, you’ll have a hard time knowing which one drove success or failure in conversion. A good test stands on its own.
  2. A clear ROI — Even quick tests require time and effort to put into place. Make sure your time investment is worthwhile. A successful test can pay for itself many times over, either monetarily or with key learnings.
  3. Statistical Significance — Pay attention to how many users you need in your experiment to get a true result. A test in which you watch real users using your experiment needs fewer users than an automated test using analytics. Tools like Optimizely will let you know when you’ve reached statistical significance or how many users you still need to get there.

Step 2: A structured approach to testing changes

Once you’re set on what to test, you’ll need a clear way forward. I’ve done my best to distill some of my own learnings from this planning process.

  1. Audience: How many people will have an opportunity to interact with your test? This should be explored as a function of likely user base/potential user base.
  2. Urgency: Is your current setup performing extremely poorly or do you just think there’s room for improvement? If you’re trying something brand new, how important is it that this gets to market quickly?
  3. Business impact: If we get this wrong, will we lose customers, money, or goodwill?
  • Controlled rollout: Roll out the change to a subset of users; monitor and potentially increase the size of the subset if we need more users to reach statistical significance.
  • Internal user rollout: Apply either of the first two approaches, but limit the experiment to internal users only.
  • User testing: Use a test environment to expose a small group of users to the change to observe their behavior.
  • Audience: potentially millions of users
  • Urgency: High; the changes were becoming a necessity
  • Business impact: large; if we fail it’d reduce conversion and lead generation
  • Controlled rollout: I feel more comfortable with this option, as we will be limiting the audience size. Our risk is lower and, since this is urgent, we stand to gather results more quickly from our public audience than if we limited it to internal users.
  • Audience: potentially hundreds of thousands of users
  • Urgency: this is an improvement and not urgent
  • Business impact: small (no immediate business impact)
  • Controlled rollout: Although the business impact is low, the urgency is also low. So while this feels like a reasonable plan, the size of the change makes me unsure.
  • Internal user rollout: This feels better! We can quickly validate our idea and once we’re happy, we can move to a Controlled rollout to our customers.

Step 3 — Settle on what success or failure might look like and how you’ll know either way

Now that you know what you’ll test, you need figure out what it will look like when you’ve achieved what you set out to. From my experience, this part is either really hard or really easy.

  1. Think about and list out an extensive set of things that might be affected by your change.
  2. Prioritize a few of these as your main metrics; try to break out any multi-part metrics. For example, let’s say “Number of completed application forms” is your metric. Think about whether there are other metrics that could impact that number, such as “Clicked signup” or “Validation error”. Capture these as new metrics to track.
  3. If one or more of your main metrics come out in an unexpected way, dig into some of the seemingly less important metrics where the answer to the surprise might exist.
  1. Conversion: Do you have a specific activity or task that must be completed? Does your change impact a part of the journey to complete this activity? Examples include things like completing a transaction, adding an item to a cart, clicking a signup button, or completing an application. Be aware that conversion is often not linear! Being more upfront about the requirements for applicants to your program might mean fewer people click to sign up, but more people submit quality applications. Ensure your conversion numbers are comprehensive; just tracking signups would lead you to a false negative in such a situation.
  2. Retention: Does your change increase or decrease the likelihood of the user coming back to your product/site/app?
  3. Page analytics: Bounce rate, exit rates, time on page, and session length are all popular page metrics. Keeping track of a few of these is usually recommended. Again, acknowledge the complexity of some of them! Whether or not a value is good or bad depends on the situation. For example, the bounce rate on a page which is supposed to be used in a flow is important to keep low. A page like a help center article, on the other hand, which users might get to from a Google search and then leave once they have the information, should have a relatively high bounce rate (depending on how your chosen analytics tool defines and implements bounce rate).

Step 4 — Choose your toolset, set it up properly, follow conventions

The three previous steps lay a strong foundation for structuring your test and knowing how to measure it. Now you will need a way to implement your test. Some teams I’ve worked with roll their own tools for testing; others use a single existing product. Some will use a plethora of tools all strung together to achieve more complex goals. No matter where you’re starting, it’s not hard to get something out there with minimal effort.

  1. Custom tooling in place to run tests
  2. A single vendor provided tool in place to run tests
  3. Multiple vendor/internal tools in place to run tests

Use conventions

There’s a good chance your team or someone in your company is using a convention when tracking metrics and setting up tests. If not- now’s the time to start! If so, follow those conventions.

In conclusion

I’ve tried to outline some of the thinking I’ve done on analytics and experimentation above. It is by no means exhaustive or even necessarily right for your team, product, or company. I do, however, hope that it will spark some conversation or debates, and maybe just maybe it’ll be insightful for somebody.

Image for post
Image for post

Written by

Design, data and delicious things.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store