by Andrea Darabos

As much as feedback is at the essence of agile product development, we often fall into the trap of Build-Build-Build and Do-Do-Do and do not seek out customer feedback.

The scientific thinking of setting learning goals and defining experiments to test our assumptions is not a born quality of our human brain. We are prone to various biases and seeing assumptions as truth, and just building these assumed features into our products. The good news is, controlled experimentation and learning through reflection can be learned through practice.

A simple Kata Practice, observed at Toyota and publicised by Mike Rother in his book Toyota Kata, can help us develop a practice of scientific thinking, of goal-setting, measurement, observation and experimentation.

For the KataCon3 conference held in San Diego 2017 this year, I developed a game to help teams practice the Improvement Kata and the Coaching Kata. To replicate the game, you only need a few supplies:

• 6 paper cups
• 1 rubber band
• 4 pieces of string
• Masking tape
• Print out the Improvement Kata and Coaching Kata templates from Rother’s website

Tie the four strings around the rubber band as you see in the video below.

Have four players around a table and one additional player who will observe the team going through the challenge and coach them through experiments and improvement learning.

The Kata starts with the Challenge. I have set it for you: Stack the six cups in a tower like this within 15 minutes. Start from cups stacked in one piece facing downwards on the table.

Simple rules: Players are not allowed to touch the cups by hand in any way, you can only hold the string. Cups fallen off the table can be picked up by the string only.

Next, players need to keep track of the time it takes for them to stack the cups. We do a practice round, where the facilitator explains the main parts of the process. It can go like this:

1. Players pick up the ends of the string.
2. Separate the one stack of cups on top of the table one by one.
3. Position the three cups as base next to each other.
4. Place two cups on top.
5. Place the top cup on top.

Then, the facilitator asks the team to repeat and measure the same process again for two more rounds. The last time you have measured is your Current Condition.

The facilitator then asks the team to set their next Target Condition, knowing that they will have five rounds to achieve it. What target total time can they achieve within five rounds of Kata Practice, experimentation and reflection?

Each round includes the following thought process:

1. What obstacles are blocking our team from achieving the Target Condition? Which obstacle do we choose to work on (prioritise)?
2. What experiment will we run to remove this obstacle?
3. What do we expect will happen?

The team then fills this in on their notes and performs the experiment.
After the experiment, the facilitator asks the team:

1. What is your target condition?
2. What is your actual condition?
3. Which obstacle are you working on?
4. What experiment have you planned to remove this obstacle? What did you expect?
5. What just happened?
6. What have you learned?

The learning is key, and the coaching questions help the team contrast their expected outcomes of an experiment to actual outcomes. Based on the learning, the team then can define their next experiment. It usually takes multiple rounds of experimentation to remove an obstacle.

The faster the team is able to run experiments and learn, the faster they can achieve their target condition.

Of course, the goal of the simulation is not solely to stack cups and achieve their time goals. It is rather to give a taster to practicing several rounds of scientific thinking, defining experiments and debriefing learning in your team. The team learns the thinking patterns of the Improvement Kata, and the facilitator learns to ask the right questions to guide the team.

In just an hour, we can develop a thinking pattern in our product team to work in smaller increments to define goals, to recognize obstacles and to experiment our way to removing them. This same Improvement Kata can be practiced in product review sessions and planning.

In the product review session, have a team member ask these questions:

• What is our current improvement goal in the product? (e.g. improve the percentage of sign-ups of User Type A)

• What is our current condition? (Last sprint we measured 10 percent sign-ups through the mobile channel)

• What obstacles are keeping us from improving this number?

• Which one should we try to remove in our next sprint cycle?

In your planning meetings then ask:

• What experiment will we run to remove the selected obstacle?

• What outcome do we expect as successful pass criteria?

Then your team needs to run your experiment and assume that it will not make a difference, i.e. not remove the obstacle. (Null hypothesis). If the outcome proves this assumption wrong, the experiment indeed made a difference and the proposed feature is useful. Otherwise we should not build in this feature. We assume that the experiment will not change the outcome, then we run the experiment, hopefully disproving our assumption. Suppose, for example, the experiment indeed managed to move the needle. In this case, we debrief what we learned and usually keep doing this thing that made an impact.

A real-life example could be that we assume raising our prices will bring in more senior management enquiries which are the target customers we would like to have more of. So we set up an experiment with two price ranges — one unchanged and one at a higher price and run it publicly. Our null hypothesis is that the higher price doesn’t impact the profile of enquiries. However, it it does, we have disproved the hypothesis and have learned that we should raise our prices.

If you want to try another Kata practice, trying Kata to Grow, a game Rother developed for both schools and teams and managers — it’s super fun!

Have you experimented with any of these Kata games? Tell us about it in the comments below!