TWO STORIES
1. High School Survey
The US government used to do an annual survey of high school
students. The objective of the survey was to track things like the levels of
drug use and sexual activity in that age group. Because the writers of the
survey were afraid that high school students would lie about their own personal
activities, the government asked the students to estimate what percentage of
students in their high school they felt were doing each of these types of
things.
At first, the government would average all of the responses
for each question to get an estimate of how prevalent various activities were.
But then, someone dug deeper into the results. What they found was that
although the average percentage number was somewhere in the middle, very few of
the individual students ever answered with a number in the middle. Instead,
there were two clusters of answers on each question—one cluster of students
which gave a very high percentage answer and one cluster of students which gave
a very low percentage answer. This caused the government to take a second look
at the results.
As it turns out, the researchers found out that most
students do not hang out with a large percentage of their fellow students. As a
result, the students had no idea of the drug and sexual activity of the greater
student body. Instead, they only had the reference point of their small band of
close friends.
And these close bands of friends tended to behave similarly to
the others in the same band. So, if a student participated in these activities,
most of their friends did also, so they concluded that most of the people in
the school must also do these things. Similarly, if a student did not
participate in these activities, most of their friends also did not, so the
student would assume that most of the students in the entire school also did
not.
Based on these insights, the government shifted its emphasis
from tracking the change in the averages from year to year to tracking the relative
sizes of the high and low clusters from year to year.
2. Amazon Music
Reviews
Before buying music, I like to read the reviews in Amazon. Over time, I realized that the vast majority of all the music on Amazon has an average ranking of 4.5 out of 5. With nearly everything rated equally, it became impossible to use these average ratings to decide which music to purchase.
Before buying music, I like to read the reviews in Amazon. Over time, I realized that the vast majority of all the music on Amazon has an average ranking of 4.5 out of 5. With nearly everything rated equally, it became impossible to use these average ratings to decide which music to purchase.
But then, I started thinking. The problem with Amazon music
ratings is that they are voluntary. This is not a random sampling. People only
turn in a rating when the mood hits them. And typically, the mood only hits
them if something they hear is especially good or especially bad. And it in the
vast majority of cases, the review came from people who thought the music was
especially good.
Once I figured that out, I stopped looking at the average
rankings of the review and instead started looking at the number of reviews in
total for a particular piece of music. My logic was that if only the ones who
loved the music send in a review, then the more reviews sent in, the more
people loved that music.
This has turned out to be a far more effective way to use
the Amazon review process.
Companies like to base their strategies on facts. Sometimes, they try to get their facts directly from the consumer. This tends to happen most often at three phases of strategic planning:
1.
At the beginning, when trying to understand the market
place.
2.
In the middle, when testing concepts
3.
At the end, when assessing whether the strategy is
working.
The good news is that in today’s interconnected world, there
are lots of ways to get consumer input.
The problem is that these sources can often have flaws like
the ones mentioned in the stories. Complainers and the people who rate
companies online are not a random sample. They are biased towards people who
like to rate or towards people with extreme views (like the Amazon music
ratings). If you just look at the average ratings and comments, you will most
likely come to the wrong conclusion. It may be better to count the reviews,
rather than average them.
And even well designed surveys can with random sampling can
have flaws. After spending decades in consumer research, I discovered that
people will try to honestly answer all of your questions well, but they often
just don’t know the answer, so they guess—and often very wrongly (like the high
school survey). I have found this to be particularly true when asking people to
predict their future behavior in areas where they have little experience (like
how they would react to a new strategic scenario).
Therefore, we need to be careful in how we interpret this
data.
The principle here is that one cannot run a strategic planning process based solely on research, especially if you only look at averages. Part of this is due to some of the research flaws mentioned earlier. Another part is due to the nature of strategic planning itself.
Strategic planning is looking for ways to build a new and
better future. It can be about finding new white spaces which have never been
exploited. It can be about inventing solutions which never before existed. It
can be about building business models that break all the old rules. It can be
about finding uncharted “Blue Oceans” of opportunity. In other words, strategy
is a lot about trying to get ahead of the curve and be an early adopter of the
next big thing.
Of course, if you are trying to lead the way into the
future, you may be several steps ahead of the general population. Questioning
the general population may not be very useful at such an early stage.
However,
if you wait to move until the consumers can speak as experts, it is too late to
be at the front end of the strategic revolution. That’s one reason why Steve
Jobs didn’t believe in consumer research. He knew that consumers can’t speak
meaningfully about a future not yet envisioned.
But that doesn’t mean that research is useless. As seen in
the stories above, there are creative ways to look at data to get insights. But
may mean you cannot take the initial results at face value. For example, we saw
that instead of averaging out what people say, it may be better to just count
how many say something. Therefore, be careful when looking at your results.
Don’t necessarily take it at face value. Search deeper for the true implications—especially
as the questioning looks into the future or is not randomly sampled.
Other things to keep in mind:
- Although consumers may be
unable to articulate how they will act in an inexperienced future, they
can articulate what irritates them in the current state. Knowing the
irritations of today can help you when designing the newer future.
- Some people may be living
closer to the leading edge than others. Focusing research on leading edge
people may give better results.
- Even though solutions my
change over time, attitudes/concerns/desires regarding the problem may be
more stable. If you focus on researching the more stable problem issues,
it may give insights into how to develop better, innovative, new
solutions.
But probably the most important thing to understand is that
strategy is not pure math or pure science. It also has an element of artistic
creativity. Creating the future is, by definition, creative. Eliminate the
creative and you will never find what you are looking for.
Although we may want a data-based approach to strategy, relying only on data—or taking the data at face value—will probably lead you in the wrong direction. The future is not precise, so you cannot take just a precise approach to get there. The consumer is not always very helpful or knowledgeable in looking beyond incremental change. Therefore, one will need to also rely on artistic creativity to get to the future. In fact, the creative part is likely be more important than the scientific part.
You cannot find the future if you are only looking backwards. And looking backwards is where the consumers are. Sometimes you have to look forward, to the places where the customers have not yet arrived. Creative insight, rather than research, may be more useful.