Finding patterns in past market research

I have a friend who worked in pharmaceutical marketing management for many years. During a conversation some time ago, she informed me that it was common practice for her department to forecast the healthcare marketplace annually, looking far into the future. 

At the time, I wondered how and even why she made those sorts of assessments scanning so far ahead, particularly in relation to people's health. Most research projects done for clients are based largely on the requirements of the immediate and near future, not years ahead in time. 

She undertook such work by looking back into the past and identifying trends that would have relevance to the future. Intrigued by her comments, I began thinking about the shelf life of good market research information.

Most market research undertaken by companies today, particularly qualitative research, is based on the specific short-term objectives of the brand, product or category managers. The marketing teams work diligently to squeeze the last drop of intelligence from the research, but eventually the reports are archived away, often gathering dust and never reviewed again. The reasoning behind this short-term approach is, of course, usually based on current need but also partly on the premise that today's insights will be much less relevant tomorrow. After all, what value has yesterday's intelligence when the world is changing so rapidly?

In reality, however, this could not be farther from the truth. New product development will always benefit from past insights, particularly when the research design has accommodated a variety of market scenarios and potential product characteristics. Furthermore, the world is not changing as rapidly as may be thought. True, elements, particularly technology, are changing at a hasty pace, but this is not so for other aspects of life.  

Good market research should have a long shelf life and not be packed away and forgotten. Former British Prime Minister Winston Churchill's famous quotation: “The farther back you can look, the farther forward you are likely to see” has relevance in virtually all businesses today. My pharma friend clearly understood the significance of this and put it into practice.

Amassing support
Of course, past research should not be used merely to support the argument that something was tried in the past and therefore because it did not work then, it will not work now. The purposeful look at a wide range of projects, with the desire to find insights for the future, is very different. The point is not to find a reason to say 'no', but rather to amass support for saying 'yes'.  

Hindsight is obviously a powerful faculty and many in business would love to possess such a talent. People often say that if they knew then (referring to some event or time in the past) what they know now, how different things would be. In business, this wisdom of hindsight is about obtaining insights or intelligence after the event, but before it has actually happened. It sounds paradoxical, so the idea should be explored further.   

The question is how to achieve foresight from hindsight.

Taking the initial starting point, the current position, plus the intelligence gathered over the intervening years, goes a long way towards finding the answer. Market research specialists gather insights from past research in a process called research reviews: in other words, a qualitative data mining. Cognitive processes are used to create the three principal values: hindsight, insight and foresight.  

Qualitative data mining generally involves the analysis of large quantities of data, both qualitative and quantitative, to extract previously undetected intelligence or interesting patterns of behaviour. Re-analysing past market research learnings is not simply a matter of recycling old information. The objective is to leverage the learnings in a wider, more holistic context. 

It is not merely a case of going back over individual research projects one by one, but literally superimposing all the insights from past projects and seeing where common threads start to appear. Taken on their own, these insights may have little or no impact, yet when seen in a cross-brand and/or cross-category context, they can start repeating and compounding, showing that something more fundamental is occurring. 

Healthcare is not that different from many other market segments. In addition to evolutionary processes, there are a number of significant drivers that impact, and often radically change, the way people live. For example, the advances made in medicine and diagnostics over the last 50-plus years have had major and positive impacts on humanity. New understanding of how bodies work; drugs and new treatments to deal with disease; new screening devices, plus many other examples, have resulted in people not only living longer, but also living healthier.

Yet, with all these advances in medical science, there has also been an enormous shift over the last 10 years towards, or perhaps it should be back to, complementary therapies. In the five years from 1999 to 2004 alone, there was a 45 per cent increase in the number of UK adults using alternative therapies. What has caused this shift and, indeed, is there a trend away from the conventional forms of clinical medicine?

The answer to the latter is probably no, but it is interesting to see the drivers of this change.   

Influence of positive memories
Research involving children and parents of children has shown that in the developing years up to the age of 18, what the child learns of life is mainly from the parents or other important adults. In the case of healthcare, children's understanding of what is beneficial when they are ill, but not necessarily what is not beneficial, is based on what the parents recommend and use. This insight is carried with children into adulthood and, more often than not, adopted in turn with their own children.

The issue of positive memories is highly relevant. There seems to be a potential strength in documenting and understanding strong collective memories of children, particularly those shared with various consumer products, and using these later in the development of products for the 'mature adults' market segment. These collective sensory/memory experiences are likely to be valuable through all life stages, and such understanding can help brands grow with their consumers, meeting their emotional needs as they mature and age. Enduring, successful products have achieved this, though probably by accident more than design. Imagine the potential if this were done intentionally.  

In some individual research cases, parents have been quite receptive to developments in healthcare, particularly around issues such as colic, influenza, vaccination, dentistry, corrective surgery and more profound conditions such as leukaemia and cancer. Yet, in other case studies, parents are often more willing to try alternative therapies when their children are sick, even in some severe cases. 

Apparent contradictions
These two scenarios seem to contradict each other, but when the data from all these relevant case studies are overlaid, as happens during data mining, interesting patterns start to emerge. Those children who grew up in families where conventional medicine was 'the norm' generally favour this approach when they become adults, irrespective of what they may learn about alternative remedies. The same applies for those who grew up in families where alternative healthcare was usual. It is quite possible for parents to embrace both, provided the drivers of change are clearly understood by the companies competing in the healthcare market.  

In research reviews, what competent researchers are looking for are patterns that are quite different from those discovered when analysing data from individual research projects. Not all researchers have the ability to identify these patterns within patterns. The required behaviour stems more from an innate curiosity rather than a function which is learned and, indeed, researchers keen on qualitative methods tend to have the right kind of curiosity.

Trying to establish a clear differentiation in areas of black and white, and identifying trends and insights that are not obvious to others, is the environment in which these researchers thrive. It is not the same as analysing using algorithms and models.  Qualitative data mining is akin to analytical cubism where the artist dissects the subject material then reconstructs it in a way that depicts its essence rather than its physical appearance.  

When undertaking research review projects, it is preferable to do the analysis manually, rather than using intelligence software. This is because there are often subtle differences in meaning and expression that can only be detected in this way. Unlike quantitative data, qualitative data sets have many more assumptions regarding drivers of attitudes and values.

Grouping ideas and concepts into themes enables the data to be thoroughly explored and cross-referenced to ensure nothing is missed. Since a much larger data set is being reviewed than would be the case with a specific product or category research project, there is always a wide range of different connections that occur. It is beneficial to identify what 'disruptions' are needed to occur to alter the rules of the categories being researched.  

When hindsight is coupled with the creation of a multi-dimensional strategic framework, the black and white spaces – the areas of practical application – can start to be separated from the theoretical 'grey space'.

This multi-dimensional framework helps to establish some suppositions regarding the best ways of tackling these areas, focusing on those which may have the greatest prospects. From the resulting strategic overview, a series of potential directions can be drafted for future exploration.  

Accurately defining the drivers of change is one of the advantages of this data-mining process. By overlaying data from several projects and viewing all the insights in a cross-brand/cross-category context, these drivers start to take shape. The drivers in one category may initiate a significant effect on other categories. These insights may not materialise during a single research project.

Guidance for future
Furthermore, the drivers of change over the preceding years may well give guidance to predict the shape of events that will occur in the future. This is an important issue for all marketing managers and one they should fully comprehend. Qualitative data mining, however, should not take precedence over the traditional focus group projects. It should be seen as an adjunct to conventional research. However, when budgets are tight, data mining may well achieve the intelligence necessary to keep the organisation moving forward.

The Author
Bryan Urbick is founder and chairman of Consumer Knowledge Centre