The Problem with Panels

Statistics and data drive business decision-making for a very good reason – informed judgment is better than instinct or blind decision-making. Research and facts reduce risk and increase confidence.

Panels – of users, experts, or whatever – are fabulous research tools when one is attempting to “sample” an audience as a statistical approximation of the full audience. That is, if one wants to know what 5 million target customers think of X, one can randomly “sample” a subset (e.g., 300 people) and know within a certain degree of confidence what the full audience thinks.

Panels, however, have a fundamental problem when you are looking for something rare or unique (e.g., you are looking for knowledge or data that isn’t broadly disseminated). By definition, a panel is a subset, or a sampling, of a larger population. That creates a problem when you are asking a question not in the scope of knowledge or experience of a meaningful % of the overall population you are sampling from.

This “solution falls between the knowledge cracks” problem is most common when the area of knowledge is specialized, infrequently discussed in the trade, and/or may be an area of cutting-edge work done by small companies. There simply isn’t a social communication / knowledge sharing infrastructure that would cause the awareness of a solution to enter sufficient minds for a panel to be effective at finding the solution. That is to say that sending an email to an online panel and asking if they know of a solution to Y is ineffective and not what panels are designed to do.

Imagine for a moment that you are looking for a unique plant that comes from the top of Mt. Hicks. Or, even more practical, imagine you are looking to speak to someone who has climbed Mt. Hicks so they can tell you about the plants and perhaps provide you with an ongoing supply of harvested plants for a new product Z that needs these plants as a key ingredient of the overall product. If Mt. Hicks is well known, and the properties of its plants are well-known, it is likely and plausible that querying a panel can lead you to an answer. But, conversely, if it’s not well known, the panel is likely to be unhelpful beyond a few members offering to drive to Mt. Hicks and pick up some plants for you. Which is both (a) expensive, and (b) likely to not be a long-term solution.

Panels are great for sampling, and they’re great when you want to “challenge” a group that loves to solve challenges.

Panels are not so great at finding hard to find things (and when finding that hard thing takes a lot of time and energy)…