Outcome Measures: the unspoken problem in natural product research

Natural products and dietary supplements are designed to keep healthy people healthy–or even make healthy people healthier. The literal distinguishing factor between dietary supplements and drugs is that healthy participant target market. Yet, despite that critical and substantial difference, many natural product studies use borrowed and sloppy methods, straight from the drug industry.

In fact, one of the biggest reasons clinical trials for natural products fail is that researchers studying these products borrow measurement tools from the pharmaceutical industry. Measurement is a critical component of trial development; it's what captures your product's effects in the human body. 

You've likely heard bakers discuss the importance of weighing ingredients rather than using measuring cups; you get a more precise measurement and therefore, a better cake. Or parents explaining how difficult it is to weigh a baby at home using your usual household scale which was designed for adults. In our everyday lives, we constantly have to make measurement decisions: do we need a measuring tape or a scale? A measuring cup or a ruler? But when it comes to natural product clinical research, measurement is often neglected. And the results are often just as disastrous as you would get trying to use your bathroom scale to measure ingredients for a batch of cookies. 

For example, did you know that the most common tools for measuring supplements which enhance memory and offer cognitive support are actually designed as dementia screening tools? The MMSE (Mini Mental State Examination) and the MoCA (Montreal Cognitive Assessment) are used widely in clinical research for brain boosting supplements, but are incapable of capturing actual brain boosting effects. 

To see this in action, click here to view the MMSE. Take a few minutes to self-administer the test. Assuming you successfully recognized a wristwatch and identified your physical location, you scored fairly well. But not having dementia doesn't mean your brain can't use a boost or that your memory is flawless. So your baseline score is about as high as the measurement can go. You're topping out the scale. 

Let's presume you took this survey around the office and administered it to 5-10 people. The top score for this assessment is 30. The average score is likely to be in the high 20s, allowing for some everyday mistakes. Your entire office is at the top of the scale. Yet, presumably, none of you believe you can't benefit from brain boosting effects. 

Now, let's imagine that you take a brain support supplement for 3 months. Each of you feel more alert. You're noticing that you make fewer mistakes at work. Your whole team is more productive. Clearly this product is useful. 

So you all take the scale again. Yet your scores haven't changed much, if at all. Maybe the average is one point higher, but having already topped out the scales, you had no room for growth. Measurement experts call this the ceiling effect. 

When those scores are statistically analyzed, one of two things will happen: 

1. You'll have a false negative. The scales were incapable of capturing the growth in cognitive function. Perhaps the average increased by a point or two, but unless the study was powered to detect tiny, insignificant changes (wasting your valuable research funds), the boost was too small to be statistically significant. Your entire project failed; not only are these research funds lost, but now you have a negative study on your product. 

2. You'll reach statistical significance with a tiny effect size. The trial will technically be deemed a "success," but it won't support the true size of your product's brain boosting abilities. And reaching significance with such a small effect means your trial was overpowered for your product's abilities. So you've wasted valuable research funds and the only evidence you've produced dramatically understates your product's actual effects. Clinicians and decision-makers will ignore your product in favor of one with a larger actual effect on the human body. 

What can be done? 

Clinical trials for the natural products industry need to use methodology that is specific to the natural products industry. Rather than borrowing screening tools designed to identify patients who have dementia, we need measurements that capture what brain support looks like in an already healthy person. Your trials need to be free of ceiling effects so that your product's entire potential can be measured. 

This example focuses on brain boosting supplements and dementia, but this process is alive and well across the field of natural products. Menstrual health supplements are measured using screening tools for pre-menstrual dysphoric disorder (PMDD), a medical condition. Sleep quality products are studied using insomnia testing tools. In each of these scenarios, ceiling effects will cause your clinical trial to either fail unnecessarily or artificially reduce the evidence of your product's efficacy. 

The solution is a suite of measurement tools which are designed specifically for the natural products industry. Tools which are comprehensive enough to capture the wide range of benefits your produce delivers, which were developed by integrative health professionals, and which were tested and refined on a sample of healthy individuals.