Is one study enough to prove a supplement works?

No. One study is a starting point, not proof. A single trial can be small, biased, or simply lucky, and a supplement ad that quotes one study tells you very little about whether the finding holds up. A systematic review that finds, appraises and combines every comparable trial is generally a more reliable guide, as long as the trials it combines are good ones.

This guide explains how the two kinds of evidence differ, what a large trial and a Cochrane review actually found for a few supplement questions, and how to judge the "scientific references" list on a sales page. It is general information, not medical advice. A doctor or pharmacist can help you weigh evidence for your own situation.

Why is a single study not enough?

A study can mislead for ordinary reasons. It may enroll too few people, so chance has more influence on the result. It may lack a placebo group. The researchers may test many outcomes and report the one that came out well. Ioannidis, in a widely cited 2005 paper in PLoS Medicine, argued that a research finding is less likely to be true when the studies in a field are smaller and when effect sizes are smaller, among other factors.

That does not mean research is worthless. It means a result gains weight when different teams, in different places, find something similar. A single positive trial is a reason to run more trials, not a reason to buy a bottle.

Sales pages often skip this step. They lead with the one study that sounds best, usually an animal study, a cell study or a trial of an isolated ingredient, and let the reader assume the product itself has been tested. The DoseAudit blog has a separate guide on animal and cell studies versus human trials.

What does a systematic review do differently?

The Cochrane Handbook describes a systematic review as an attempt to collate all the empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods chosen to minimize bias. Cochrane, an independent non-profit network, says on its own site that combining and comparing studies, sometimes hundreds of them, makes its reviews more reliable than individual studies.

In practice, reviewers decide in advance which question they will answer and which trials qualify. They search several databases, screen what they find, rate each trial for risk of bias, and then, where trials are similar enough, combine the numbers in a meta-analysis. Because the rules are written before the results are known, the reviewer cannot easily pick only the favorable trials.

A meta-analysis is the statistical part: it pools the results of several trials into one estimate with a confidence interval. A systematic review may or may not contain a meta-analysis. A plain "literature review" or a list of references on a sales page is neither.

How does the evidence ladder work?

People often draw evidence as a pyramid. From weakest to strongest, a rough order looks like this:

1. Opinions, testimonials and case stories.
2. Laboratory and animal studies.
3. Observational studies in people, where researchers watch what happens without assigning treatments.
4. A single randomized controlled trial (RCT), where participants are randomly assigned to the supplement or a placebo.
5. A systematic review of several RCTs, with or without a meta-analysis.

The ladder is a rule of thumb, not a law. A large, well-run RCT can be more informative than a systematic review made of small, flawed trials. The quality of the inputs matters as much as the label on the study.

What happens when a large trial or review meets the hype?

Two real examples show why the bigger picture matters.

Ginkgo biloba. Ginkgo is a common ingredient in memory formulas. The Ginkgo Evaluation of Memory (GEM) study, published in JAMA in 2008, was a randomized, double-blind, placebo-controlled trial of 3,069 adults aged 75 or older, followed for a median of 6.1 years. Participants took a 120 mg ginkgo extract twice a day or a placebo. The dementia rate was 3.3 per 100 person-years with ginkgo and 2.9 with placebo, and the authors concluded that ginkgo at 120 mg twice a day was not effective in reducing the incidence of dementia or Alzheimer disease. Note the scope: it tested dementia prevention in older adults, not every possible memory claim. But it shows how a large, long trial can sit very far from a confident ad.

Antioxidant vitamins. A Cochrane review of antioxidant supplements, updated in 2012, pooled 78 randomized trials with 296,707 participants. Overall, antioxidant supplements had no significant effect on mortality in the main random-effects analysis. In the 56 trials judged to have low risk of bias, the supplements significantly increased mortality (relative risk 1.04), and the authors concluded there was no evidence to support them for primary or secondary prevention. The point is not that every vitamin is harmful. The point is that individual small trials had pointed in more hopeful directions, and only pooling many trials and filtering them by quality exposed the pattern.

How do sales pages use single studies?

Here is what we found when we audited offers in the brain and memory category. Each item repeats what the review reports.

• The Mindoxa review found a "Scientific References" list made up of papers on sleep duration, shift work and obesity, unrelated to brain health or any ingredient in the capsule, and no amount for any of its mushrooms.
• The NeuroZoom review found a science section built mostly on studies of fluoride-exposed rodents plus market reports on the Alzheimer's drug industry, with no dose given for any of 35 ingredients.
• The Pineal Pure review found a list of roughly 30 references that mostly cover prostate health, kidney stones, iodine and disinfectants, not memory.
• The Java Brain review found a longer list of 37 references, but every study was on an isolated ingredient and none tested the finished formula.
• The ProMind Complex review found a full label, which is rare, with ginkgo at 50 mg against the 120 to 240 mg range the review cites for ginkgo studies, and vinpocetine at 2 mg against 10 to 40 mg.

None of these offers cites a systematic review of the finished product, and we did not find a systematic review of any of these finished formulas. A reference list, however long, is not the same thing as evidence that the product does anything.

A seven-question checklist for any study a seller quotes

1. Was it done in people, or in animals or cells?
2. Did it test this product, or only one ingredient on its own?
3. Was there a placebo or control group, and were participants randomly assigned?
4. How many people were in it, and for how long?
5. Did the dose in the study match the dose on the label?
6. Do independent teams and a systematic review report a similar result?
7. Does the reference list actually discuss the claim, or does it drift to unrelated topics?

For help reading a journal article, NIH's National Center for Complementary and Integrative Health runs a "Know the Science" series with a guide to making sense of a scientific journal article. Our own method is on the how we review page, and the reviews index shows how each offer scored.

FAQ

Is a meta-analysis always better than a single trial?

Not automatically. A meta-analysis can only be as good as the trials it combines. If the underlying studies are small, biased or very different from each other, the pooled number can still mislead. Look for the quality ratings and for whether the authors discuss heterogeneity.

What does "clinically proven" tell me?

By itself, very little. It does not tell you how many people were studied, whether there was a placebo, whether the study tested the finished product, or whether the dose matches the label. Ask which study, how large it was and what dose it used.

Where can I find systematic reviews on a supplement?

Search PubMed for the ingredient plus "systematic review" or "meta-analysis", and look at Cochrane reviews, which are indexed there. Reading the abstract is often enough to see how many trials were pooled and what the authors concluded.

If no systematic review exists, is the supplement useless?

No, but it means the evidence is thin. Missing evidence is not proof of no effect. It means you cannot rely on a promise, so check the dose, the refund policy and the interactions with your doctor or pharmacist before spending money.