In the coming week, the BHA will be providing detailed rebuttals to the ill-conceived recommendations found in the Science & Technology Committee’s “Evidence Check 2: Homeopathy” issued on 22 February 2010.
Response to the report’s criticisms of the evidence submitted by the BHA
This document addresses the report’s comments on Professor Edzard Ernst’s stated opinions about details in the BHA’s summary of clinical research evidence in homeopathy. We provide a detailed rebuttal of the statements made in paragraphs 66–69 of the report:
66. We received conflicting opinions on whether homeopathic products are efficacious (that is, whether they work better than a placebo treatment). The British Homeopathic Association (BHA) told us that:
Four out of five comprehensive systematic reviews of RCTs in homeopathy have reached the qualified conclusion that homeopathy differs from placebo.
67. Professor Edzard Ernst, Director of the Complementary Medicine Group at the Peninsula Medical School, disputed this summary of the evidence in detail. The systematic reviews to which the BHA refers are: Kleijnen et al, 1991;Boissel et al, 1996;Cucherat et al, 2000;Lindeet al, 1997;and Shang et al, 2005.
It seems to have escaped the committee’s attention that the BHA explicitly finds only limited value in the findings of comprehensive systematic reviews. Moreover, our expression “qualified conclusion”, quoted in para 66 above, makes it perfectly clear we believe that none of the comprehensive systematic reviews has yielded unequivocal conclusions. In the interests of completeness, the BHA submission includes a brief summary of these reviews. Our submission also makes clear, for example, that Linde’s 1997 analysis failed to draw conclusions “about the efficacy of homeopathy for any specific medical condition”. A principal feature of our submission is the evidence available in condition-specific systematic reviews, highlighting the fact that these have variously reported positive, negative and non-conclusive findings.
Professor Ernst pointed out that:
1. The Kleijnen review is now 18 years old and thus outdated.
Obviously we do not claim otherwise. The findings reflected the balance of evidence available at the time.
2. Boissel et al merely combined p-valuesof the included studies. This article is now also outdated. Furthermore it is not unambiguously positive.
Again, it is self-evident that we acknowledge the contemporary context of the review’s findings. Having referred to “qualified conclusions” above, we explicitly make no claim that the review was “unambiguously positive”. The authors state, “From the available evidence it is likely that among the tested homoeopathic approaches some had an added effect over nothing or placebo. However, it is not possible to identify which one.” The mean P value from 17 comparisons was 0.00027.
3. Cucherat et al is the publication of the Boissel document which was a EU-sponsored report. [The authors themselves noted that “there is some evidence that homeopathic treatments are more effective than placebo; however, the strength of this evidence is low because of the low methodological quality of the trials.”]
Compared with Boissel’s article, Cucherat’s paper extended the sample size for data analysis by some 578 patients. We repeat: the authors’ comments highlighted above by Professor Ernst led to our inclusion of the phrase “qualified conclusion” in the BHA’s submission: “Four out of five comprehensive systematic reviews of RCTs in homeopathy have reached the qualified conclusion that homeopathy differs from placebo”.
4. Linde et al has been re-analysed by various authors, including Linde himself, and all of the 6 re-analyses (none of which were cited in the BHA’s submission) have come out negative.
We re-emphasise the points made above about the summary format of the BHA submission and again, crucially, the lesser value we attribute to the findings of comprehensive systematic reviews of heterogeneous RCTs.
The principal re-review of Linde’s 1997 analysis was by Linde himself in 1999: that paper yielded statistically positive results no matter how the original RCTs were analysed. The 1999 paper also acknowledged that the 1997 analysis “at least overestimated the effects of homeopathic treatments”. As in all medical research, better quality trials were more likely to be negative, but for homeopathy this relationship was non-linear. Nor did the 1999 paper refute the main conclusion of the 1997 study: “The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homeopathy are completely due to placebo”.
Another re-review by Linde (1998) focused only on a sub-set of RCTs of individualised homeopathy; it did not re-analyse the full data set of the original 1997 article. Three of the other four re-reviews highlighted by Professor Ernst (Ev 52) are minor and/or non-peer-reviewed papers by Professor Ernst himself or by Professor Matthias Egger, each a well-known detractor of homeopathy. The fifth re-review (Morrison, Lilford and Ernst, 2000) included a recommendation for large multi-centre trials in homeopathy.
5. Shang et al very clearly arrived at a devastatingly negative overall conclusion.
Professor Ernst’s conclusion about the Shang review is greatly overstated. Only by narrowing its analysis to 8 “larger studies”, and assuming homeopathy works for every medical condition, was the Shang analysis able to draw negative conclusions about homeopathy. Moreover, it is hardly reasonable to damn the whole of homeopathy based on a conclusion from 8 diverse RCTs! Shang’s assessment of “higher quality” was restricted to key internal validity issues; it paid no attention to what extent the 8 RCTs’ findings possessed external validity (i.e. if their results could be applied in the “real world”). In any event, the BHA’s submission to the committee accurately stated that Shang concluded there was “weak evidence for a specific effect of homeopathic remedies” (Ev 37). Given the above, the review’s overall conclusion (that their findings were “compatible with the notion that the clinical effects of homoeopathy are placebo effects”) cannot possibly be regarded as “devastatingly negative”. The rigour and validity of Shang’s conclusions have been directly challenged (Lüdtke R, Rutten AL (2008). The conclusions on the effectiveness of homeopathy highly depend on the set of analyzed trials. Journal of Clinical Epidemiology,61: 1197–1204).
68. Professor Ernst also commented on the BHA’s claims about reviews that offered positive reviews for allergies,upper respiratory tract infectionsand rheumatic diseaseswere equally flawed: the “review” on allergies was a lecture series, not a systematic review; the “reviews” on upper respiratory tract infections were health technology assessments, not systematic reviews, and mostly contained uncontrolled data; and the “review” on rheumatic diseases was not conclusive.Finally, he pointed out that the BHA had omitted several systematic reviews and meta-analyses, each of which “must have been known to the BHA” and “all of them arrived at negative conclusions”.
It is disingenuous to contest the BHA’s inclusion of the first-named three reviews based on study design and/or study findings:
The review on allergies was indeed labelled “Lecture Series” by the journal, but it is a 13-page paper in a peer-reviewed publication that systematically appraised the findings of RCTs, leading to a detailed evaluation of quality of evidence.
The methods of the review on upper respiratory tract infections were summarised by the authors as follows: “Databases accessible by Internet were systematically searched, complemented by manual search and contacts with experts, and evaluated according to internal and external validity criteria”. Of 17 primary studies reviewed in the diagnosis “upper respiratory tract infections”, 11 (the majority) were controlled trials.
As Professor Ernst himself has summarised in a supplementary submission to the Committee (Ev 32), the review on rheumatic diseases concluded that “homeopathic remedies work better than placebo” (a positive conclusion), AND that there was “not enough evidence for any specific condition to allow reliable assessment” (non-conclusive). It is reasonable to draw a conclusion of “positive” and/or “non-conclusive”, based on very different attributes of the review.
The Committee’s report has referred to Professor Ernst’s note that “the BHA had omitted several systematic reviews and meta-analyses”. On scrutinising Professor Ernst’s additional submission to the Committee (Ev 52), it becomes clear that he is actually citing 5 reviews: 4 of those reviews are precisely the same re-analyses of Linde’s 1997 paper that were discussed in para 67.4 above! It is wholly misleading that the report implies the BHA had “omitted” an unspecified number of further reviews in other areas of homeopathy.
The fifth review (Ernst, 1999) listed in Ev 52 is a systematic review of classical homeopathy versus conventional treatments. No clear trend emerged from 6 eligible trials (2 positive, 2 negative, 2 inconclusive), each of which studied a different medical condition. It is surprising to see Professor Ernst referring to it above as a “negative” review, for he himself did not come to that conclusion in the original paper. Its summary states: “A clear trend is not visible. It is concluded that at present the relative efficacy of homoeopathic remedies is not known.” The conclusion in the main text of the paper states: “the value of individualized homoeopathy relative to allopathic treatments in unknown.” With the necessity for brevity in our submission to the Committee, we chose not to include this particular review because it did not focus on placebo-controlled trials.
69. The review which we consider the most comprehensive to date is that by Shang et al. The review compared 110 placebo-controlled trials of homeopathy matched according to disorder and type of outcome to trials of conventional medicine. The study only included trials that were controlled, included randomised assignment to treatment or placebo groups and were accompanied by sufficient data for odds ratio calculations. The authors concluded that “when analyses were restricted to large trials of higher quality there was no convincing evidence that homeopathy was superior to placebo”.
See our comments at Para 67.5 above. As Dr Robert Mathie, the BHA’s Research Development Adviser, pointed out to the Committee during the hearing on 25 November 2009, the methods of meta-analyses and the interpretation of findings in areas of research that possess small and heterogeneous data sets (such as complementary medicine) vary greatly according to the perspective of the individual (Ev 47). The discrepancy of opinion among individual readers of the Shang review’s methods and conclusions is an extreme example of this situation. There is no good reason to accept, as final authority, the interpretation of one opinion-leader in homeopathy research, such as Professor Ernst, over that of any other, such as Dr Mathie or Dr Peter Fisher.