Current deficiencies of discussion sections
Accounting for the totality of evidence
Overinterpretation and misinterpretation of results
Lack of appropriate discussion of limitations
Potential solutions proposed to date
An alternative approach: an independent second discussant
Ideal attributes of an independent discussant
Barriers to implementation
Testing whether discussions are improved
An example of an independent discussion
|State main findings—This does not mean a repetition of all the results with their statistics. It should provide a concise overview of the study. For example, ‘Drug X produced a greater haemodynamic change on induction of anaesthesia than occurred with drug Y, resulting in a greater fall in arterial pressure and a higher incidence of tachycardia’. Do not repeat data already presented in the Results, figures, and tables.|
|Relation to previous studies—This section should relate directly to the statements made in the Introduction and qualify your findings in relation to previous studies of the subject. Mention any important uncertainties in the methods of measurement. In laboratory studies, try to relate the concentrations used to those encountered clinically.|
|Why are they different/same—Avoid excessive speculation or speculation beyond your results. It is quite reasonable to suggest possible explanations for your findings and any differences from previous studies, but the 'missing parts' of such reasoning must be acknowledged.|
|Additions to knowledge of the subject—Summarise previous sections by pulling together the implications of your main findings, studies by other workers and their combined contribution to our knowledge of the subject. This should not be just another repetition of the results and preceding discussion but more of an expanded conclusion. A meta-analysis incorporating the results of the new study and previous relevant studies could be included in this paragraph.|
|Weaknesses and strengths in study—This is the most important part of the Discussion. Acknowledge any limitations of your study at this point. Examples here could include the generalisability of the patient population, patients lost to follow-up (or missing data in general), and limitations of analytical test. Authors are advised to be honest but succinct in this section. It is also reasonable to comment briefly on the particular strengths of the study, especially in comparison with comparable studies.|
|Future studies—Identify future studies that would address some of the potential explanations and limitations discussed earlier. This section should be very brief.|
|Conclusions—The original contribution to knowledge from the present study should be stated. A common fault here is to overstate the findings from a study. It may be appropriate to give the implications of the conclusions for (anaesthetic) practice and the indications for further enquiry in this area of interest.|
|Title of Manuscript: Effect of Haloperidol on Survival Among Critically Ill Adults With a High Risk of Delirium: The REDUCE Randomized Clinical Trial|
Original Discussion: ‘In this large multicenter double-blind randomized clinical trial involving critically ill adults at high risk of delirium, there was no significant difference in the number of days survived at 28 days following inclusion between patients who received prophylactic haloperidol therapy and patients who received placebo. The pathophysiological mechanism of delirium is poorly understood. Delirium is considered a multifactorial disorder and many different pathways to its occurrence have been postulated resulting in many hypotheses. The fact that the average of 11 risk factors are present at the same time in ICU patients with delirium suggests the involvement of multiple pathways in its development. Therefore, it seems plausible that a mediator that alters causal pathways of delirium may be helpful. Haloperidol is an antipsychotic agent with antidopaminergic, antiadrenergic, limited anticholinergic properties and possibly has anti-inflammatory effects that potentially antagonizes multiple pathways of delirium.’
Independent Discussion: Delirium is associated with death. Haloperidol might prevent delirium in critically ill patients. Therefore, if haloperidol prevents delirium, it might also prevent death, assuming that delirium causes death. There was no evidence in this study that haloperidol compared with placebo increases or decreases 28 day mortality. Based on a precision estimate (95% confidence interval for the hazard ratio of 0.78–1.30), the results of this study remain consistent with a clinically meaningful decrease or increase in 28 day mortality attributable to haloperidol.
Commentary: The original investigators provided rationale for the study in relation to delirium, but did not provide rationale in relation to the primary outcome, which was mortality. The summary of the main finding was the same (i.e. that there is no difference in mortality), although the independent discussant acknowledged also that clinically meaningful changes in mortality cannot be excluded.
|Relationship of Main Finding to Previous Studies|
Independent Discussion: Based on systematic review of the scientific literature, there is no previous evidence on the effectiveness of haloperidol prophylaxis in preventing mortality in critically ill patients. In other settings (e.g. those with myocardial infarction, dementia, or in nursing homes), there is some evidence that prophylactic haloperidol is associated with increased mortality. Overall, the evidence for a causal link between haloperidol and either increased or decreased mortality is weak.
Commentary: In the original manuscript the Discussion did not contextualise the main finding in relation to previous studies, perhaps because this specific question has not previously been addressed.
|Additional (Secondary) Findings|
Original Discussion: ‘Also, no significant differences were found in the number of days patients survived at 90 days between the haloperidol group and the placebo group. No differences were found for any other reported secondary end points. Furthermore, across predefined subgroups, the lack of a prophylactic effect was very consistent. Prophylactic haloperidol therapy was not associated with haloperidol-induced adverse effects.’
Independent Discussion: There were no differences between groups in 90 day mortality, safety outcomes, delirium incidence, or any delirium related outcome. Higher APACHE-II (severity of acute illness) and PRE-DERELIC (delirium prediction) scores were independently associated with increased delirium risk and decreased survival. Delirium incidence was high (33%), as expected, in these critically ill patients. Surprisingly, the side-effects anticipated with haloperidol (sedation and prolonged QTc) were not seen in this study.
Commentary: The summary of additional findings was similar in both Discussions. In both Discussions, surprise was expressed that there were no apparent side-effects or safety concerns with prophylactic (low dose) haloperidol. The independent Discussion addressed some predictive analyses that were not discussed in the original.
|Relationship of Additional (Secondary) Findings to Previous Studies|
Original Discussion: ‘Haloperidol has been the first-line drug of choice to treat delirium for decades despite the lack of evidence that haloperidol is effective. For this reason, the Society of Critical Care Medicine in its last guideline on pain, agitation, and delirium did not recommend the use of haloperidol for treatment or for delirium prevention for critically ill adults. However, several ICU studies have evaluated possible prophylactic effects of haloperidol but demonstrate contradictory effects. In one randomized clinical trial, postoperative ICU patients received a maximum of 1.2 mg of haloperidol and showed a reduced delirium incidence and more delirium-free days. Another randomized clinical trial involving severely ill medical ICU patients receiving 2.5 mg 3 times daily, showed no beneficial effect, and no effect was found in reducing subsyndromal delirium with prophylactic haloperidol. Although a previous before–after study showed clinical relevant and favorable effects in a similar group of high-risk and critically ill adults, these beneficial effects could not be replicated in the current randomized clinical trial. However, the findings of this study corroborate the findings of other randomized clinical trials involving critically ill adults.’
Independent Discussion: Studies have generally not found that haloperidol is effective in preventing or treating delirium.
50Specifically, the evidence for haloperidol being effective at preventing delirium is sparse. Thus, the results of this trial are consistent with previous randomised trials.
Commentary: There is agreement between the Discussions that the findings in relation to delirium prevention are largely consistent with the findings of previous randomised trials.
Original Discussion: ‘This study has several limitations. First, the 1 mg haloperidol group was terminated early, as a predefined consequence of the adaptive design, which is considered a strength of our study. This discontinuation did not affect the presented study findings. Second, the duration of prophylactic therapy (median, 2 days) could be too short to prevent delirium and its deleterious outcome. It cannot be excluded that longer exposure to haloperidol may be needed to influence patient outcome. However, subgroup analysis in patients treated for more than 2 days also did not show any beneficial effect. Third, the dose may have been too low. In a before–after study a 1 mg dose every 8 hours demonstrated beneficial effects without relevant adverse effects. For this reason, using a higher dose was included as part of the current study, similar to the study of Page et al. In both haloperidol dosage groups no beneficial effects were found. Fourth, it was not feasible to collect data for all secondary outcome measures in some centers due to research staff limitations. However, the median number of delirium- and coma-free days between both groups did not differ; therefore, collecting these data in a somewhat smaller group did not affect the results. Fifth, the study population included severely ill ICU adults, whose brains may have been too seriously affected for haloperidol to exert a prophylactic effect, since in non-ICU adults, prophylactic haloperidol may have beneficial effects. But the subgroup of patients with a low severity of illness score also demonstrated no beneficial effects. Nevertheless, it cannot be ruled out that delirium may be more easily and favorably affected in non-ICU adults than critically ill ICU adults. Sixth, the results of the long-term quality of life are not included herein, and therefore the effects of haloperidol on quality of life remain to be determined.’
Independent Discussion: A single study seldom provides definitive evidence,
52and although this was a ‘negative’ study (there were no estimated differences between the groups in the primary or secondary outcomes), the 95% confidence intervals for differences in both 28 day mortality (–3.4%–4.6%) and delirium (–4.6%–5.4%) included clinically important both benefit and harm. Because haloperidol was administered open label for treatment of delirium in both groups, its potential effectiveness for treatment could not be determined in this study. Delirium was not assessed in all the participating sites resulting in missing assessments; thus, the effective sample size for the delirium outcomes was reduced. It is possible that the dose of haloperidol was insufficient or that the time required for effective prophylaxis was insufficient. The hypothesis underpinning this study appears somewhat implausible. In order for haloperidol to prevent death, it would have to be effective first in preventing delirium, and second delirium would have to be causally implicated in (rather than merely being associated with) mortality. Delirium is a complex syndrome that is likely to have multiple pathophysiological pathways
53; it is therefore improbable that any single intervention will have a marked impact on delirium incidence. Thus, the pre–study likelihood of a positive primary outcome with the intervention was low.
Commentary: The main difference in emphasis was that the independent discussant noted the imprecision around the estimates for death and delirium. The independent discussant therefore inferred that this study should not be regarded as providing definitive evidence in relation to these questions. The independent discussant also felt that the initial premise behind this study was not compelling; there was no strong biological plausibility for the hypothesis. Both the independent Discussion and the original listed several limitations, with some overlap, but also differences, and these limitations can form a complementary list.
Original Discussion: ‘The large sample size of the current study allowed us to perform several sensitivity analyses, confirming the lack of effect across the different subgroups.’
Independent Discussion: This was a rigorously conducted, large multi-centre pragmatic trial. The protocol for the study was thorough and comprehensible. The study was pre-registered, and the protocol was published ahead of time. Overall there were few deviations from the protocol and outcomes were unchanged with per-protocol analysis. The adaptive design allowed for a more efficient study.
All these factors increase confidence in relation to the robustness of the findings.
Commentary: The independent discussant highlighted more strengths of the study than the investigators themselves did. The lists of strengths in the two Discussions are complementary.
Independent Discussion: It will be important to clarify the pathophysiology of delirium so that mechanistically targeted treatments can be developed and tested
Original Discussion: ‘Among critically ill adults with high-risk of delirium, the use of prophylactic haloperidol compared with placebo did not improve 28-day survival. These findings do not support the use of prophylactic haloperidol in critically ill adults.’
Independent Discussion: It is unlikely that prophylactic haloperidol for critically ill patients on intensive care units is effective for preventing either death or delirium.
Commentary: The take-home message was the same.
Commentary: Overall, there is considerable concordance between the two Discussions, and inferential reproducibility is generally endorsed. There is some discordance regarding the extent to which the ‘negative’ findings are interpreted as being ‘definitive’, and also in relation to the biological plausibility of the hypothesis on which this study was predicated.
Declarations of interest
- Policy: NIH plans to enhance reproducibility.Nature. 2014; 505: 612-613
- Relevance relations for the concept of reproducibility.J R Soc Interface. 2014; 11: 20131030
- What does research reproducibility mean?.Sci Transl Med. 2016; 8: 341ps12
- Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents?.JAMA. 1998; 280: 280-282
- Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report.J R Soc Med. 2007; 100: 187-190
- Discussion sections in reports of controlled trials published in general medical journals.JAMA. 2002; 287: 2799-2801
- Of P-values and Bayes: a modest proposal.Epidemiology. 2001; 12: 295-297
- Toward evidence-based medical statistics: 1. The P value fallacy.Ann Intern Med. 1999; 130: 995-1004
- The association between exaggeration in health related science news and academic press releases: retrospective observational study.BMJ. 2014; 349: g7015
- Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study.PLoS Med. 2012; 9 (e1001308)
- Improving the quality of reporting of randomized controlled trials. The CONSORT statement.JAMA. 1996; 276: 637-639
- Association between success rate and citation count of studies of radiofrequency catheter ablation for atrial fibrillation: possible evidence of citation bias.Circ Cardiovasc Qual Outcome. 2014; 7: 687-692
- Don’t let the truth get in the way of a good story: an illustration of citation bias in epidemiologic research.Am J Epidemiol. 2014; 180: 446-448
- Citation bias favoring statistically significant studies was present in medical research.J Clin Epidemiol. 2013; 66: 296-301
- Overinterpretation of clinical applicability in molecular diagnostic research.Clin Chem. 2009; 55: 786-794
- Overinterpretation and misreporting of diagnostic accuracy studies: evidence of “spin”.Radiology. 2013; 267: 581-588
- Data interpretation in analgesic clinical trials with statistically nonsignificant primary analyses: an ACTTION systematic review.J Pain. 2015; 16: 3-10
- Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes.JAMA. 2010; 303: 2058-2064
- Presence of spin in the abstracts and titles of anaesthesiology randomized controlled trials.Br J Anaesth. 2019; 122: e13-e14
- Factors associated with findings of published trials of drug–drug comparisons: why some statins appear more efficacious than others.PLoS Med. 2007; 4: e184
- Industry sponsorship and research outcome.Cochrane Database Syst Rev. 2012; 12: MR000033
- A dirty dozen: twelve p-value misconceptions.Semin Hematol. 2008; 45: 135-140
- Limitations are not properly acknowledged in the scientific literature.J Clin Epidemiol. 2007; 60: 324-329
- There was less self-critique among basic than in clinical science articles in three rheumatology journals.J Clin Epidemiol. 2014; 67: 654-657
- Manuscript quality before and after peer review and editing at Annals of Internal Medicine.Ann Intern Med. 1994; 121: 11-21
- All that glitters isn't gold: a survey on acknowledgment of limitations in biomedical studies.PLoS One. 2013; 8e73623
- Discussing study limitations in reports of biomedical studies—the need for more transparency.Health Qual Life Outcome. 2012; 10: 23
- Users' guide to detecting misleading claims in clinical research reports.BMJ. 2004; 329: 1093-1096
- Inadequate post-publication review of medical research.BMJ. 2010; 341: c3803
- Reproducible research practices and transparency across the biomedical literature.PLoS Biol. 2016; 14 (e1002333)
- Sharing clinical trial data: a proposal from the International Committee of Medical Journal Editors.PLoS Med. 2016; 13 (e1001950)
- Preparing raw clinical data for publication: guidance for journal editors, authors, and peer reviewers.Trials. 2010; 11: 9
- Increasing value and reducing waste in research design, conduct, and analysis.Lancet. 2014; 383: 166-175
- Reproducible research: moving toward research the public can really trust.Ann Intern Med. 2007; 146: 450-453
- Reporting of observational research in anesthesiology: the importance of the analysis plan.Anesthesiology. 2016; 124: 998-1000
- A catalogue of reporting guidelines for health research.Eur J Clin Invest. 2010; 40: 35-53
- How to make more published research true.PLoS Med. 2014; 11 (e1001747)
- Clarity and strength of implications for practice in medical journal articles: an exploratory analysis.BMJ Qual Saf. 2011; 20 (i52–7)
- The fickle P value generates irreproducible results.Nat Methods. 2015; 12: 179-185
- Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting.Lancet. 2010; 376: 20-21
- Word associations contribute to machine learning in automatic scoring of degree of emotional tones in dream reports.Conscious Cogn. 2011; 20: 1570-1576
- Towards NLP-based psychological evaluation.AMIA Annu Symp Proc. 2007; : 1005
- A hybrid approach to sentiment sentence classification in suicide notes.Biomed Inform Insights. 2012; 5: 43-50
- A Fuzzy computing model for identifying polarity of Chinese sentiment words.Comput Intell Neurosci. 2015; 2015: 525437
- Effect of haloperidol on survival among critically ill adults with a high risk of delirium: the REDUCE randomized clinical trial.JAMA. 2018; 319: 680-690
- Use of haloperidol versus atypical antipsychotics and risk of in-hospital death in patients with acute myocardial infarction: cohort study.BMJ. 2018; 360: k1218
- Antipsychotics, other psychotropics, and the risk of death in patients with dementia: number needed to harm.JAMA Psychiatry. 2015; 72: 438-445
- Comparative mortality risks of antipsychotic medications in community-dwelling older adults.Br J Psychiatry. 2014; 205: 44-51
- The mortality risk of conventional antipsychotics in elderly patients: a systematic review and meta-analysis of randomized placebo-controlled trials.J Am Med Dir Assoc. 2015; 16: 817-824
- Effectiveness of haloperidol prophylaxis in critically ill patients with a high risk of delirium: a systematic review.JBI Database Syst Rev Implement Rep. 2017; 15: 1440-1472
- Synthesizing evidence: shifting the focus from individual studies to the body of evidence.JAMA. 2013; 309: 2217-2218
- In the era of systematic reviews, does the size of an individual trial still matter.PLoS Med. 2008; 5: e4
- Delirium in elderly people.Lancet. 2014; 383: 911-922
- Scientific method: statistical errors.Nature. 2014; 506: 150-152
This article is accompanied by an editorial: Who watches the watchmen and the problem of recursive flea bites by J. R. Sneyd, Br J Anaesth 2019:122:407–408, doi: https://doi.org/10.1016/j.bja.2018.11.013.
User LicenseElsevier user license |
For non-commercial purposes:
- Read, print & download
- Text & data mine
- Translate the article
- Reuse portions or extracts from the article in other works
- Redistribute or republish the final article
- Sell or re-use for commercial purposes
Elsevier's open access license policy