Abstract

Clinicians regularly have to trade benefits and harms to choose between testing and treatment strategies. This process is often done by making global and implicit judgments. A decision analysis is an analytic method that makes this process more explicit, reproducible, and evidence-based. While clinicians are unlikely to conduct their own decision analysis, they will read publications of such analyses or use guidelines based on them. This review outlines the anatomy of a decision tree and provides clinicians with the tools to critically appraise a decision analysis and apply its results to medical decision making. Clinicians reading about a decision analysis can make two judgments. The first judgment is about the credibility of the methods, such as whether the decision analysis addressed a relevant clinical question, included all important outcomes, used the current best evidence to derive variables in the model, and adopted the appropriate time horizon. The second judgment is about rating confidence in the preferred course of action by determining the certainty in the model variables, whether the results are robust in sensitivity analyses and if the results are applicable to a specific patient. Results from a valid and robust decision analysis can inform both guideline panels and the patient-clinician dyad engaged in shared decision-making.

Abbreviations and Acronyms: ICER (incremental cost-effectiveness ratio), IGRA (interferon-gamma release assay test), QALY (quality-adjusted life year), TB (tuberculosis), TST (tuberculin skin test)Article Highlights•

This guide provides clinicians with the tools to critically appraise a decision analysis and apply its results to medical decision making.

•

We describe how clinicians can judge the credibility of the methods of a decision analysis and rate the confidence in the preferred course of action.

•

The outlined principles can also be applied to appraise a cost-effectiveness analysis.

•

We use a clinical scenario and analyze a published decision analysis to demonstrate the practical application of this guide.

Clinical ScenarioYou are a family physician who sees a 54-year old man, born in India, who has lived in the United States for the past 3 years and was recently diagnosed with diabetes mellitus. You are aware that India has a high incidence of tuberculosis and therefore your patient could be infected with Mycobacterium tuberculosis (latent tuberculosis [TB] infection). Moreover, you are aware that patients with diabetes have a two- to four-fold increased risk of developing active tuberculosis.1Al-Rifai R.H. Pearson F. Critchley J.A. Abu-Raddad L.J. Association between diabetes mellitus and active tuberculosis: a systematic review and meta-analysis. You wonder if you should test your patient for latent TB using an interferon-gamma release assay test (IGRA) or a tuberculin skin test (TST) and prescribe preventive treatment if the test is positive.To inform your discussion, you search first for an evidence-based recommendation and find a guideline indicating that there is only low-quality evidence addressing latent TB screening and treatment in immigrants from high-TB-burden countries. The same guideline concludes that the benefits of systematic and routine latent TB testing and treatment in patients with diabetes “do not outweigh the risks unless individuals/patients also belong to the groups mentioned in the above recommendations [including immigrants from high burden countries].”As your patient is an immigrant from a high-TB-burden country and also has diabetes, you conclude that your patient might potentially benefit from latent TB testing and treatment. However, because the evidence is low quality, you explore further and find a decision analysis with cost-effectiveness measures addressing latent TB testing and treatment in residents born outside the United States with and without medical comorbidities.3Tasillo A. Salomon J.A. Trikalinos T.A. Horsburgh Jr., C.R. Marks S.M. Linas B.P. Cost-effectiveness of testing and treatment for latent tuberculosis infection in residents born outside the United States with and without medical comorbidities in a simulation model. You ascertain that there is reasonable match between the decision problem examined in the analysis and your clinical question. How should you use this decision analysis/cost-effectiveness analysis to help guide you and your patient in deciding whether to pursue the screening option?Introduction

Medical decisions are often complex, and clinicians must regularly weigh the pros and cons of interventions in individual patients. Even treatments that have been found to be effective in large trials are associated with adverse events, and it may not always be apparent whether the benefits (in this case reduced risk of developing TB with treatment of latent TB) outweigh the potential harms (here, receiving treatment based on a false-positive test result, or having an adverse effect, such as drug-induced hepatitis, from latent TB treatment).

Clinicians often assess such risk-benefit decisions based on their intuition and often make decisions based on mental shortcuts, also called heuristics.4Judgment under uncertainty: heuristics and biases. This form of intuitive decision-making, although often helpful, can lead to inaccurate assessment of the likely harms and benefits of an intervention.

Decision analysis addresses medical decision-making by providing a systematic approach to evaluating alternative strategies using objective evidence and careful, model-based evaluation. High-quality medical decision analyses and cost-effectiveness analyses provide decision makers — both clinicians and their patients as well as those involved in health policy — with guidance regarding complex decision problems. Although clinicians will seldom construct their own decision trees when faced with a clinical conundrum, the insights they gain from understanding medical decision analysis will help them to critically reflect on their own decision making.

The purpose of this users’ guide is to introduce the fundamental concepts of decision analysis and cost-effectiveness analysis and provide users with the tools to appraise studies using these analyses. In-depth descriptions of decision modeling techniques can be found elsewhere.5Caro J.J. Briggs A.H. Siebert U. Kuntz K.M. Force I-SMGRPT
Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—1., 6Roberts M. Russell L.B. Paltiel A.D. et al.Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—2., 7Siebert U. Alagoz O. Bayoumi A.M. et al.State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—3., 8Karnon J. Stahl J. Brennan A. et al.Modeling using discrete event simulation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—4., 9Pitman R. Fisman D. Zaric G.S. et al.Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--5., 10Briggs A.H. Weinstein M.C. Fenwick E.A.L. et al.Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—6., 11Eddy D.M. Hollingworth W. Caro J.J. et al.Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—7. In 1995, Richardson et al12Richardson W.S. Detsky A.S. Users' guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid? Evidence-Based Medicine Working Group.,13Richardson W.S. Detsky A.S. Users' guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients? Evidence Based Medicine Working Group. developed a users’ guide (in two parts) on how to use a “clinical decision analysis” and presented a framework for critical appraisal of clinical decision analyses.12Richardson W.S. Detsky A.S. Users' guides to the medical literature. VII. How to use a clinical decision analysis. A. Are the results of the study valid? Evidence-Based Medicine Working Group.,13Richardson W.S. Detsky A.S. Users' guides to the medical literature. VII. How to use a clinical decision analysis. B. What are the results and will they help me in caring for my patients? Evidence Based Medicine Working Group. Since then, there has been a proliferation of medical decision analysis studies, in particular cost-effectiveness analyses. Markov or state-transition models, not explained in the original users’ guide, are now included in most decision analyses. This current guide presents a contemporary conceptualization and update on decision analysis. It is structured using the following section headings: “Are the Results Valid?” “What Are the Results?” and “How Do I Apply the Results to My Patient?” Before explicating the criteria for using a decision analysis, we will comment on when such an analysis might be helpful and present its anatomy.

Branch and node decision trees and state-transition models, which are explained in this guide, are “basic” decision model techniques. There are numerous other modeling techniques that are used to predict the events that occur after a clinical or policy decision is made.

When Might Medical Decision Analysis Be Helpful?In clinical practice it may not always be clear whether the benefits of an intervention outweigh the potential harms. Therefore, there is often uncertainty regarding the best course of action, even when randomized controlled trials have shown an intervention’s effectiveness. For example, prophylactic anticoagulation in a hospitalized patient reduces the risk of venous thromboembolism but increases the risk of a major bleeding event. The best course of action in this situation will depend on patients’ risk factors for venous thromboembolism, bleeding risk, and possible contraindications to anticoagulation.14Qaseem A. Chou R. Humphrey L.L. Starkey M. Shekelle P. Venous thromboembolism prophylaxis in hospitalized patients: a clinical practice guideline from the American College of Physicians.,15Grant P.J. Conlon A. Chopra V. Flanders S.A. Use of venous thromboembolism prophylaxis in hospitalized patients.

Medical decision analysis helps to determine the best course of action when there is a trade-off between different management strategies.

Although the risk-benefit trade-off commonly refers to clinical benefits and harms, it can also include the trade-off between greater effectiveness and higher cost. In this case, the appropriate decision model is a cost-effectiveness analysis that compares the relative costs and values (often expressed as per quality-adjusted life year [QALY]) of different courses of action). Cost-effectiveness analyses that address patient-important effectiveness outcomes can inform clinical decisions as well as health policy decisions.

The Anatomy of a Decision Analysis Based on a Decision TreeTable 1 summarizes the typical process of a decision analysis based on a decision tree, and Figure 1A shows an example of a decision tree that lies at the heart of decision analysis. Decision trees include a decision node (represented by a square) and a number of decision branches (Figure 1A). Every decision branch is followed by further branches representing the occurrence or non-occurrence of relevant events that may follow from the management decision (eg, developing drug-induced hepatitis from latent TB treatment, dying from drug-induced hepatitis, or dying from an unrelated cause). These events are represented by chance nodes (circles) with the associated probability for alternative outcomes at that node. Terminal nodes (triangles) show the final outcome of a decision path.

Table 1The Process of Conducting a Decision Analysis

Figure 1A, Example of decision tree structure. B, Markov sub-tree. A simplified decision tree depicts two courses of action: no testing/treatment and interferon-gamma release assay (IGRA) and treatment for positive IGRA that impact outcomes associated with the presence or absence of latent tuberculosis (TB).

Figure 1A, Example of decision tree structure. B, Markov sub-tree. A simplified decision tree depicts two courses of action: no testing/treatment and interferon-gamma release assay (IGRA) and treatment for positive IGRA that impact outcomes associated with the presence or absence of latent tuberculosis (TB).

Most outcomes occur over varying time frames, and patients may experience more than one outcome over time. An optimal decision analysis will be able to take the time factor and the probability of a patient experiencing more than one event over time into account. The methodology for this process involves what is called a Markov or state-transition model (Figure 1B), which allows modeling a decision problem that involves a risk that continuous over time and has outcomes that occur at different points in time (eg, ongoing risk over a lifetime of developing TB in a patient with latent TB), and when outcomes of interest may happen more than once (eg, development of a repeat episode of TB).7Siebert U. Alagoz O. Bayoumi A.M. et al.State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—3.A state-transition model consists of cycles of equal increments of time (eg, 1 month or 12 months). One assumption of the model is that in every cycle the patient resides in a finite number of "health states."7Siebert U. Alagoz O. Bayoumi A.M. et al.State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—3. Thus, in the model, a patient may be alive and TB free, develop TB, be cured from TB, die from TB, or die from an unrelated cause (Figure 1B). When moving through cycles, patients transition between these different outcomes (eg, after developing active TB a patient might recover from TB or die of TB). Another assumption is that future outcome only depends on current outcome, not past outcome.

Beware that figures accompanying analyses in published papers are highly simplified schematics. They will usually not allow the reader to understand the detailed model structure.

We will now move to our criteria for assessing the credibility and applying a decision analysis (Table 2).

Table 2Guide for Appraising and Applying the Results of a Decision Analysis

First Judgment: Was the Methodology of the Decision Analysis Credible?Table 3 summarizes all components of judgment 1 — determining the credibility of the methods of a decision analysis.Table 3Using the Guide: Judgment 1, Determining the Credibility of the Methods of a Decision Analysis (Cost-Effectiveness Analysis for Latent TB), applied to the study of Tasillo et al.Did the Decision Analysis Address a Relevant Clinical Question including all Important Outcomes?

Medical decision analyses should have a clear focus 1) addressing a clinical question for a specific group of patients; 2) using an actionable intervention and one or multiple clear alternative course(s) of action; and 3) using outcomes that are relevant for clinical decision making (ie, participants, intervention, comparator, and outcome, aka the PICO process). The decision tree on which the decision analysis is based must include all choice alternatives that are relevant to clinical decision making. Omitting management alternatives can provide very misleading results in a cost-effectiveness analysis (when an effective but less expensive alternative is not included). In our clinical example, relevant alternative strategies include a TST (and treating if there is evidence of infection), or an IGRA (and treating if there is evidence of infection), and no testing and treating.

A medical decision analysis should include all outcomes that are important to patients (eg, mortality, quality of life, and functional status) rather than surrogate outcomes (eg, result of a laboratory or imaging test). In their efforts to simplify the decision model, decision analysts should not omit rare but important outcomes such as death; any such omission reduces the trustworthiness of the result. Including harms of an intervention is as important as including benefits. For example, a cost-effectiveness analysis of latent TB treatment with isoniazid in close contacts of patients with infectious TB did not include any adverse events of preventive TB treatment and might therefore have overestimated the benefit of preventive TB treatment compared with no treatment.17Diel R. Nienhaus A. Schaberg T. Cost-effectiveness of isoniazid chemoprevention in close contacts.Is the Analytic Time Horizon of the Decision Analysis Appropriate?The time frame of the model, often referred to as the time horizon of the analysis, should be long enough to capture any costs and benefits that occur because of the intervention. For some decisions, choosing a lifetime horizon is appropriate, for example, when modeling a latent infection that can develop into active disease at any point during the patient’s life.18Dobler C.C. Martin A. Marks G.B. Benefit of treatment of latent tuberculosis infection in individual patients. A study of the risk of developing TB in young adults with latent TB that only has a time horizon of 20 years19Taylor W.C. Aronson M.D. Delbanco T.L. Should young adults with a positive tuberculin test take isoniazid?. will therefore not capture the cumulative lifetime risk. In other situations it may be appropriate to choose a time frame over a few years, for example, when the decision model focuses on interventions to prevent the recurrence of cancer.20Younis T. Rayson D. Skedgel C. The cost-utility of adjuvant chemotherapy using docetaxel and cyclophosphamide compared with doxorubicin and cyclophosphamide in breast cancer.Are Variables in the Model (Probabilities and Utilities) Based on Current Best Evidence?

A decision tree is populated with events and health states, each with their associated probabilities (eg, the likelihood of developing active TB) and utilities (eg, a year of life with active TB may be judged as having only half the value of a year of life in full health).

Every branch in a decision tree is assigned a probability (Figure 2A). Together the probabilities of all branches following a chance node add up to 1. Probabilities that reflect the risk that a certain event will happen are often modified by patient characteristics such as age or gender. It is therefore necessary to specify the population characteristics to which the results of the decision analysis apply.

Figure 2A, Simplified decision tree. B, Simplified “folded back” decision tree. This simplified decision tree (without Markov model, for illustrative purposes only) shows the process of “folding back the tree” to determine the best course of action in somebody who has latent tuberculosis (TB). In this case, the best case of action is to give latent TB treatment with the “no latent TB treatment” alternative marked as the rejected alternative. Total utility for latent TB treatment is 0.9992 versus 0.999 for no latent TB treatment. The outcomes at chance nodes (green) are mutually exclusive, and their combined probability must sum to 1.

Figure 2A, Simplified decision tree. B, Simplified “folded back” decision tree. This simplified decision tree (without Markov model, for illustrative purposes only) shows the process of “folding back the tree” to determine the best course of action in somebody who has latent tuberculosis (TB). In this case, the best case of action is to give latent TB treatment with the “no latent TB treatment” alternative marked as the rejected alternative. Total utility for latent TB treatment is 0.9992 versus 0.999 for no latent TB treatment. The outcomes at chance nodes (green) are mutually exclusive, and their combined probability must sum to 1.

The combination of probabilities derived from the literature and values (utilities) in a decision model determines the output of the model in terms of the best course of action that maximizes utility.21Science in a health service.,22Hunink M. Glasziou P. Siegel J. et al.Decision Making in Health and Medicine: Integrating Evidence and Values. On a utility scale, perfect health is typically assigned a value of 1 and being dead a value of 0. Many health states in a decision model are associated with decrements in quality of life (eg, a patient experiencing an adverse effect of a TB medication has a health state utility of less than 1). Health state utilities enable calculation of QALYs as an outcome. If, for example, a patient lives for 10 years in a health state that is associated with a utility of 0.7, this is equal to 7 QALYs.A well-conducted decision analysis should present the sources of all parameters (probabilities, utilities, and cost in a cost-effectiveness analysis; see below) used in the decision tree. Most published decision analyses report the sources of the parameter estimates and some include a systematic review and meta-analysis at least as source for the effectiveness of the intervention(s) under consideration.23Malhotra A. Wu X. Forman H.P. Matouk C.C. Gandhi D. Sanelli P. Management of tiny unruptured intracranial aneurysms: a comparative effectiveness analysis.

A critical look at the sources of the probabilities and utilities is perhaps the easiest and most important critical judgement for the clinician user of a decision analysis. Ideally, all important parameters that materially affect the results are based on systematic reviews. However, not all evidence derived from systematic reviews is high quality depending on the quality of the included studies and the review methodology. To the extent that evidence is low quality (that is, associated with high uncertainty), results of the decision analysis will be less robust.

For Cost-Effectiveness Analysis: What Was the Analysis Perspective?Decision analysis can include the expected costs of decision alternatives, referred to as cost-effectiveness analysis. Cost-effectiveness analysis is commonly performed to compare two or more decision alternatives where one offers an improved health outcome but at increased cost.24Ryder H.F. McDonough C. Tosteson A.N. Lurie J.D. Decision Analysis and Cost-effectiveness Analysis. Cost-effectiveness analyses focus on the incremental cost of an intervention for a particular difference in outcome — for instance, the additional cost of a management strategy to delay a single death from TB — referred to as an incremental cost-effectiveness ratio (ICER).25Sanders G.D. Neumann P.J. Basu A. et al.Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine., Thus, the ICER quantifies the trade-offs between improved health outcomes and resources spent.27Median-based incremental cost-effectiveness ratio (ICER).An intervention that is both less effective and more expensive than an alternative intervention is considered to be “strictly dominated” and is clearly inferior.28Interpreting the results of cost-effectiveness studies. If there are multiple interventions analyzed, interventions are listed according to cost, with the cheapest and least effective intervention forming the baseline comparison. The ICER for the next-most-expensive intervention is then calculated by dividing the difference in cost by the difference in effectiveness. In this way the analysts can calculate the ICER for all remaining options (Table 4). Dominated interventions are listed but not included in the ICER calculation.Table 4Cost-Effectiveness League Tablea

In the league table, the possible courses of action are listed in ascending order of their cost. For each course of action the additional (incremental) cost and effectiveness is compared to the previously listed course of action. The incremental cost-effectiveness ratio (ICER) for the tuberculin skin test (TST) strategy is higher than that of the next, more effective, alternative--the interferon-gamma release assay (IGRA) strategy. It is therefore dominated by the combination of 2 alternatives and should not be used to calculate appropriate ICERs. ICER, incremental cost effectiveness ratio; QALY, quality-adjusted life year.

The unit of effectiveness in a cost-effectiveness analysis is commonly the QALY, which reflects the quantity and quality of life. The use of a utility (perceived health-related quality of life for a health state) allows comparisons across different diseases and health sectors by using a common unit of measure (cost/QALY gained).29Screening strategies for active tuberculosis: focus on cost-effectiveness.Benefits, and particularly costs, are likely to differ depending on the perspective of the target audience be it the patient, a hospital, a health maintenance organization, an employer, a government payer, or society. The perspective will determine which costs and health outcomes are included in the model and how they are valued. A cost-effectiveness analysis of an infectious disease from a societal perspective, for example, would include modeling of disease transmission and associated health outcomes in affected people. The most common perspective of cost-effective analyses is the payer perspective, which is not recommended by experts.25Sanders G.D. Neumann P.J. Basu A. et al.Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine.,If the effectiveness results of a cost-effectiveness analysis are used for clinical decision making in individual patients, the model should not include any components that are not relevant to the patient perspective. To the extent that the perspective differs from the one that is relevant to you, you should take that into account. For instance, the individual clinician and patient may be uninterested in system costs. Coming back to the cost-effectiveness analysis in our clinical scenario3Tasillo A. Salomon J.A. Trikalinos T.A. Horsburgh Jr., C.R. Marks S.M. Linas B.P. Cost-effectiveness of testing and treatment for latent tuberculosis infection in residents born outside the United States with and without medical comorbidities in a simulation model.: you would ignore the overall cost estimates, as they are from a health sector rather than a patient perspective, and focus on the health outcomes for decision making in your patient including QALYs and the number of persons needed to test and treat to prevent one case of TB. You might want to add any information on costs that are relevant to the patient, if available. Importantly, as secondary cases from TB transmission are included in this analysis, you would tell your patient that the outcomes also include how other people close to them would be affected by the chosen course.Second Judgment: What Is the Confidence in the Preferred Course of Action?To determine the preferred course of action from a decision tree, one works backwards (ie, from right to left, that is from the terminal nodes to the initial decision node) by “folding back” the tree. At every chance node the utilities from all chance branches to the right are summed up by multiplying the probability assigned to every chance branch (providing the weight to derive the expected value) with its associated utility and adding the results from all branches. When arriving back at the decision node, the model will present the total utility associated with each option — the one with the highest utility represents the best choice (unless, in a cost-effectiveness analysis, it proves prohibitively expensive). Figure 2B gives an example of the process of folding back a decision tree.How Robust Are the Decision Analysis Results?To test the impact of uncertainties on the model outcome, the analyst should perform sensitivity analysis — that is, varying probability and utility values through their plausible range — for all key variables in the decision tree. For this purpose, each variable value (for probabilities and utilities) is assigned upper and lower bounds or a distribution reflecting the associated uncertainty. As we have noted, the analyst should base estimates on the best existing evidence from systematic review of the relevant literature. From such reviews will come the upper and lower bounds for the estimates (eg, 95% confidence intervals from a systematic review of primary studies). One-way sensitivity analysis — that is, considering the possible true values of one variable at a time — is commonly used to explore the influence of plausible variation on the decision problem. Analysts can present results of one-way sensitivity analyses in a tornado diagram, which provides information on the relative plausible impact of the variables addressed in sensitivity analyses (see Figure 3, which shows a fictitious example).18Dobler C.C. Martin A. Marks G.B. Benefit of treatment of latent tuberculosis infection in individual patients.