The purpose of this study was to 1) help novice students scaffold problem-solving and engage safely in the deliberate practice of diagnostic reasoning and medical decision-making in real time; 2) assess how accurately students gather and apply data in medical reasoning and treatment during high-fidelity patient simulations (HFPSs); 3) identify students’ scientific misconceptions related to the case; 4) promote student metacognitive processing, self-assessment, and self-efficacy; and 5) facilitate the explicit calibration of student confidence in deliberate reasoning with patient outcomes. In a mixed-method design, a metacognitive calibration self-assessing (MCC) survey tool was applied to HFPS (n = 80, 20 teams of 6 medical students) and semistructured interviews were conducted with faculty (n = 5). When scored by faculty with a rubric, the mean student accuracy ranged from 23% to 74%, whereas their self-assessment of confidence ranged from 71% to 86%. This result revealed overconfidence bias in novice students regarding the correctness of their wrong responses. The most common misconception identified was inverting cause and effect: metabolic acidosis was pointed to as the cause of the patient’s problems rather than a consequence of untreated diabetes mellitus. The most common treatment error was overtreatment, with unnecessary added medication. Interviews with faculty suggested that the MCC tool improved the team process by slowing students down, requiring them to think through their answers, and that overall the tool improved their critical thinking. This study demonstrated the feasibility of using a metacognitive confidence calibration tool to assist novice students in learning safely to make deliberate diagnostic reasoning and decisions on patient care in real time during complex simulations while observing objectively their levels of psychological confidence against patient outcomes.

NEW & NOTEWORTHY This study demonstrates the feasibility of a metacognitive confidence calibration tool (MCC) to assess and promote novices in the learning of diagnostic reasoning and treatment decisions on patient care in real time during high-fidelity patient simulations while comparing confidence and accuracy data and identifying students’ scientific misconceptions. Results revealed the presence of overconfidence bias, overtreatment, and the misconception of metabolic acidosis as the cause of the patient’s problems rather than a consequence of untreated diabetes mellitus.