Background: In addiction medicine training, self-assessment is increasingly used to support self-regulation learning by identifying standards of excellence, competence gaps, and training needs. To ensure psychiatrists in Lithuania also develop specific addiction competencies, the Lithuanian Health Sciences University faculty in Kaunas developed an addiction psychiatry curriculum. Objectives: The aim of this research is to explore the efficacy of the AM-TNA scale to measure individual and group differences in proficiency in the core competencies of addiction medicine. A cross-sectional study and a convenience sample were used. Method: We studied the differences in performance in addiction medicine competencies between 4 successive year groups and analysed the variance to determine the statistical differences between the means of 4 year groups with biases, resulting from repeated measurement statistically corrected-for. Results: Of the psychiatrists in training, 41% or 59% completed the scale. The assessment of competencies suggested that all but 2 competencies differ significantly (p < 0.05) between the 4 groups. The post hoc analyses indicated that mean scores for 24 of the 30 core competencies differed significantly between the year groups (p < 0.05) and showed a gradual increase in scores of self-assessed competencies over the 4 year groups. We found adequate scale variance and a gradual increase in self-assessed competencies between the 4 year groups, suggesting a positive association between the results of incremental professional training and improved self-assessed substance use disorders (SUD) competency scores. Conclusions: This study illustrates the efficacy of the AM-TNA scale as an assessment instrument in a local training context. Future research should aim to have larger sample sizes, be longitudinal in design, assess individual progress, and focus on comparing and combining self-reported competencies with validated objective external assessment and feedback.

© 2023 The Author(s). Published by S. Karger AG, Basel

IntroductionLifelong Self-Regulated Learning and Self-Assessment

Substance use disorders (SUD) are a major contributor to the global burden of disease, ranking fifth in the top-10 causes of disability and death and a major cause of health care expenditure [1, 2], requiring not only improvement of effectiveness and organization of addiction medicine training across the globe [3], but also adequate postgraduate and continuous medical education for addiction medicine specialists.

In order to stay abreast of the latest developments and to strengthen professional expertise and competence, addiction specialists are required to be successful lifelong self-regulated learners: being able to identify their own strengths and weaknesses and having effective self-assessment and improvement strategies and tools [4]. Self-assessment can be used as an explicit way to strengthen formative assessment and is, in established addiction medicine curricula, increasingly used to support self-regulation learning [5]. However, stand-alone self-assessment has, especially when used as summative evaluation for medical students, accuracy limitations [6]. In order to encourage the use of valid self-assessment tools by addiction professionals, we chose to study self-assessment from the perspective of learning experience and as a formative assessment tool. We also stress the importance of combining self-assessment with other forms of assessment, such as feedback by peers, senior colleagues, preferably using multiple assessment tools [7-9].

Addiction, Training, and Assessment in Lithuania

In Lithuania, alcohol use contributes to 25% and smoking to 14% of total deaths [10]. The Lithuanian SUD-services are the responsibility of psychiatrists, who study SUD as a common comorbidity of other psychiatric conditions during their 4-year general psychiatry residential training. In order to ensure these trainees also develop specific SUD competencies, the Kaunas Lithuanian University of Health Sciences (LUHS) developed a specific addiction psychiatry curriculum, including a 3-month supervised addiction medicine placement in the final 2 years. Progress on SUD competencies is assessed using the Lithuanian version of the Addiction Medicine Training Needs Assessment scale (AM-TNA) [11]. This self-assessment instrument is useful for connecting addiction medicine training needs to the content of the training courses, through supporting self-regulated learning by both generalists and specialists. As a result, the aim of this study is to explore the AM-TNA’s self-assessment efficacy to identify both individual and group differences in strengths and weaknesses in core competencies among 4 successive year groups (YG) of psychiatrists in training enrolled in the aforementioned addiction psychiatry course.

Objectives

Exploring the efficacy of the AM-TNA scale in assessing individual differences in proficiency in addiction competencies.

Exploring the association (if any) between the AM-TNA self-assessment scores and the level of postgraduate training groups (years 1–4) of psychiatrists in training.

Materials and MethodsDesign

In this preliminary evaluation of self-assessed addiction medicine core competencies and training, a cross-sectional comparison was chosen to measure differences in perceived proficiency of 30 core addiction competencies, in 4 successive YG of psychiatrists in training at Kaunas LUHS.

Sample and Participants

To measures differences in training outcomes, we used a convenience sample of all psychiatrists in training at the Kaunas LUHS psychiatry clinic in May 2019. The Lithuanian training capacity for psychiatrists is around 100: being the responsibility of the medical schools of the Universities of Vilnius and Kaunas.

Ethical Considerations

The study complies with the Declaration of Helsinki: written informed consent from the participating psychiatrists in training was established before the completion of the anonymous questionnaire.

The Instrument

The AM-TNA was developed to measure progress and gaps on addiction medicine competencies in Indonesia. The instrument consists of a 30-item questionnaire containing self-reported proficiency on internationally recognized addiction medicine specific core competencies, using a 5-point Likert-scale ranging from not-at-all proficient-fully proficient. The instrument assesses 21 clinical and 9 non-clinical competencies in 3 professional domains: assessment and diagnosis, starting treatment, and maintaining treatment [12]. The Indonesian AM-TNA scale was translated into Dutch, Lithuanian, and English. The scale has besides a proven reliability and internal validity a clear-cut 2-dimensional factor structure [11].

Analysis

To demonstrate psychiatrists in training can self-assess their proficiency in addiction competencies using the AM-TNA scale we expect, besides a high response rate, very few missing values, an adequate variance in results, and a significant difference in means between the 4 YG. To assess the relationship between these groups and self-assessed competencies, an a priori analysis of variance (ANOVA) is planned. To test the mean differences of 30 competencies between the 4 YG, a between-subject test will be used.

To explore the association between the AM-TNA self-assessment scores and the level of postgraduate training (YG 1–4), we expect that mean scores will gradually increase over the years. In order to correct for the risk of false positives resulting from repeated measurement and to limit the instance of false positive results (Type-I errors), we will execute a Bonferroni-adjusted post hoc analysis: a series of t tests on each pair of classes using a corrected p value.

Results

All 52 participants received the AM-TNA scale, of which 41 questionnaires (79%) were completed by 4 YG: YG 1:14, YG 2:8, YG 3:10, and YG 4:9. Mean age, respectively: 26, 26, 32, and 29, M = 28.29 (SD: 4.627). Gender: 29 (70.7%) female and 12 (29.3%) male. There were no missing values. The sum scores of 28 competencies differed significantly between the 4 YG (p < 0.05) with the exception of the competencies 6 and 10 (Table 1, Column 8).

Table 1.

Mean addiction medicine competencies of 4 YG of psychiatrists in training

All competencies except # 6 had F values well above 1, indicating that in-between group variance was much more important than with-in group variance. The Bonferroni-adjusted post hoc analysis of 6 pairs and 4 YG is presented in Table 1. Bonferroni correction was α/n = 0.05/6 = p< 0.008 indicates that the mean scores of 19 of the 30 competencies differ significantly between one or more training years (Table 1, column 2). 11 clinical competencies (1, 2, 3, 6, 7, 10, 12, 15, 16, 18, 20) do not show significant statistical differences between the training groups, whilst all competencies but 6 and 10 showed a tendency in the predicted direction. Comparing the total of the clinical and non-clinical competencies resulted in significant differences (p < 0.001) between the 4 YG (resp. 1 < 4, 2 < 4 and 1 < 4, 2 < 4, 3 < 4). The gradual increase in marginal means of the combined clinical and non-clinical competencies over the 4 classes in training is visualized in Figure 1.

Fig. 1.

Bonferroni plots: estimated marginal means (y-axis) of combined clinical addiction medicine competencies (top) and combined non-clinical Competences (bottom) against 4 year groups of residential psychiatry specialists in training (x-axis).

Conclusion

The cross-sectional study design fitted the preliminary evaluation need and the sample represented around 50% the psychiatrists in training in Lithuania. We found a significant difference in mean scores between the 4 YG of psychiatrists in training. This indicates that the AM-TNA scale can measure systematic group differences in perceived proficiency in the SUD core competencies.

Furthermore, an excellent response rate (79%) and no missing values supported the efficacy of the AM-TNA scale (Objective 1). The Bonferroni plots illustrated a gradual increase in marginal means of the combined clinical and non-clinical competencies in the 4 training groups. This suggests an association between the AM-TNA scale self-assessed learning (vertical axis) and cumulative training years (horizontal axis) (Objective 2).

This explorative, cross-sectional study has limitations. The results are based on a small convenience sample, and recruiting respondents from a single training location (LUHS) may have introduced some selection bias into our study. The study design choice also means we cannot determine causality.

This study clearly confirms the relationship, between AM-TNA scale self-assessment scores and the number of years spent in psychiatry training. We may have missed some individual variability and some individual self-assessment of proficiency in SUD competencies may have plateaued or even declined. Finally, the AM-TNA is a self-reported scale, reflecting the respondents’ perceived competency and not their objectively measured skills. This limitation would be concerning if self-ratings in the YG 1 were unusually high as it may indicate socially desirable responses. However, this was not detected in our study.

Competency-based assessment is gold standard in medical training. It is also worthwhile to note that self-assessment of medical competencies, despite its accuracy limitations, is an important component of lifelong professional learning, like continuous medical education and an established tool to promote both academic achievement and self-regulated medical learning [13].

Validated self-assessment tools are particularly useful in encouraging self-assessment of addiction caregivers' perceived SUD competencies, and as such stimulate the acquisition of addiction skills by specialized and non-specialized professionals. Despite the study’s limitations, the AM-TNA self-assessment scale measured individual proficiency and produced significant group differences in perceived SUD competencies in a small local training context.

Accuracy and external validity issues of self-assessment scales like the AM-TNA can be overcome when used as formative evaluation tools in combination and in support of external and objective evaluations and observations, critical appraisal, reflection reports. This study illustrates that the self-assessment AM-TNA scale can measure individual proficiency and significant incremental differences between mean scores of perceived proficiencies in addiction medicine core competencies in a 4 YG psychiatry training setting. We suggest it could also function well in larger-scale training settings. The AM-TNA scale might also support individual self-regulated learning and could be helpful to improve individual competencies. Future research should aim to investigate differences in larger sample sizes over time, assess individual progress, and focus on comparing and combining self-reported competencies with validated objective external assessments of learning, attitudes, and clinical practice.

Acknowledgments

We would like to thank the Kaunas Lithuanian University of Health Sciences and the participating psychiatry in training for their collaboration. The authors would like to thank Hugh Claridge M.A., Population Health Research Institute, St George’s, University of London and Juul Pinxten, World Bank Washington DC for their language review and comments on an earlier version of this paper. They gave their written consent on being acknowledged.

Statement of Ethics

The research was conducted ethically in accordance with the World Medical Association Declaration of Helsinki. Study approval: the study protocol was reviewed and approved by the Kaunas Lithuanian University of Health Sciences Ethics Committee: approval number Nr-BC-LSMU-121. Participation consent: written consent from participating psychiatry specialists in training to participate in the study was obtained before completion of the anonymous questionnaire.

Conflict of Interest Statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the publication.

Funding Sources

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Author Contributions

Lucas Pinxten: corresponding author, research conception and design, data collection, contributed to data analysis and interpretation of results, writing, revising, and finalizing the manuscript. Darius Jokūbonis: contributed to data collection, interpretation of results, revising, and approving the manuscript. Virginija Adomaitiene and Darius Leskauskas: contributed to the interpretation of the results, revising, and approving the manuscript. Giel Hutschemaekers: contributed to the research design, the interpretation of the results, writing, revising, and approving the manuscript. Cornelis de Jong: contributed to research conception and design, data collection, analysis, and interpretation, writing, revising, and approving the manuscript.

Data Availability Statement

After publication of this article, all participating psychiatrist in training will receive a copy of this article and the university research protocol/ethics clearance (Nr-BC-LSMU-121) from the authors related to the Kaunas Lithuanian University of Health Sciences. On request, the corresponding author will share the anonymized raw data set and the SPSS syntax and output after approval by the Kaunas Lithuanian University of Health Sciences.

This article is licensed under the Creative Commons Attribution 4.0 International License (CC BY). Usage, derivative works and distribution are permitted provided that proper credit is given to the author and the original publisher.Drug Dosage: The authors and the publisher have exerted every effort to ensure that drug selection and dosage set forth in this text are in accord with current recommendations and practice at the time of publication. However, in view of ongoing research, changes in government regulations, and the constant flow of information relating to drug therapy and drug reactions, the reader is urged to check the package insert for each drug for any changes in indications and dosage and for added warnings and precautions. This is particularly important when the recommended agent is a new and/or infrequently employed drug. Disclaimer: The statements, opinions and data contained in this publication are solely those of the individual authors and contributors and not of the publishers and the editor(s). The appearance of advertisements or/and product references in the publication is not a warranty, endorsement, or approval of the products or services advertised or of their effectiveness, quality or safety. The publisher and the editor(s) disclaim responsibility for any injury to persons or property resulting from any ideas, methods, instructions or products referred to in the content or advertisements.