Promoting consistent, good-quality education through faculty development activities is a priority in family medicine.1 With the implementation of competency-based medical education (CBME) systems, medical teachers require faculty development support to prepare them with effective strategies for training and assessing learners in this new climate.2 Over the past decade, the field of faculty development has grown substantially, with more than 100 published articles describing the outcomes of short- and long-term educational interventions.3 Despite the growing efforts to build programs for the professional development of clinical teachers, a recent systematic assessment yielded a lack of strong evidence to support the decisions made for choosing faculty development activities.4 Furthermore, evidence on the organizational changes that evolve from faculty development initiatives has been largely underexplored.3

In performing their commitment to supporting its members in their educational roles, the College of Family Physicians of Canada (CFPC) created the Fundamental Teaching Activities in Family Medicine Framework (FTA) in 2015. Designed to assist family medicine educators, programs, and faculty developers understand the core activities of teachers, the FTA emphasizes the importance of precepting and coaching within and outside of clinical settings.5 Intended to guide teacher development and frame faculty development needs for assessment or curricular programming, the FTA provides a roadmap for a novice to experienced teachers in family medicine. However, before this evaluation, there was minimal understanding about teachers' awareness of the FTA. Thus, the purpose of this project was to evaluate its awareness, application, and utilization in Canadian departments of family medicine, their educational programs, and by family medicine teachers.

For the purposes of this innovative project, we conceptualized “awareness” as a self-assessment of generalized knowledge about this framework.6 Application-wise, we refer to an operational definition endorsed by the Canadian Institutes of Health Research, “an iterative process by which knowledge about [the FTA] is put into practice”.7 For those aware of and applying it, our evaluation sought to further understand their experiences with using the FTA within their academic roles (ie, utilization).

EVALUATION METHODOLOGY

We used a practical participatory evaluation (P-PE) approach8 to conduct an evaluation of the FTA. This P-PE approach involved a partnership between the research team and key stakeholders from the CFPC's Faculty Development Education Committee. By establishing a partnership between the research team and three Faculty Development Education Committee members, we leveraged the professional practices of evaluators and first-hand knowledge provided by nonevaluators to conduct an evaluation in a formative, improvement-based context. This expert panel advised on project design, development of data collection protocols and instruments, project implementation, and validation of findings. We considered strategies from explanatory, sequential mixed-methods research as a compatible approach to conducting a utilization-oriented evaluation. From an epistemological standpoint, mixed-methods inquiry enables a broad yet comprehensive understanding of how the FTA supports faculty development practices. The quantitative examination allowed us to first gather information about the current landscape with how the framework is being used. Next, key informant interviewing encouraged us to better understand how academic leaders viewed the FTA. Thus, an evaluation guided by mixed-methods and P-PE approaches helped justify the social accountability needs for faculty development and capacity building across family medicine education communities. Exemption from research ethics for this two-phase mixed-methods evaluation was granted by the University of Ottawa Research Ethics Board.

Phase 1 Participants

For phase 1 participation, a CFPC administrator identified and contacted all 15 faculty development directors (FDDs), 18 postgraduate program directors (PDs), and 174 site directors (SDs) in Canadian university departments of family medicine.

Data Collection and Analysis

We developed a survey to examine the degree of awareness, application, and utilization toward the FTA. Before survey distribution through Qualtrics, we piloted the surveys to determine the appropriateness, comprehensibility, and feasibility.9 The final surveys consisted of 38 questionnaire items; 33 were closed-ended, and 5 were open-ended (see Material I, Supplemental Digital Content 1, https://links.lww.com/JCEHP/A155). We administered the surveys online in both English and French (translated by a professional translator). To maximize participation, we adhered to a modified version of Dillman10 Tailored Design Method when distributing the surveys. After the initial distribution of the survey (September 7, 2018, to FDDs and PDs; October 3, 2018, to SDs), CFPC administrators subsequently sent two email reminders to potential respondents for survey completion (September 26 and October 10, 2018, to FDDs and PDs; and October 17 and October 31, 2018, to SDs). Data collection for the survey was officially closed on November 23, 2018. We also reminded potential respondents about the ongoing surveys in-person at research conferences hosted during this recruitment period (ie, International Conference on Residency Education, Family Medicine Forum). K.E. and C.G. analyzed all closed-ended survey responses in IBM SPSS v.25 using descriptive statistics (ie, frequencies and percentages for dichotomous rating items) and analyzed text responses to open-ended items using qualitative content analysis.11

Phase 2 Participants

We used convenience sampling to identify individuals to participate in an interview.12 R.L.K. and M.V. emailed information letters to eligible participants who on their surveys expressed interest to participate in the second phase.

Data Collection and Analysis

We developed interview guides in English and French to elucidate participant perspectives about the FTA (see Material II, Supplemental Digital Content 2, https://links.lww.com/JCEHP/A156). All expert panel members piloted the interview guides, and then, they were translated to French by a professional translator. The final interview guides included 15 open-ended questions about the facilitators and barriers that may influence use of the framework, as well as questions to explore participants' needs, factors for buy-in, and ways to apply the FTA.

Between January 8 to May 30, 2019, R.L.K., D.A., or M.V. conducted 30- to 60-minute semistructured interviews with participants in their preferred language through telephone. All interviews were audio-recorded and transcribed verbatim. Using NVivo v.12, the interview transcripts were independently analyzed by two researchers (R.L.K. and D.A.) after thematic analysis.13 R.L.K. and D.A. first generated initial codes to organize the data at a granular level and then used them to search for major themes. In addition, selected transcripts were reviewed by two expert panel members for codebook verification, and any differences were resolved before generating conclusions. We used a mix of deductive and inductive coding to ensure that key themes were not missed or force-fitted into a pre-existing coding system.

Integration of Phase I and Phase II

The mixing of data collection and analytical strategies occurred at two stages. The first point of integration occurred before the start of phase II. To preserve the sequential nature of this evaluation, we integrated the survey findings into the design of the interview guides. The second point of integration occurred at the final stage of data analysis. Using a “merging integration” technique, we linked key findings and presented meta-interpretations for both phases on a side-by-side joint display.14

RESULTS Demographics

A total of 58 participants completed the survey (overall response rate of 28%): 12 FDDs, 12 PDs, and 34 SDs in Canadian university departments of family medicine, resulting in 80%, 66.7%, and 19.5% individual response rates, respectively. Demographic details about the surveyed participants are presented in Table 1. Most FDDs and PDs had 1 to 5 years of experience within their academic roles and are distributed across different geographic regions of Canada. Most SDs held a leadership role for program development in Western Canada or Ontario regions, but none were previously involved with constructing the FTA.

TABLE 1. - Participant Demographics Characteristic N n (%) Academic role 39  FDD* 12 (30.8)  PD 9 (23.1)  SD 18 (46.2) Years in role  FDD 10   Less than 1 y 1 (10)   1–5 y 8 (80)   6–10 y 1 (10)  PD 7   Less than 1 y 1 (14.3)   1–5 y 4 (57.1)   6–10 y 2 (28.6)  SD 14   Prefer not to specify 14 (100.0) Region of Canada  FDD 11   Western Canada 3 (27.3)   Eastern Canada 2 (18.2)   Ontario 4 (36.4)   Quebec 2 (18.2)  PD 7   Western Canada 2 (28.6)   Eastern Canada 1 (14.3)   Ontario 2 (28.6)   Quebec 2 (28.6)  SD 14   Western Canada 4 (28.6)   Eastern Canada 0 (0.0)   Ontario 10 (71.4)   Quebec 0 (0.0) Involved in developing the FTA  FDD 11 4 (36.4)  PD 7 2 (28.6)  SD 14 0 (0.0) Leadership role in educational program development  PD 7 6 (85.7)  SD 14 12 (85.7)

*Two FDD participants served delegate roles: professor and director.

Of the 12 participants who were interviewed, 6 were FDDs (1 Francophone), 3 PDs, and 3 SDs across several Canadian provinces: British Columbia, Alberta, Manitoba, Ontario, and Quebec.

Survey

Table 2 presents information about the degree of awareness of the FTA among academic leaders. Our findings indicate that most FDDs, PDs, and SDs were familiar with the framework's content through various methods of dissemination (eg, CFPC website/events and colleagues). Hard copy and electronic versions of the framework were accessed similarly across stakeholder groups. Reported by academic leaders, the perceived purposes for using the FTA included education programming in faculty development or career planning for clinical teachers. Several respondents also believed that the FTA serves as a self-reflective tool for improving their own teaching practices.

TABLE 2. - Awareness of the FTA Survey Items FDD PD SD N n (%) N n (%) N n (%) Awareness of the FTA 12 12 (100.0) 12 9 (75.0) 34 18 (52.9) Method of awareness* 12 9 18  College meeting 8 (66.7) 5 (55.6) 7 (38.9)  Website 4 (33.3) 3 (33.3) 3 (16.7)  Newsletter 1 (8.3) 1 (11.1) 0 (0.0)  Local meeting 1 (8.3) 3 (33.3) 6 (33.3)  Provincial meeting 1 (8.3) 2 (22.2) 2 (11.1)  National meeting 5 (41.7) 4 (44.4) 6 (33.3)  Other (eg, colleagues) 6 (50.0) 2 (22.2) 1 (0.0) Format of framework accessed* 12 9 18  Hard copy 12 (100) 8 (88.9) 10 (55.6)  Electronic 12 (100) 8 (88.9) 11 (61.1)  Have not accessed 0 (0.0) 0 (0.0) 3 (16.7) Perceived purpose of the FTA* 12 9 18  To provide teachers with an understanding of the activities that are expected of them, depending on their task(s) 8 (66.7) 6 (66.7) 9 (50.0)  To provide a road map for teachers to guide their self-reflection and continuing professional development 9 (75.0) 9 (100.0) 13 (72.2)  To assist programs, departments, and faculty members in developing educational programming for faculty development 10 (83.3) 8 (88.9) 11 (61.1)  To provide an organizational framework for faculty development materials, tools, and strategies, both locally and nationally 8 (66.7) 6 (66.7) 9 (50.0)  Other† 2 (16.7) 0 (0.0) 1 (5.6) The FTA as a self-reflective tool 12 7 (58.3) 9 4 (44.4) 18 5 (27.8)

*Respondents selected all options that apply.

†Additional responses received: all of the above; transform principles into practical reflection exercises; not practical; another physician has oversight over faculty development activities.

In application, Table 3 describes the current and future uses of the FTA in family medicine education programs. Applications of the FTA in educational programming were reported highest among FDDs. Specifically, participants identified that the FTA helps them establish program standards for faculty development, develop educational resources for clinical teachers, and promote faculty development activities. Some PDs agreed that the framework can be used to support teachers of family medicine residency programs (eg, inform assessment approaches). Contrarily, most SDs did not identify clear applications of the FTA within or beyond clinical contexts.

TABLE 3. - Application of the FTA Framework Survey Items FDD PD SD N n (%) N n (%) N n (%) Use of the FTA for educational programming 12 9 (75.0) 9 4 (44.4) 16 3 (18.8) Use of the FTA for FDDs* 12  To develop strategies for teachers working with learners experiencing progression challenges 2 (16.7) — —  To evaluate educational programs 1 (8.3) — —  To apply and develop your individual program standards 4 (33.3) — —  To engage stakeholders such as your institution and the CFPC 3 (25.0) — —  To use and develop resources 4 (33.3) — —  Do not currently use the FTA 1 (8.3) — —  Other 6 (50.0) — — Use of the FTA for clinical preceptors* 9 16  To explicitly embody the roles, attitudes, and competencies of a family physician in clinical work — 2 (22.2) 4 (25.0)  To promote and stimulate clinical reasoning and problem solving — 1 (11.1) 1 (6.3)  To give timely, learner-centered, and constructive feedback — 1 (11.1) 4 (25.0)  To use assessment tools to document observed learner performance according to training level — 0 (0) 1 (6.3)  To use reflective processes to refine clinical supervision — 2 (22.2) 3 (18.8)  To help learners design and update their individual learning plans — 2 (22.2) 2 (12.5)  To guide comprehensive periodic progress reviews informed by the learners' self-analyses — 2 (22.2) 0 (0)  To assist learners in their professional development — 1 (11.1) 2 (12.5)  To adjust teaching interventions to support learners facing progression challenges — 2 (22.2) 3 (18.8)  None of the above — 2 (22.2) 7 (43.8)  Other (ie, not sure, orientation training) — 2 (22.2) 2 (12.5) Plans to use the FTA* 12  To develop strategies for teachers working with learners experiencing progression challenges 5 (41.7) — —  To evaluate educational programs 3 (25.0) — —  To apply and develop your individual program standards 5 (41.7) — —  To engage stakeholders such as your institution and the CFPC 5 (41.7) — —  To use and develop resources 5 (41.7) — —  Other 2 (16.7) — — Use of the FTA by teachers outside of clinical setting* 9 15  To prepare teaching sessions — 0 (0) 3 (20)  To facilitate teaching sessions — 2 (22.2) 1 (6.7)  To reflect on teaching sessions — 3 (33.3) 0 (0)  None of the above — 5 (55.6) 8 (53.3)  Other (eg, respondent not sure) — 1 (11.1) 3 (20) A person is responsible for creating faculty development programming 12 10 (83.3) 9 9 (100.0) 16 14 (87.5)  Informed by the FTA framework? 10 7 (70) 8 7 (87.5) 13 9 (69.2) Institution supports faculty development activities 12 11 (91.7) 8 8 (100) 16 15 (93.8) FTA strategies to develop faculty development education* 12 9 16  Needs assessment 7 (58.3) 6 (66.7) 5 (31.3)  Implementation 6 (50) 3 (33.3) 4 (25)  Evaluation of educational activities 4 (33.3) 3 (33.3) 7 (43.8)  Program adjustment 4 (33.3) 3 (33.3) 4 (25)  Adjust learning plans to support learners with progression challenges 2 (16.7) 2 (22.2) 5 (31.3)  Adjust educational programming to support learners and teachers 6 (50) 4 (44.4) 3 (18.8)  Develop resources to support learners and teachers 9 (75) 4 (44.4) 8 (50)  None of the above 2 (16.7) 2 (22.2) 3 (18.8)  Other 1 (8.3) 0 (0) 0 (0)

*Respondents selected all options that apply.

Our evaluation measured the extent to which the FTA is operationalized in family medicine education programs. Table 4 outlines the approaches that FDDs, PDs, and SDs performed to use the FTA. When asked to rate the usefulness of the FTA, academic leaders scored clarity, utility, and feasibility of the FTA document as moderate to high. However, most participants reported that the level of utilization at their institution remained low to none. Although some participants accessed the online repository of educational materials about the FTA, several did not or perceived them to be underdeveloped.

TABLE 4. - Utilization of FTA Framework Survey Items FDD PD SD N n (%) N n (%) N n (%) FTA strategies to implement faculty development/educational programming* 11 9 14  Collaboration across sites 4 (36.4) 3 (33.3) 8 (57.1)  Collaboration across program specialties