Silbaugh and El Fattal (2022) identify the first two steps in their call to action as the need to establish strategic plans and QD-KPIs necessary to accomplish an organization’s strategic goals. The ASAI agencies’ QD-KPIs, listed in Table 3, include measures of client progress towards defined goals and the satisfaction of consumers involved with the agency. These QD-KPIs are used to assess attainment of delivering high-quality services that are based on both professional and consumer standards. Of equal importance is the need to use standardized, well-defined, and consistent data collection methods across agencies to allow for a comparison of outcomes relative to system components (BHCOE, 2021). Regular, annual collection and review of standard, but individual, client-based data across multiple agencies enables analysis of the extent to which high-quality outcomes are attained over agencies and across time, regardless of shifts in cultural, social, and other factors (including the COVID-19 pandemic). In addition, these data indicate needed areas of improvement and future strategic development. The collection and organization of these data align with Silbaugh and El Fattal’s Step 3, recommending the establishment of a dashboard that allows leadership to monitor progress over time, not only in an individual agency but across multiple agencies, with respect to attainment of QD-KPIs.

Skilled professionals are one of the most important elements of any educational program. Instructional staff must demonstrate proficiency in key areas that are defined and measured, including clinical, professional, data collection, and data analysis skills (Ellis & Glenn, 1995; McClannahan & Krantz, 1981). The ASAI agencies adhere to a consistent model of training and evaluation to promote successful staff development and assessment of performance relative to standards set by the agency, a system that aligns with Step 4 of Silbaugh and El Fattal’s (2022) call to action. Each ASAI agency uses a similar protocol to train and evaluate staff members. A minimum of 25 hr of didactic training in behavior analysis is provided outside the educational context through workshops delivered at hire and in an ongoing manner throughout each academic year. More important, hands-on training within educational programs is a daily occurrence in each agency. Behavioral skills training (Parsons et al., 2012; Sherman et al., 2021) is expected to occur regularly with a trainer modeling skills, providing practice opportunities, and delivering feedback and ongoing practice until criterion is achieved. This high level of daily, in-classroom training requires trainers to be highly skillful themselves, regularly available, and accountable to the instructional staff and clients with respect to producing skill acquisition (McClannahan & Krantz, 1993; Reid et al., 2017).

All ASAI agencies employ instructional staff who are responsible for providing hands-on teaching, collecting and analyzing data, preparing curriculum, and advancing programming for the clients. All instructional staff members have, at minimum, a bachelor’s level degree in education, psychology, or a related field. Each instructional staff member is assigned to a particular client to ensure regular data collection and program oversight for a 1-year period, but works with all of the clients in a classroom throughout a day and is responsible for ensuring client progress for all assigned teaching tasks.

In each ASAI agency, hands-on-training is provided by highly skilled trainers who meet defined performance criteria. One trainer is assigned to each classroom to provide training to instructional staff in that classroom and oversight for client progress for those clients in that classroom. Trainers are staff who hold, at minimum, a bachelor’s degree, have passed at least two staff evaluations, and have demonstrated advanced competence in clinical, professional, and behavior analytic repertoires measured on the staff evaluation protocol. A trainer’s primary responsibilities include providing hands-on training to instructional staff, assessing instructional staff performance through direct observation, reviewing client progress, and meeting with instructional staff members to discuss necessary programming changes. Trainers spend at least 80% of their 40-hr (at minimum) work week completing these responsibilities.

The availability of sufficient training staff is necessary to develop skillful instructional staff members who can pass an evaluation and produce client progress towards goals and objectives, two critical QD-KPIs. As such, each ASAI agency ensures a high in-class trainer to instructional staff ratio and provides each trainer and instructional staff member with the standardized training and assessment tool at the commencement of their employment. This tool is referenced repeatedly throughout training sessions. As can be seen in Table 4, a trainer was responsible for no more than seven staff members in any given year in each agency, ensuring a high trainer to instructional staff (and client) ratio.

The professional staff training and evaluation protocol measures both client performance and instructional staff performance. All ASAI agencies use a professional evaluation process that includes a full 6-hr day of observation of an instructional staff member using the standardized evaluation tool. Direct observation and data collection throughout the 6-hr evaluation assess (1) client engagement; (2) the number of opportunities to respond that are provided by the instructional staff member; (3) the number of behavior-specific praise statements that are provided by the instructional staff member; and (4) the number of incidental teaching episodes and script and script-fading instructional programs used to develop language. Several other areas are evaluated using a 7-point Likert scale ranging from completely satisfied (7) to completely dissatisfied (1), as seen in Table 5. The areas rated include (1) professionalism; (2) teaching new skills using behavioral strategies and evidence-based practice; (3) increasing social competence; (4) decreasing challenging behavior and teaching functional replacement behavior; (5) developing positive relationships; (6) arranging the environment to promote learning; (7) programming for generalization; (8) discussing intervention technology; and (9) ensuring individualized and effective programming through a complete review of every individualized program and associated data summary in the client’s notebook for which the instructional staff member is responsible.

Staff evaluation occurs in an ongoing manner by the trainer using components of the staff evaluation protocol during training sessions. However, a complete evaluation is conducted twice a year to assess staff clinical and professional performance, client performance, and skills in graphing/analyzing data. A practice opportunity (pre-evaluation) is conducted at least 3 months after an instructional staff member is hired in every ASAI agency. This assessment is conducted by the trainer assigned to the classroom. It provides an opportunity for the staff member to experience the process, ask questions, and gain feedback from his/her direct trainer prior to the formal evaluation. It also allows a trainer to obtain data on each staff member’s skill set and to identify and prioritize upcoming training opportunities.

The formal professional staff evaluation is conducted at least 6 months after the instructional staff member begins employment. This assessment is conducted by a trainer who is not directly assigned as the classroom trainer. This promotes a higher level of accountability in the process, such that the outcome from the evaluation can be used to measure both the instructional staff member’s and classroom trainer’s effectiveness. This is part of the quality management process similar to that suggested by Silbaugh and El Fattal in Step 5. In addition, on a number of evaluations two evaluators are assigned to allow for the collection of interobserver agreement data on all measures of the protocol. The data collected across agencies consistently meet or exceed the 80% criterion agreed upon by all agencies. All evaluators use the standardized evaluation protocol, taking observational data and completing all of the protocol sections. For each section, a score is provided based upon direct observational measures and/or the percentage of items within an area rated above, at, or below criterion. The evaluator then summarizes the data as the percentage of evaluation areas above, at, or below criterion to determine if the staff member has passed the evaluation (i.e., a minimum of 80% of the evaluation areas being at criterion). It is important to note that, in each of the agencies, appointment for the upcoming year is based upon passing this evaluation. Only those instructional staff members who pass the evaluation will be invited back as team members the following year—a powerful contingency between performance and employment.

The percentage of staff who pass an evaluation is summarized both within classrooms and across the agency to assess adherence to consistent professional standards. As such, all ASAI agencies annually report the percentage of staff members who were evaluated and, as one QD-KPI, the percentage of those staff members who demonstrated criterion skills in delivering services to clients by passing their evaluation. As seen in Table 6, the mean annual percentage of staff evaluated in each ASAI agency was close to 100% for the 10-year time frame, except for one agency below the 80% criterion. It should be noted that the lowest percentages for all agencies on this QD-KPI occurred during the 2020 year due to the pandemic and virtual instruction replacing in-person instruction for a period (with the full members evaluating 62%, 100%, 65%, 17%, and 0% of their staff, respectively). Despite this disruption, the high mean annual percentage of staff passing their evaluation support the organizations’ commitment to operating with low trainer-to-staff ratios; effectiveness of the training protocol; and staff’s demonstration of requisite skills for delivering effective intervention services and meeting professional standards, including those mentioned by Silbaugh and El Fattal (2022) such as the use of evidenced-based and best practices, implementation of the seven dimensions of behavior analysis, and adherence to the BACB code of ethics.

Although it is not necessarily easy to do, an investment of time and energy in developing skillful and positive staff members using a standardized training and evaluation protocol results in those staff members displaying necessary clinical skills. Analysis of the data collected by that staff member on client progress also demonstrates that meaningful outcomes for clients are achieved as a result. Employing highly skilled trainers who are able to display the skills they are responsible for training, along with successful, positive, and frequent training interactions, is essential to any program. These practices produce a positive workplace culture where the focus is on client progress and high-quality service delivery.

Every client in each ASAI agency has a defined set of comprehensive annual goals determined to be important by the guardians (as all clients are under the age of 21) and agency team, including but not limited to academic, language and communication, social interaction, recreational, independence, self-help, prevocational, and technology goals. These goals are translated into individualized programs that consist of objective response definitions; measurement procedures; and teaching, generalization, and maintenance conditions. Data on client performance are collected, graphed, and analyzed on an ongoing basis to determine progress towards the annual goals. These data are reviewed regularly by instructional staff (daily review), training staff (weekly review), guardians/families (biweekly to monthly review), and program administrators (quarterly review) to assess acquisition of target goals, define needed modifications to teaching procedures to enhance skill acquisition, and advance programming in an efficient manner. Likewise, as part of each instructional staff member’s evaluation, the entire set of client programs that he/she is responsible for is reviewed using a standardized protocol. This is done to determine if a program is (1) individualized; (2) producing desired behavior change; (3) appropriate (based on current research and ethical practices); (4) has quarterly interobserver agreement data; and (5) contains guardian consent, at a minimum. Table 7 lists both the evaluation dimension and the response definition for each. These data are reviewed on a client-by-client basis to determine the quality of programming delivered to that client by the agency and the staff members involved in the educational process. Additional measures include responsivity to the data, maintenance of behavior, and demonstration of generalized behavior change.

As organizations dedicated to producing meaningful and socially significant improvement in client behavior, it is important that ASAI agencies involve guardians and family members in the behavior change process (McClannahan et al., 1982; Rohrer et al., 2021). Each ASAI agency is committed to providing support, mentorship, guidance, and training to guardians to promote high-quality functional outcomes. This regular on-going support promotes the generalization of skills from school to home, the acquisition of environment-specific goals, and the ability of families to complete activities together. Each agency assigns an instructional staff member to each client’s family to provide regular home-programming services for the year.

To evaluate the extent of support in the home and community, each ASAI agency annually evaluates the number of hours of home programming and the presence of individualized programs targeting parent-selected goals in the home setting. Data are collected on the number of sessions and hours of one-on-one training that are delivered to a guardian or other caregiver each year to determine if all families receive regular support and mentorship. Data are also collected on the number of individualized home programs for each client within each agency and the percentage of programs leading to desired behavioral outcomes in the home and community. These home programs are similar to the previously described instructional programs in the education setting, in that they target specific client behavioral responses and are documented in the same technologically sound manner to allow for assessment of outcomes. Collectively, these data are used by each agency to evaluate the extent to which the instructional staff have provided ongoing support to families and guardians through regular visits to the home and direct teaching of new target responses in the home and community environments. Table 8 presents the range and annual mean total hours of home programming across the 10-year time span for each ASAI agency across their entire client population, as well as the mean number of home programming hours per client. The range and mean annual total number of home programs are also presented for the same 10-year time span, as well as the mean annual number of home programs per client.

These data indicate some variability in the number of hours of home programming provided across agencies, as well as across years. However, all clearly exceed the ASAI minimum requirement of 22 one-hr visits to the home per year, per client, and at least one active home program per client. A wide variety of factors influence home programming data, including age of the client (e.g., families with young children often require and receive more hours of home programming), accessibility of the guardian/family (e.g., families with two working parents vs. a stay-at-home parent), afterschool activities (e.g., gymnastics class, swimming), and general client/family need. Each agency is committed to providing individualized support based on these variables. The COVID-19 pandemic greatly affected home programming, because many services were provided virtually in the home setting, explaining increases in the number of hours and home programs for some agencies. Outcome data from the home programs is one QD-KPI used to evaluate an ASAI agency and can be found in the section on Performance Indicators and Outcome Standards.

As noted by Wolf (1978), change in client performance can only be deemed important and meaningful when consumers are satisfied with the process and results. To that end, ASAI agencies annually collect data from a number of consumers, including staff, guardians, and governing board members. A QD-KPI for all of the agencies is the percentage of consumers satisfied with service delivery. Consumer ratings and feedback about aspects of the implemented model and leadership are obtained each year allowing the agency to evaluate the need for strategic changes to meet consumer needs and desires, aligning with Step 4 in Silbaugh and El Fattal’s (2022) call to action . In addition, as new strategies are implemented the consumer data collected can be used to ascertain satisfaction and effectiveness of new initiatives.

The process of consumer evaluation involves sending surveys annually to each of the consumer groups with both general questions (e.g., How satisfied are you with the pleasantness and helpfulness of staff?) and consumer-specific questions (e.g., How satisfied are you with the staff evaluation process? or How satisfied are you with the communication you have with your home programmer?). All consumers receive either a paper survey or a digital survey that asks them to answer the questions found in Table 9 using a 7-point Likert scale similar to that in Table 5. Consumer evaluation surveys are sent in the spring of each year. Each consumer is asked to complete the survey, make comments, and return the survey to the evaluation administrator. Follow-up reminders are sent to all consumers to increase returns rates, the criterion is 80%. Return rates typically exceed this criterion for staff members, guardians, and Board of Trustee members. Responses are confidential and only aggregate data are shared with team members to ensure anonymity of the responder. The data are summarized by agency for both the satisfaction ratings and the written comments from the consumers themselves. The review of agency data allows administrators to determine if the agency is meeting strategic goals and identify future changes in strategy, when necessary, to improve consumer satisfaction. This is a necessary step for organizational leadership to take to ensure analysis of performance and feedback into the system, as suggested by Silbaugh and El Fattal (2022) in Step 6 of their call to action.