From the initial cohort of students without training, 26 students out of 51 total students participated in the extra-credit assignment while the second cohort with introductory SolidWorks knowledge only had 21 participants out of 55 students. For each student cohort, participating students had a 100% completion rate. Student responses gauging the previous knowledge and experience possessed in CAD-related software and simulations were assessed to determine a baseline for each cohort. In the untrained cohort, 88.5% of students had zero prior experience with any 3D design software while 11.5% had some experience in 3D design and CAD programs (Fig. 6). An even lower 3.8% of students reported experience in using SolidWorks and no students in the first cohort of untrained students had any experience with SolidWorks Simulations. Due to the SolidWorks module provided in the introductory course, all students in the second group had some experience in 3D design software with 76.9% of students reporting experience with SolidWorks in particular (Fig. 6). Furthermore, 14.3% of students reported experience in SolidWorks simulations from independent exploration outside of coursework. For each skill, differences for untrained students were statistically significant for 3D design, CAD, and SolidWorks (p < 0.05) while trained students had no statistical significance except for SolidWorks Simulations.

Distribution of student experiences in 3D design, CAD programs, SolidWorks, and SolidWorks simulations. In the untrained cohort, > 88% of students in the untrained cohort responded with no experience in 3D design. In the trained student a higher proportion of students indicated some training in 3D design skills

Likert-like data based on responses that allowed students to rate their likeliness, interest, confidence, or difficulty in completing an assignment are represented by assigning each response a number increasing in positivity from 1 to 5. Overall, 51% of untrained students in the Biomechanics course participated in the simulation assignment while 38% of trained students participated. A larger proportion of untrained students participated in the extra credit assignment, and further in-depth analysis of the degree of interest participating students expressed was further investigated using survey data. Student interest prior to the extra credit assignment in the untrained group majorly fell within the range of no interest to a slight interest in CAD-based software (Fig. 7). However, untrained negative responses, which is slight or no interest, collected after the survey decreased from 61.54% to 23.07%. Responses indicating that the student in the untrained group is very or extremely interested, which is considered a positive response, in CAD software increased from 11.54% to 46.15% indicating that untrained students that initially expected the assignment to be of no interest to their careers had responded with increased positivity upon completion of the assignment (p < 0.001). The trained cohort of students experienced a similar trend and distribution of students expressing an interest in SolidWorks and CAD. Students in the trained group decreased in negative responses from 38.09% to 9.52% after completing the assignment (Fig. 7). Positive responses in the trained group increased from 23.81% to 57.14%. Between the trained and untrained cohorts, there was no statistically significant difference in responses between these groups before or after completing the assignment based on Wilcoxon-ranked sum tests (p > 0.05).

Likert plot of student interest in SolidWorks and CAD before and after completion of the assignment for untrained and trained cohorts

The question gauging the likeliness of students to use SolidWorks in their future career followed a similar pattern in the untrained group with responses indicating extremely or somewhat unlikely decreased from 65.38% to 19.23% while responses indicating extremely or somewhat likely increased from 11.54% to 65.39% (p < 0.001) (Fig. 8). Interestingly, in the trained group, most students responded positively both before and after completing the assignment with 52.38% and 85.72% of students expressing extreme or somewhat likeliness to use SolidWorks and CAD, respectively (Fig. 8). The previously larger neutral group of 23.81% of students decreased to 9.52% and the students responding negatively decreased from 23.81% to 9.31%. Between the trained and untrained cohorts, trained students exhibited a statistically significant increase in likeliness to use SolidWorks and CAD both before (p < 0.001) and after (p < 0.01) assignment delivery.

Likert plot of student likeliness to use SolidWorks and CAD in the future before and after completion of the assignment for the untrained and trained cohort. The likeliness to use SolidWorks and CAD increased in both groups with a significantly more severe increase both before and after assignment delivery in the trained cohort

To diagnose the confidence in 3D design and simulation skills students had obtained upon completion of the simulation toolkit assignment, we also asked students to rank their confidence in each skill using a Likert scale. Student confidence in operating SolidWorks, analyzing simulations, and creating simulations between the trained and untrained groups demonstrated a stronger consensus in response compared to other questions. The student confidence in using SolidWorks, analyzing simulations, and creating simulations for the untrained cohort before was heavily biased toward 65.38%, 69.23%, and 57.69%, respectively, of students responding with no or slight confidence (Fig. 9). The trained students, however, responded with 28.57%, 33.33%, and 57.14% slight or no confidence in SolidWorks, analyzing simulations, and creating simulations, respectively (Fig. 9). A larger percentage of students in the trained cohort had responded neutrally to their strength of confidence in using SolidWorks and SolidWorks simulations compared to the untrained group of students. (Fig. 9). Based on a series of Wilcoxon tests between the trained and untrained cohorts, confidence significantly increased in SolidWorks (p < 0.01) and analyzing simulations (p < 0.05) in the trained group. Untrained and trained students exhibited similar confidence in creating simulations, indicating a deficiency in the assignment design for developing this particular skill. Internally, the untrained students were statistically significantly inclined to the slight or no confidence in all skillsets (p < 0.05), while the trained students exhibited no statistical significance in a Kruskal–Wallis test. These data indicate that confidence levels were more evenly distributed, but overall more positive, in the trained cohort.

Likert plot of student responses assessing confidence in using SolidWorks and SolidWorks Simulation software in the post-assignment survey. Confidence in SolidWorks and analyzing simulations increased significantly in the trained group while creating simulations did not significantly differ

Students assessed the usefulness of SolidWorks Simulations, and their ability to complete the assignment, and provided additional feedback after the assignment through the post-survey. Consistent with the increasing positive opinion of SolidWorks and CAD in previous questions, 84.6% of untrained students indicated that SolidWorks Simulations is a very or extremely useful tool (Fig. 10). The trained group responded with a slightly lower 76.2% of students responding positively and a neutral group consisting of 23.8% of students (Fig. 10). There were no negative responses to the usefulness of SolidWorks in the trained group while the untrained group had 15.3% of students expressing that SolidWorks is slightly or not useful. Based on a Wilcoxon test, there is no statistically significant difference between the two groups (p > 0.05). Internally, the trained students had no statistical significance between each group while the untrained students were significantly responded positively.

Student responses regarding their perception of the usefulness of SolidWorks Simulations after completion of the assignment revealed highly similar responses between trained and untrained students

Most students participating in the study also encountered errors but were ultimately able to compare the results of the simulation with their theoretical solutions for the problem, which reduced in the trained group with a consistent 90.5% of students successfully completing the assignment with few errors (Fig. 11). However, responses regarding the ability to compare results, replicate simulations, and whether errors were encountered had no statistically significant differences using a Kruskal–Wallis between trained and untrained for each question (p > 0.05).

Likert plot of student responses in the post-assignment survey regarding the ability to compare simulation to analytical results, ability to replicate the simulation, and whether students encountered errors. The ability and errors encountered increased in the trained group in all areas but were not statistically significant

Questions diagnosing the difficulty students had in completing and working through the assignment and the confidence of students in operating CAD-related software in the future were included to determine the effectiveness of the content delivery method. For the untrained group of students, while most responded that their ability to follow along with the demo videos was easy (50.0%), a large percentage of students also displayed difficulty in following along (34.61%), which may indicate that the videos need either a more in-depth tutorial or a slower delivery (Fig. 12). Furthermore, 76.92% of untrained students indicated that navigating SolidWorks was difficult (Fig. 12). The trained cohort of students, however, overwhelmingly considered navigating the demo videos extremely or somewhat easy with 76.91% of students providing a positive response while navigating SolidWorks was still majorly considered somewhat difficult by 42.86% of students (Fig. 12). Despite the majority of trained students still considering SolidWorks software navigation difficult, a statistically significant increase in ease of navigating SolidWorks was still detected between the trained and untrained cohort (p < 0.01). This significant increase was also present between the trained and untrained groups for ease of navigating demo videos (p < 0.05). Between scores for written instructions and video demos, there was no significant difference between ease of use if the trained group is only considered (p > 0.05). However, ignoring the potential confounding variable of written instructions, the trained and untrained score for navigating video demos is significantly lower than for navigating written instructions (p < 0.05).

Likert plot of student responses in the post-assignment survey grading the ease with which the assignment could be navigated and interpreted for the untrained and trained student cohorts

To determine whether student performance in the course impacts their confidence in their ability to operate SolidWorks and SolidWorks Simulations, the grade distribution of each cohort was assessed for each self-assessed skill. Using linear regression analysis, correlation strengths between increasing confidence in SolidWorks, creating a simulation, and analyzing simulations and grades in the course were determined. For the untrained group, linear regressions detected no correlation between confidence in all groups and course grades (p > 0.05) (Fig. 13). Similarly, the trained group also had no significant correlation (p > 0.05), indicating that standing in the course did not influence confidence responses for 3D design skills (Fig. 14).

Distribution of student self-assessment of confidence assessed in the post-assignment survey in operating SolidWorks, creating simulations, and analyzing the resulting simulations from the post-assignment quiz for the untrained cohort. No significant correlation between student performance in the course and the confidence reported was detected

Distribution of student self-assessment of confidence assessed in the post-assignment survey in operating SolidWorks, creating simulations, and analyzing the resulting simulations from the post-assignment quiz for the trained cohort. No significant correlation between student performance in the course and the confidence reported was detected

Based on the student feedback solicited after the completion of the assignment, the errors encountered were primarily due to long runtime, the inability to access SolidWorks through Citrix, and other technical difficulties in the untrained cohort (Fig. 15). The trained students also reported technical errors related to computer issues in running SolidWorks despite the low percentage of students responding positively to the question explicitly asking if errors were encountered (Fig. 11). Untrained students indicated that more detailed instructions within the assignment that addressed and developed countermeasures for these errors are needed and tended to focus on specific problems encountered in the assignment, which includes issues encountered with particular simulation studies due to either computational limitations or a lack of understanding relating to simulation restriction (Fig. 15). The second cohort of students also requested more detailed instructions focusing on the ineffectiveness of the demo videos alone (Fig. 15). Three students reported that the written instructions were a more effective delivery of assignment instructions and four students recommended that we reduce the number of simulations and increase the detail for each simulation’s set of instructions. However, the second cohort also overwhelmingly emphasized that this assignment effectively illustrated how helpful SolidWorks is (Fig. 15).

Word cloud of student open responses soliciting holistic feedback on the assignment in the post-assignment survey with a minimum frequency of 4 words detected to be present on the graphic