ENTrain: an online otolaryngology work based assessment tool

Introduction

Setting training standards, overseeing examinations and certification remains a continual responsibility for medical colleges, surgical colleges and regulatory boards. Ensuring that trainees meet these rigorous requirements is crucial for the profession. Historically, these organisations have relied heavily on end-of-training examinations or assessments conducted throughout the training period, requiring candidates to demonstrate competence at specific points in time. In recent decades, there has been a shift towards valuing Workplace-Based Assessments (WBAs) performed by colleagues who directly observe and support the trainee’s practical skills in daily tasks. In Otorhinolaryngology Head and Neck Surgery (ORLHNS) training in Australia and Aotearoa New Zealand (AoNZ), such formative assessments were limited to four assessments performed only in the first year of training prior to 2012. To evaluate trainees across a range of competencies in accordance with the Royal Australasian College of Surgeons (RACS) objectives and guidelines, a variety of WBAs were introduced. These include Direct Observation of Procedural Skills (DOPS), Mini-Clinical Evaluation Exercise (Mini-CEX), Procedure-Based Assessment (PBA), and Case-Based Discussion (CBD). The mandated number of WBAs has gradually increased; currently, under the January 2023 assessment guidelines for the usual five-year training pathway, trainees are required to submit a minimum of 126 formative assessments before being eligible to sit for the RACS Fellowship Examination (FEX) (1,2).

The increased use of WBAs in surgical training has been associated with variable trainee and trainer perceptions of their educational value, both internationally and within the AoNZ training context (2-6). Reported barriers include structural and environmental factors, such as limited time within busy clinics and operating lists and variation in opportunities for completing mandatory assessments across training locations. The educational value of WBAs is also influenced by trainee and trainer engagement, with a perceived ‘tick-box’ culture potentially prioritising assessment numbers over assessment quality. Administrative and technological barriers, including cumbersome processes, poor form design and reliance on paper-based systems, may further reduce usability. However, when based on direct observation, adequately supported and accompanied by meaningful feedback, WBAs provide valuable learning opportunities and support trainee development (2-6).

When first introduced, WBAs were completed using paper forms and mailed to the local RACS office. This process was time-consuming for trainees, trainers and supervisors and generated a substantial administrative burden associated with receiving, verifying, recording and storing completed assessments (3,4). The 2019 Training, Education and Accreditation Committee (TEAC) Chairperson's Report noted that correspondence from RACS indicated that receiving, verifying, recording and storing paper-based WBA forms required administrative support equivalent to approximately 20 hours per week for a cohort of 17–20 trainees (7). In response to increasing workforce demands, the number of ORLHNS training positions has increased to 24 [2025) with further expansion expected. This growth was anticipated to result in further increases in the administrative workload associated with processing and managing paper-based assessments.

To maximise the usefulness of WBAs, training committees and governing bodies must identify and address the barriers that undermine their educational impact. Barriers such as the time pressures faced by clinicians and trainees are difficult to address rapidly at a national level; however, systems can be improved to reduce administrative burden and eliminate minor inefficiencies.

Streamlining assessment processes may allow trainers and trainees to devote greater attention to clinical evaluation and feedback rather than administrative tasks. A recent study of ORLHNS trainees in AoNZ has suggested that web-based assessments may help address these challenges. In the United Kingdom, trainees have implemented digital assessment tools that decrease reliance on physical paperwork and minimise the risk of losing completed forms. Additionally, these tools simplify storage and enable more efficient analysis of trainee progress. The introduction of web-based assessment tools has generally been positive, provided the assessment tool is reliable and functional (8).

As an initial step towards addressing barriers to effective assessment, the TEAC developed and introduced “ENTrain”, a web-based assessment platform designed to replace the existing paper-based WBA system. Although the ORLHNS training programme operates across Australia and AoNZ, ENTrain was developed and implemented within the AoNZ component of the programme. This system was intended to address practical limitations associated with paper-based assessments by reducing administrative burden and improving usability, thereby allowing greater focus on feedback, learning, and supervision.

It was anticipated that there would be a period of adjustment with the introduction of any new system. While access to the internet and digital technology is widespread in AoNZ, assumptions should not be made about individuals’ access or competence in using the digital technology. Ability to use technology may not directly transfer to ability to use the digital tools (9). Any new system must offer an improvement in efficiency and be acceptable to supervisors, trainers and trainees, otherwise, it may not address the perceived barriers to useful WBAs. This study aimed to compare trainee and consultant experiences before and after implementation of ENTrain for ORLHNS training in AoNZ and hypothesised that transition to a web-based platform would improve usability and reduce perceived administrative burden.

Methods

This study is reported according to the STROBE reporting guidelines (available at https://www.theajo.com/article/view/10.21037/ajo-2026-1-0014/rc).

Two cross-sectional surveys were conducted before and after implementation of the web-based assessment system, ENTrain. The survey was developed and distributed via SurveyMonkey (SurveyMonkey Inc., San Mateo, CA, USA) and comprised three sections. Section I collected demographic data including age, gender, post-fellowship years and number of assessments completed in the previous rotation. Section II comprised closed-ended questions about the assessment with responses measured using a five-point Likert scale ranging from strongly disagree (one) to strongly agree (five). Section III included open-ended free-text questions inviting respondents to share both positive and negative experiences. The draft questionnaire underwent iterative review by all authors to assess content relevance, clarity, and face validity. Each author independently completed the survey prior to distribution, and minor revisions were made to improve question wording, survey flow, and usability before final deployment via SurveyMonkey (Figure 1).

Figure 1 Survey instrument used to evaluate workplace-based assessment experience among OHNS trainees and consultants in Aotearoa New Zealand. OHNS, Otolaryngology Head and Neck Surgery.

The primary outcomes were responses to the Likert-scale items in Section II. Secondary outcomes were themes derived from the open-ended responses in Section III. The first survey was conducted in August 2022, 6 months prior to implementation of ENTrain. The second survey was administered in August 2024, 18 months (three full surgical rotations) after implementation.

Consultants are vocationally trained surgeons and may also be referred to as Senior Medical Officers (SMOs). In this study, “trainer” refers to consultants who supervise, assess and teach ORLHNS trainees, whereas “supervisor” is used in accordance with the RACS definition to refer to those who manage, assess and support ORLHNS trainees. In AoNZ, supervisors comprise most members of the TEAC.

Eligible participants comprised all current trainees, regional TEAC members (supervisors), and other ORLHNS consultants involved in surgical training (trainers). All eligible participants were invited by email to minimise selection bias and ensure representation across accredited training centres. Email distribution was coordinated by the Secretary of the New Zealand Society of Otolaryngology Head and Neck Surgery (NZSOHNS). Consultants working solely in private practice or in centres without accredited trainees were excluded. A recent workforce survey identified 121 active ORLHNS consultants and 24 trainees in AoNZ, of whom 78 consultants worked in centres with trainees and were therefore eligible as trainers (10).

The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was submitted through the New Zealand Health and Disability Ethics Committees (HDEC) online screening process and was determined to be out of scope for HDEC review (reference No. 2026 OOS 26290). Formal ethics approval was therefore not required because the study involved voluntary, fully anonymous cross-sectional surveys and collected no identifiable personal data. Informed consent was taken from all individual participants.

Data analysis

Individual Likert-scale responses were analysed using two-sided Mann-Whitney U tests because responses were ordinal. Means with 95% confidence intervals (CIs) were additionally reported as descriptive estimates to facilitate interpretation; these estimates should be interpreted in the context of the ordinal response scale. For positively worded survey items, higher scores represented more favourable perceptions, whereas for negatively worded items, lower scores represented more favourable perceptions. Mann–Whitney U statistics, P values and rank-based effect sizes (r) were reported for between-cohort comparisons. Effect size r was calculated as |Z|/√N, where Z was the standardised Mann–Whitney statistic and N was the total number of valid responses across both cohorts. Effect sizes were interpreted as negligible (r<0.10), small (r=0.10 to <0.30), moderate (r=0.30 to <0.50), or large (r≥0.50). The number of forms completed during a six-month rotation was categorised into ordinal groups and compared using Pearson’s Chi-squared test. Statistical significance was defined as P<0.05. Statistical analyses were performed using jamovi version 2.7 (The jamovi project, 2025) (11).

Open-ended survey responses were analysed using Braun and Clarke’s six-phase thematic analysis framework (12). Responses were first reviewed to achieve familiarisation with the data. Initial coding was then supported using artificial intelligence (AI) tools (ChatGPT Plus, OpenAI; Microsoft Copilot Pro, Microsoft; and Google NotebookLM Plus, Google). Standardised prompts were used to identify recurring concepts and patterns from the de-identified free-text responses. AI-generated codes and candidate themes were used only as an adjunct to the analysis and were independently reviewed against the original responses by the research team. Themes were subsequently refined, discussed, and verified by the authors, with disagreements resolved through consensus. Final themes were determined solely by the research team after comparison with the raw data and AI-generated outputs to ensure completeness and consistency.

All submitted surveys, including partially completed surveys, were included for analysis. Analyses were conducted using available responses for each item, with denominators reported accordingly.

Results

A total of 71 responses were collected for the first survey on paper-based assessment and 58 responses for the second survey on web-based assessment. There were 20 trainee responses in both surveys. Two respondents in the first survey and three in the second had partial completion of the survey with only the demographic information provided.

Approximate consultant response rates were 65.4% (51/78) for the paper-based survey and 48.7% (38/78) for the ENTrain survey. The trainee response rate was 83.3% (20/24) in each survey. The demographic characteristics of respondents were generally similar across the two survey cohorts, although differences were observed in age and gender distribution (Table 1). The proportion of respondents aged >55 years was lower in the second survey than in the first [17.5% (10/57) versus 24.6% (17/69)], while the proportion of female respondents was higher [42.1% (24/57) versus 27.5% (19/69)].

The distribution of the self-reported number of WBAs completed by consultants during a six-month rotation differed following implementation of ENTrain [χ²(2)=8.45; P=0.012]. Although fewer than 10 WBAs remained the most frequently reported category in both survey cohorts, the proportion of consultants reporting fewer than 10 assessments decreased from 92.0% (46/50) with the paper-based system to 81.6% (31/38) following implementation of ENTrain.

Most respondents reported that assessments took less than 30 minutes to complete (89.7% (61/68) in the paper-based survey versus 92.7% (51/55) following implementation of ENTrain).

Mean scores were higher for all positively worded survey items and lower for all negatively worded items following implementation of ENTrain (Table 2). However, not all between-cohort differences were statistically significant.

Respondents gave higher ratings for ease of locating WBAs following implementation of ENTrain, although the difference was not statistically significant [paper-based mean 3.29 (95% CI: 3.08–3.50) versus ENTrain mean 3.54 (95% CI: 3.25–3.83); U =1,481; P=0.089; r=0.16].

Respondents gave significantly higher ratings for ease of completion following implementation of ENTrain [paper-based mean 3.60 (95% CI: 3.41–3.80) versus ENTrain mean 3.91 (95% CI: 3.70–4.12); U =1,491.5; P=0.027; r=0.20]. Ratings for the statement that WBAs supported learning and training were also higher following implementation of ENTrain, but the difference was not statistically significant [paper-based mean 3.56 (95% CI: 3.37–3.75) versus ENTrain mean 3.80 (95% CI: 3.53–4.07); U =1,537; P=0.063; r=0.17].

For negatively worded items, respondents gave lower ratings to the statements that WBAs were time-consuming [paper-based mean 3.18 (95% CI: 2.95–3.41) versus ENTrain mean 2.65 (95% CI: 2.38–2.92); U=1296.5; P=0.004; r=0.28], increased stress [paper-based mean 2.78 (95% CI: 2.55–3.00) versus ENTrain mean 2.44 (95% CI: 2.17–2.70); U =1,492; P=0.036; r=0.17], and had a restrictive format when a suitable case was available [paper-based mean 3.07 (95% CI: 2.86–3.29) versus ENTrain mean 2.66 (95% CI: 2.42–2.90); U =1,300; P=0.006; r=0.24] (Table 2).

Most respondents in both surveys agreed or strongly agreed that WBAs supported learning and training [63.2% (43/68) versus 70.9% (39/55)] and facilitated useful discussion and feedback [72.1% (49/68) versus 76.4% (42/55)] (Table 2 and Figure 2).

Figure 2 Overall comparison between paper-based and online forms.

Thematic analysis

Thematic analysis was completed for open-ended responses to questions on the issues, likes, dislikes and suggestions for each of the assessment formats. Direct quotations are presented verbatim; therefore, respondents’ original terminology, including “SMO”, has been retained. Open-ended feedback was categorised into four main themes (Table 3). Respondents also provided suggestions for improvement.

Administrative and system factors

While present in the initial survey, this topic predominated in the second survey. Initial themes of general dissatisfaction amongst both consultants and trainees with the assessment forms being difficult to find and time-consuming progressed to mixed themes of appreciation of the new system “so easy to do online” (consultant, ENTrain survey), “the paperless forms are much better” (trainee, ENTrain survey) and new themes of consultant reluctance to engage with the new web-based system. Although there may have been some improvement with time, one respondent noted: “SMO unwilling to navigate online forms, but better this year” (trainee, ENTrain survey). Both trainees and consultants also identified there was a reduction in environmental impact with the web-based tool.

Assessment design and content relevance

This theme remained consistent across both surveys with concerns from both consultants and trainees that the forms are “too generic”, “formulaic” and that improvements could be made if they were made more tailored to the specific procedure. In particular, many respondents across both surveys noted that it has become a “tick box exercise” which reduces the educational value of the assessment. For example, “Key component is the procedure-specific part but SMO often have 'form fatigue' by the time you get to the last component” (trainee, ENTrain survey).

Educational value and learning objectives

Across both surveys, respondents described WBAs as structured and allowed for a “structured framework for assessment” (consultant, ENTrain survey) and “a format for discussing a procedure/case” (consultant, Paper-based survey) to facilitate feedback between consultants and trainees. Binary or checklist-style grading was seen as limiting the ability to assess progression, nuance, or development over time.

Engagement, culture and behaviour

These themes predominated in trainee responses with many noting the educational value of having an “engaged SMO”, however many noted having “SMO engagement exceptionally challenging” (trainee, ENTrain Survey) which became more explicit in the second survey. Many responses noted consultant difficulties with the new web-based system and “hard to get buy in” (trainee, ENTrain Survey). Challenges with consultant engagement were not limited to the new system with several trainees noting a lack of consistency in the assessments for example, “SMO lack of knowledge of levels of competency” (trainee, ENTrain survey), “Make it clearer what the different PBA levels mean e.g. variability between SMOs re grading 2/3” (trainee, ENTrain survey).

Suggestions for improvement

Trainees and consultants made suggestions for web-based assessments to improve access and user-friendliness. They included mobile optimisation and the ability for trainees to initiate assessments and upload forms or certificates. Suggestions were also made for the web-based assessment forms to be more standardised, simplified and better categorised.

Discussion

There are currently limited studies exploring WBAs in surgical training. This study contributes both quantitative and qualitative data from ORLHNS trainees and consultants in AoNZ, providing unique insight into the experiences of ORLHNS trainees and consultants during the transition from paper-based to web-based assessments.

In AoNZ ORLHNS training, implementation of ENTrain was associated with modest improvements in respondents’ ratings of ease of completion, perceived time burden, assessment-related stress and format flexibility. All statistically significant differences had small effect sizes. However, concerns regarding consultant engagement, educational value and design of WBAs persisted following digital implementation. These findings are consistent with previous literature suggesting WBAs are viewed less favourably when they are “treated as a ‘tick box’ exercise”. WBAs are most effective when consultants are actively engaged and able to provide timely, meaningful feedback. When forms are completed retrospectively, there is limited opportunity for useful feedback (2,5). While practical aspects of assessment delivery have improved with ENTrain, it does not overcome the cultural and behavioural barriers that influence meaningful engagement with WBAs.

When performed well, WBAs can be highly valuable. In both surveys, most respondents perceived WBAs as supporting learning and facilitating discussion and feedback. Despite this, WBAs continue to be perceived negatively both in this study and in the wider literature. Key reported issues include poor understanding of the purpose of WBAs, insufficient time and inadequate training of trainers (13). All factors are influenced by availability, trust and mutual understanding, all of which may be challenged by regular clinical rotations and a dynamic clinical environment (14).

Results from the qualitative data suggest clinical duties and the administrative burden associated with paper-based assessments can lead to disengagement from both the trainees and consultants. This is consistent with the available literature. Approximately 60% of consultant respondents were more than ten years post-Fellowship. Consultants who had not used WBAs during their own training may have had less familiarity with their intended educational purpose; however, previous WBA experience and formal assessor training were not measured in this study. It is recognised in previous studies that variability in consultant engagement and understanding contributes to trainees’ perceptions of the usefulness of WBAs. It is interesting to note that, in this study, engagement, culture and behaviour were less prominent in consultant responses than in trainee responses.

Transition to digital assessment for medical education offers significant advantages and has generally been viewed favourably (15). Respondents have reported improvement in ease of assessment completion and time efficiency along with reduced stress and a decreased sense of restriction while maintaining the educational benefit as found in previous studies (9,16,17). From a supervisor perspective, ENTrain allows tracking of trainee progress and identification of deficiencies in competencies. With regard to the training scheme, web-based assessment also allows for feedback on supervisors and the quality of the curriculum (8). It is important to use this information to further refine the assessments so that the time spent is maximised. For example, several respondents commented that the forms were generic, with large sections of the form perceived to be unnecessary for the assessment. Continual refinement and review may be able to streamline the assessment so that time is only spent on the key components to maximise the educational benefit.

In addition to logistical improvements in assessments and feedback, it was also appreciated that the web-based assessment system would have additional potential environmental benefit. Paper is the largest annual global contributor to healthcare waste (18,19). Reducing reliance on paper forms aligns with broader healthcare sustainability initiatives.

Implementation challenges

Challenges associated with a significant transition to a new assessment platform are not unexpected, these challenges may reflect unfamiliarity with the platform, implementation difficulties and the technical demands associated with transition to a new system. Such transitions create an unfamiliar environment and require consultants and trainees to relearn the assessment process. In a health system under pressure, with substantial clinical demands, there may be limited time for users to familiarise themselves with the web-based platform.

Any new system must be dynamic and able to respond readily to technical issues and user feedback. During implementation of ENTrain, several technical issues were encountered in the early stages of the transition, and adjustments to the platform were made in response to feedback. Initial issues, including password failure and missing trainee profiles, were quickly rectified. Instructional videos were uploaded to assist trainers and trainees in locating and completing the forms. Despite these adaptations, trainees continued to report reluctance among some consultants to engage with the new system. The study identified a difference in the distribution of the self-reported number of WBAs completed by consultants following implementation of ENTrain. Other temporal changes may have contributed to this finding, and causation cannot be established. As respondents perceived web-based assessments to be less time-consuming and easier to complete, consultant engagement may improve as familiarity with the platform increases; however, longitudinal evaluation is required to determine whether this occurs.

Limitations

The overall response rate from trainees was high, while the consultant response rate was moderate, particularly for the follow-up survey. All trainees and ORLHNS surgeons in public training departments across AoNZ were invited to participate. The sample size was determined pragmatically by the available national cohort. Non-response bias is possible, as respondents may have differed systematically from non-respondents in their engagement with or views of WBAs and ENTrain. Respondents may have been either more supportive of or more dissatisfied with the assessment systems.

While the survey had been independently reviewed by the authors prior to distribution, it remains a non-validated survey instrument. Findings should therefore be interpreted cautiously. The transition to ENTrain created anticipated technical challenges. The second survey was intentionally administered 18 months (three training rotations) after implementation so that respondents could focus on the assessment platform rather than its early technical problems. A further survey once consultants are more familiar with the assessment platform and trainees have no experience of a paper-based system would be useful.

The cross-sectional pre-post survey design means responses are captured at specific time points and not a longitudinal follow up of the same respondents. Causation cannot be established as differences may be due to differences in respondent composition and unintended selection bias. The study assessed perceived usability, burden and educational value rather than objective outcomes such as completion time, feedback quality or trainee performance. Because the surveys were anonymous, individual responses could not be linked across time points and the degree of participant overlap between cohorts was unknown. Differences between cohorts may therefore reflect changes in respondent composition or other temporal factors. Free-text survey responses also provide less depth than interviews or focus groups. Multiple comparisons increased the risk of Type I error, and all statistically significant findings had small effect sizes.

Future direction

Overall, respondents were positive about the transition to a web-based system. Although many early technical issues were resolved, respondents continued to identify opportunities to improve platform usability, assessment design, grading consistency and trainer engagement. Ongoing feedback from trainers and trainees may support further refinement of the platform and future evaluations could include objective outcomes such as assessment-completion time and rates, timeliness of completion, quality and specificity of narrative feedback, inter-rater consistency, trainee and trainer engagement, and longitudinal trainee progression.

With an established web-based assessment tool now in place, attention can turn to refinement of the assessments themselves. If the training board remains committed to the ongoing use of WBAs, these assessments should be reviewed to ensure they provide meaningful, effective, and purpose-driven evaluation of trainees. Future work should focus on clarifying the educational purpose and content of WBAs, identifying factors that contribute to consultant disengagement, and identifying the supports required for more meaningful educational interactions between consultants and trainees.

The assessment formats themselves may also benefit from further refinement to ensure they are appropriately tailored to their intended purpose. For example, the PBA format could be reviewed so that a minimum number of assessments evaluate all domains, while subsequent assessments focus more specifically on procedure-specific technical ability and an overall global rating. This may help streamline assessment completion while preserving educational value. In addition, determining the extent of consultant training in WBAs and developing targeted resources to address identified knowledge gaps may improve consistency, engagement, and the overall usefulness of WBAs.

Conclusions

This national study provides insight into the perceptions and reported experiences of consultants and trainees in AoNZ ORLHNS training. Transition from paper-based assessment to ENTrain was associated with modest improvements in perceived usability, including higher ratings for ease of completion and lower ratings for perceived time burden, assessment-related stress and restrictive format. However, all statistically significant differences had small effect sizes, and concerns regarding consultant engagement, assessment design and the educational use of WBAs persisted following digital implementation. These findings suggest that technological redesign may improve aspects of the assessment process but is insufficient in isolation to optimise the educational value of WBAs.

Acknowledgments

None.

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at https://www.theajo.com/article/view/10.21037/ajo-2026-1-0014/rc

Data Sharing Statement: Available at https://www.theajo.com/article/view/10.21037/ajo-2026-1-0014/dss

Peer Review File: Available at https://www.theajo.com/article/view/10.21037/ajo-2026-1-0014/prf

Funding: None.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://www.theajo.com/article/view/10.21037/ajo-2026-1-0014/coif). A.S. serves as an unpaid editorial board member of Australian Journal of Otolaryngology from December 2024 to December 2026. The other authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki and its subsequent amendments. The study was submitted through the New Zealand Health and Disability Ethics Committees (HDEC) online screening process and was determined to be out of scope for HDEC review (reference No. 2026 OOS 26290). Formal ethics approval was therefore not required because the study involved voluntary, fully anonymous cross-sectional surveys and collected no identifiable personal data. Informed consent was taken from all individual participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Australia and New Zealand Surgical Education and Training Program in Otolaryngology Head and Neck Surgery. SET curriculum: guidelines for assessment. 2023. Available online: https://www.nzsohns.org.nz/aonz-ohns-set.
2. Care R. Otolaryngology trainees' experiences of and attitudes towards workplace-based assessments. Auckland: University of Auckland; 2019.
3. Cohen SN, Farrant PB, Taibjee SM. Assessing the assessments: U.K. dermatology trainees' views of the workplace assessment tools. Br J Dermatol 2009;161:34-9. [Crossref] [PubMed]
4. Aryal KR, Currow C, Downey S, et al. Work-Based Assessments in Higher General Surgical Training Program: A Mixed Methods Study Exploring Trainers' and Trainees' Views and Experiences. Surg J (N Y) 2020;6:e49-61. [Crossref] [PubMed]
5. Castanelli DJ, Jowsey T, Chen Y, et al. Perceptions of purpose, value, and process of the mini-Clinical Evaluation Exercise in anesthesia training. Can J Anaesth 2016;63:1345-56. [Crossref] [PubMed]
6. Aryal K, Hamed M, Currow C. The usefulness of work-based assessments in higher surgical training: A systematic review. Int J Surg 2021;94:106127. [Crossref] [PubMed]
7. Garland RC. Training, Education and Accreditation Committee Chairperson's Report. New Zealand Society of Otolaryngology Head and Neck Surgery Annual General Meeting. Dunedin, New Zealand: 2019.
8. Weston PS, Smith CA. The use of mini-CEX in UK foundation training six years following its introduction: lessons still to be learned and the benefit of formal teaching regarding its utility. Med Teach 2014;36:155-63. [Crossref] [PubMed]
9. Oh DD, Bains H, Agostinho N, et al. Utility of digitally supported surgical competency assessments in a work-based setting: a systematic review of the literature. ANZ J Surg 2020;90:977-84.
10. Napier T, Waterhouse D. Urban-rural geographic distribution of Otorhinolaryngologist, Head and Neck surgeons in Aotearoa New Zealand. N Z Med J 2024;137:59-66. [Crossref] [PubMed]
11. The jamovi project. jamovi. Version 2.7. 2025. Available online: https://www.jamovi.org.
12. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3:77-101.
13. Massie J, Ali JM. Workplace-based assessment: a review of user perceptions and strategies to address the identified shortcomings. Adv Health Sci Educ Theory Pract 2016;21:455-73. [Crossref] [PubMed]
14. Andreou V, Peters S, Eggermont J, et al. A needs assessment for enhancing workplace-based assessment: a grounded theory study. BMC Med Educ 2024;24:659. [Crossref] [PubMed]
15. Ang CS, Ito S, Cleland J. Navigating digital assessments in medical education: Findings from a scoping review. Med Teach 2025;47:1274-89. [Crossref] [PubMed]
16. Fokkema JP, Teunissen PW, Westerman M, et al. Exploration of perceived effects of innovations in postgraduate medical education. Med Educ 2013;47:271-81. [Crossref] [PubMed]
17. Benjamin S, Robbins LI, Kung S. Online resources for assessment and evaluation. Acad Psychiatry 2006;30:498-504. [Crossref] [PubMed]
18. Shaw K. Paper waste in health care: reducing health and environmental impact. Infection Control Today. 2024. Available online: https://www.infectioncontroltoday.com/view/paper-waste-in-health-care-reducing-health-environmental-impact.
19. Healthcare Environmental Resource Center. Paper recycling. Washington (DC): HERC; 2015. Available online:.