The role of artificial intelligence in training ENT residents: a survey on ChatGPT, a new method of investigation| ACTA Otorhinolaryngologica Italica

Abstract

Objective. The primary focus of this study was to analyze the adoption of ChatGPT among Ear, Nose, and Throat (ENT) trainees, encompassing its role in scientific research and personal study. We examined in which year ENT trainees become involved in clinical research and how many scientific investigations they have been engaged in.
Methods. An online survey was distributed to ENT residents employed in Italian University Hospitals.
Results. Out of 609 Italian ENT trainees, 181 (29.7%) responded to the survey. Among these, 67.4% were familiar with ChatGPT, and 18.9% of them used artificial intelligence as a tool for research and study. In all, 32.6% were not familiar with ChatGPT and its functions. Within our sample, there was an increasing trend of participation by ENT trainees in scientific publications throughout their training.
Conclusions. ChatGPT remains relatively unfamiliar and underutilised in Italy, even though it could be a valuable and efficient tool for ENT trainees, providing quick access for study and research through both personal computers and smartphones.

Introduction

The worldwide spread of ChatGPT (open artificial intelligence, OpenAI) has been remarkable, and has found applications in various fields, including medicine. ChatGPT has shown potential in assisting medical education, providing quick access to information, and supporting research, also in the field of Ear, Nose, and Throat (ENT).

A recent survey conducted by Temsah et al. ¹ assessed the perceptions of healthcare professionals regarding ChatGPT in Saudi Arabia. The survey revealed that approximately one-fifth of the respondents had used ChatGPT for medical purposes. Interestingly, almost half of participants were familiar with ChatGPT, though their knowledge about this AI tool appeared to be somewhat limited. Furthermore, the survey indicated that there were moderate to low levels of trust in ChatGPT’s ability to generate medical decisions among respondents. Some participants also raised concerns about the potential medicolegal implications associated with the use of this chatbot. These findings underscore the importance of conducting additional research on the adoption of ChatGPT within specific medical specialties and training programmes.

To date, being an ENT resident requires continuous and constant updates due to the vast amount of scientific work published daily. Therefore, finding a way to speed up the search for ENT topics is highly beneficial to trainees ². The growing interest in ChatGPT within ENT education and research is highlighted in a recent study by Park et al. ³. They directly inquired with ChatGPT about its potential clinical applications including its role in supporting clinical decisions, patient education, assist in research, and for literature searches. They were also asked to identify its key limitations, such as data reliability issues, the inability to conduct physical examinations, and potential inaccuracies. Ethical implications were discussed, and suggestions for future implementations were proposed. The study concluded that despite the current limitations, ChatGPT holds significant promise in clinical practice. It has the potential to assist healthcare providers in both clinical practice and training. In fact, several authors have investigated the potential of ChatGPT in facilitating medical education, employing quizzes as a resource for both Orthopaedics and ENT residents ^4,5. In particular, Lum ⁴ tested ChatGPT’s ability to answer questions from the American Board of Orthopaedic Surgery examination. He found that the chatbot would not have passed the written examination initially. However, the AI model exhibited a learning capability when wrong answers were corrected, suggesting its potential to assist and enhance medical education in the future. Hoch et al. ⁵ conducted an examination of ChatGPT’s performance across various Otolaryngology subspecialties. The authors concluded that the integration of AI language models should be approached cautiously and in close cooperation with human experts to ensure reliability and accuracy. Moreover, Dallari et al. ⁶ attempted to pose a specific question to ChatGPT regarding ENT knowledge, emphasizing that the bot is not as precise as it should be about a well-known topic in ENT.

To date, ChatGPT might be a valuable tool for ENT residents to quickly access and review ENT topics during their ongoing education. It may assist residents with exam preparation by summarising key concepts. Additionally, ChatGPT can support ENT research activities, such as conducting literature reviews, suggesting relevant citations, and reviewing manuscript drafts.

A recent study by Lechien et al. ⁷ investigated the reliability of ChatGPT in making clinical decisions. They used the Artificial Intelligence Performance Instrument (AIPI) to assess ChatGPT’s performance in clinical practice, analyzing various clinical cases. They found that the chatbot still has several limitations compared to human discernment in clinical practice. It is crucial to remember that ChatGPT is an AI bot that lacks human knowledge, empathy, and critical thinking. Therefore, despite its numerous potential uses, clinical professionals and patients should be aware of its significant limitations.

However, no studies have specifically examined if and how ENT residents are adopting ChatGPT for research and education purposes. Therefore, this study aimed to assess the familiarity and usage of ChatGPT among Italian ENT residents through a survey. Additionally, the goal was to understand patterns of research involvement throughout ENT training.

Materials and methods

Study design

This is a cross-sectional survey study.

Participants

The sample consisted of Italian ENT residents (609 at the time of the survey), from the first to the fourth year of residency. The total number of trainees was determined based on Ministerial sources for the 4 years of residency (cohorts 2019 to 2022).

Survey development

A short, anonymous survey was created using Google Sheets, a web-based platform (Tab. I). The survey consisted of eight questions designed to capture basic demographics, including the year of training and university of residency. It aimed to explore perceptions of ENT residents of ChatGPT, encompassing their familiarity with the tool, the purposes for which they use it, and its perceived usefulness. Additionally, the survey delved into the potential applications of ChatGPT in research. It also investigated the number of scientific works in which the respondents had participated in and from which year of their residency. The survey used multiple-choice questions, and no personal data was collected from the respondents. It was disseminated via email, Facebook and Whatsapp and limited to one submission for each participant. It remained open for one month from July 1 to July 31, 2023, and multiple reminders were sent during that time to encourage participation.

The questionnaire was circulated among ENT Italian trainees. The survey was sent to at least one resident per institution, requesting them to share it with their colleagues.

Data analysis

Data were collected using a Microsoft Excel spreadsheet (Microsoft, Redmond, Washington, USA), and subsequently analysed using descriptive statistics.

Results

Respondent demographics

The survey had a response rate of 29.7%, with a total of 181 participants among the 609 Italian ENT residents. Among the participants, the breakdown of ENT residents across the four years of training was as follows: 37 (20.5%) in year 1, 50 (27.6%) in year 2, 63 (34.8%) in year 3, and 31 (17.1%) in year 4 (Fig. 1).

The distribution of residents according to their city of residence is shown in Figure 2.

ChatGPT familiarity and use

Among the 181 ENT residents who participated in the survey, 122 (67.4%) of them were acquainted with ChatGPT. Out of these 122 residents, only 23 (18.9%) indicated using ChatGPT for scientific purposes, and an even smaller number, 20 (16.4%) residents, utilised it specifically for studying.

Through the survey, our objective was to identify the field in which residents most frequently employ ChatGPT. The options provided were searching for citations, searching for content, and using it as a language corrector. Seven ENT residents use ChatGPT for citation searches, 8 for content searches, and 18 for language correction purposes. Some of the surveyed ENT residents use ChatGPT for multiple purposes. For each of the fields investigated, the respondents were requested to score the utility of ChatGPT on a scale from 1 to 5. A score of 1 indicated ”not useful at all”, while a score of 5 indicated ”extremely useful”. The same scoring system from 1 to 5 was used to evaluate the usefulness of ChatGPT in studying. Details about the results of the survey are shown in Figure 3.

ENT residents’ research involvement

We also explored the moment during their residency when residents initiated involvement in scientific research and the extent of their participation in various scientific works. The majority of residents began their involvement in scientific work from their first year of residency (n = 125), with fewer starting in the second year (n = 42), and even fewer in the third (n = 8) and last year (n = 6). The average number of scientific publications per residency year was 1.49 for the first year, 2.18 for the second year, 3.37 for the third year, and 3.78 for the last year.

Details are shown in Figures 4 and 5.

Discussion

ChatGPT is an OpenAI model that can be used for various purposes, including information retrieval, text generation, language translation and correction, text summarisation, question-answering, and more.

In the last year, ChatGPT has gained significant interest in the scientific field, particularly in medicine, because of its vast potential to develop specific topics and facilitate faster and easier access to knowledge for healthcare workers.

A recent review delving into the utilisation of ChatGPT in the medical domain has been published. Temsah et al. categorised the papers into eight distinct themes: (1) medical writing; (2) medical education; (3) diagnostic decision-making; (4) public health; (5) scientific research; (6) ethical considerations surrounding ChatGPT use; (7) ChatGPT’s potential for automating medical tasks, and (8) critique of ChatGPT’s usage ¹. As for our survey, we investigated the role of ChatGPT in medical writing and scientific research. They found that the majority of respondents expressed a desire to use ChatGPT in the future to support patient care, medical research, and the appraisal of medical literature. Despite its potential, it is not yet widely used in healthcare, as we found in our survey. Major concerns about its accuracy, reliability, and medicolegal implications still persist. The authors recommend that future studies also consider exploring the impact of ChatGPT on healthcare practice among both familiar and unfamiliar users to gain a more comprehensive understanding of its potential benefits and limitations. The publications available on medical indexing platforms offer a valuable overview of ChatGPT’s potential applications in medical education, scientific research, medical writing, and diagnostic decision-making, shedding light on its usefulness in these domains.

Regarding the field of Otolaryngology, studies concerning ChatGPT are still limited, and the results are diverse. Some articles highlight its utility in specific areas of Otolaryngology, while others underscore its limitations, especially in the clinical application of ChatGPT.

In a recent study by Hoch et al., the precision of ChatGPT’s answers to practice questions for the German ENT board certification was evaluated, revealing variations in performance among different subspecialties within this medical field. ChatGPT demonstrated potential efficacy in providing educational assistance in specific subspecialties. However, it also provided a significant number of incorrect responses within certain Otolaryngology subdomains, rendering it unreliable as the sole resource for residents preparing for Otolaryngology board examinations ⁵.

In another study, Chee et al. investigated the capacity of AI in medical decision-making to identify vestibular causes of dizziness. They presented ChatGPT with eight hypothetical scenarios of patients with dizziness with varying clinical pictures and types of prompts. The AI correctly diagnosed the condition in six of the cases, but with important limitations compared to human practice ⁸.

Additionally, Chiesa-Estomba et al. compared expert sialoendoscopists with ChatGPT to assess the level of agreement in common clinical scenarios encountered in the salivary gland clinic. They demonstrated that ChatGPT is a valuable tool in the clinical decision-making process, especially in the salivary gland clinic, particularly for patients who are eligible for sialendoscopy treatment ⁹. A recent study compared ChatGPT’s responses to ENT clinical cases with those of specialists demonstrating the potential usefulness of ChatGPT in ENT diagnosis. However, the authors highlighted the instability in certain responses and the bot’s inability to recognise some clinical elements as the main limitations of the current system ¹⁰.

Intrigued by the broad applications of ChatGPT in medicine, we pondered its practical implementation in our daily medical studies and research as ENT residents. Therefore, following the approach of Temsah et al. ¹, we developed a survey to share with ENT residents.

Our survey aimed to evaluate the impact of ChatGPT on the training of Italian ENT residents. Our primary purpose was to determine the number of ENT residents using ChatGPT for scientific purposes and their level of satisfaction with the tool. We identified two main areas of usage for ChatGPT by ENT trainees: research and study. We further divided the ”research” domain into three categories: citation searches, content searches, and language review.

The findings from our survey indicate that despite the considerable interest in the potential applications of ChatGPT in the scientific realm, its utilisation in specific areas such as ENT research remains limited.

Indeed, over the course of July 2023, we circulated our survey to at least one ENT resident in each Italian training institution, aiming to personally reach out to as many trainees as possible. Despite sending multiple reminders throughout the month of July, only 181 of 609 ENT residents in Italy responded to our survey, for a response rate of 29.7%.

Significantly, among the 181 respondents, 122 (67.4%) residents were familiar with ChatGPT as an AI tool. A particularly striking observation is that only 23 (18.9%) of the 122 residents who were acquainted with ChatGPT actually employ AI for scientific endeavours. Specifically, 7 trainees use ChatGPT for searching citations or references, while 8 residents use it for searching content. The most commonly used feature of ChatGPT among ENT residents is language correction, with 18 using Open AI for this purpose. Only 20 ENT residents use ChatGPT for study. We also asked our respondents to give a subjective evaluation of the chatbot’s usefulness. As shown in Figure 3C, the feature that gained major positive reports in terms of usefulness was language correction, with 44% of residents answering with a score of 5 (i.e. ”extremely useful”) and 39% with a score of 4. To summarise, the item ”citation search” obtained satisfying responses, with 43% of residents answering a score of 4 and 14% a score of 5, as shown in Figure 3A. Regarding the topic of ”content search,” 50% of residents answered with a score of 4 and 13% with a score of 5 (Fig. 3B). Finally, none answered with a score of 5 for the usefulness of ChatGPT in studying. Instead, 30% of respondents answered with a score of 4, 35% with a score of 3, 20% with a score of 2, and 15% with a score of 1, as shown in Figure 3D.

The analysis of our data reveals that the utilisation of ChatGPT remains modest among Italian ENT residents. Despite its various functions and diverse range of applications, the adoption of ChatGPT for scientific purposes is still confined to a small number of ENT trainees. Some reasons for this limited spread of the chatbot could be linked to the lack of formal guidance on how to best utilise ChatGPT as an AI tool for training and research in the medical field. Additionally, the scarcity of specific literature contributes to a lack of awareness regarding the capabilities and applications of the bot. Furthermore, some studies have raised concerns about the accuracy and appropriateness of information derived from ChatGPT. Residents are accustomed to studying a vast quantity of books and papers using various sources after years of medical school. The introduction of a new instrument provided by AI tools is still finding its place in medical education after years of standardised study methods. Therefore, there is an inherent delay in the integration of AI tools into study and research programmes. While the capabilities of ChatGPT might not have yet gained widespread attention, in the authors’ opinion it holds significant promise for ENT trainees who are seeking innovative tools to enhance their learning experiences. While ChatGPT is widely recognised as one of the most prominent AI systems today, numerous other AI systems are being developed, each with specific features and diverse capabilities. For instance, Humata.ai is a platform capable of extracting information from PDF files uploaded by users, analysing large datasets, and extracting key insights. Wolfram Alpha, another notable AI system, excels at providing expert-level answers to complex questions in mathematics, science, and various other subjects. In the scientific field, Elicit stands out as an AI platform that generates a list of relevant papers from a database of 200 million on specific topics. Another noteworthy system is Jenni, which aids users in composing various types of texts. It is essential to note that knowledge in this field is continually advancing, and a multitude of AI tools are emerging, proving to be increasingly beneficial, such as in the training of ENT residents, each with its distinct set of features.

During the years of residency, a fundamental role in training is occupied by scientific research and publication. In fact, being a resident requires constant information and improvement in knowledge to stay updated and contribute to the innovation and development of new discoveries. Scientific research demands considerable effort due to the continuous innovation and frequent updates across various topics. Today, residents are expected to be dynamic and provide the latest information quickly every day. Using a chatbot such as ChatGPT, which can help assess a vast amount of information and data easily, may be very useful to residents for scientific purposes.

Shaidi et al. ¹¹ performed a national survey in the United Kingdom (UK) to identify patterns in academic publications among ENT trainees during their higher surgical training. It is important to note that in the UK there is a strict standard for scientific production during residency. All ENT trainees are required to participate in two peer-reviewed publications during surgical training to obtain a Certificate of Completion of Training. This requirement reflects the emphasis on academic and research activities in the ENT training programme in the UK. The Italian ENT residency policy does not have a strict requirement for academic publications to complete residency. While scientific production is not compulsory, it is highly recommended during the Italian ENT training programme. As shown by our survey, all the ENT Italian residents are involved in at least one scientific activity since their first year of residency. Thus, while performing scientific research is mandatory in the UK, in Italy there is significant voluntary participation in scientific production, even when it is not compulsory.

With our survey we investigated the involvement of residents in scientific study production throughout their different years of residency. As shown in Figure 4, almost all ENT residents take part in scientific studies from their first years of residency. In fact, 69.1% (n = 125) of residents commenced their involvement in scientific papers during their first year, and 23.2% (n = 42) from the second year. Only 7.7% (n = 14) of the surveyed residents refrained from participating in scientific research until their third year of residency.

It is interesting to observe that the majority of residents (98 of 181) were engaged in one or two scientific projects. A substantial group of residents contributed to three (24 residents) or four (16 residents) studies. Additionally, there are 43 ENT residents who participated in five or more scientific works throughout their residency (see Figure 5).

Analysing our data, we confirmed that the number of scientific works in which ENT residents are involved increases with each subsequent year of residency. In fact, the average number of scientific articles is 1.49 for the first year, 2.18 for the second, 3.37 for the third, and 3.78 for the fourth year of training.

Our investigation highlights that nearly every ENT resident engages in scientific output from the very beginning of their residency, and the quantity of works in which residents participate rises as their years of residency progress, as demonstrated by Shaidi et al. ¹¹ in the context of UK ENT residents.

To conclude, conducting research and staying constantly updated is essential for any medical professional, especially during specialisation. In the field of Otolaryngology, there are still very few studies on the use of ChatGPT in clinical practice, and none that analyse its potential utility in the context of study and clinical research. It is undeniable that ChatGPT accelerates research and study, and it can be a valuable tool for ENT trainees, despite having some limitations concerning the pool of ENT scientific knowledge ^5,6,8,10.

To the best of our knowledge, this is the only study so far in which the usage of ChatGPT by ENT residents for daily research, learning, and updates has been analysed.

Limitations of the study

Being a cross-sectional survey study, our research is subject to some limitations, including sampling, response bias, and recall. The survey sample is quite homogeneous, as it focuses solely on ENT Italian residents. However, it is regrettably limited due to the inability to reach every ENT resident in Italy, resulting in a partially representative sample of 181 trainees out of the total 609 ENT residents in Italy (29.7%). As a result, the generalisability of the study findings to the broader population may be limited. Our study includes respondents from one subspecialty in a single country, and there is a need for larger multi-country and multi-specialty studies.

Response bias may occur if participants misunderstand the questions or if other unmeasured confounding factors influence the results. It is worth noting that some responses may be influenced by participants’ familiarity with the chatbot, and some residents may not have completed the survey due to a lack of interest in the topic. To the best of our knowledge, respondents accurately and candidly completed the survey, making the results trustworthy. The anonymity of the survey responses ensured that participants felt comfortable providing honest and accurate responses. However, a limitation is that we did not collect specific information from the respondents due to privacy concerns.

Our survey investigated generic topics related to the use and the utility of ChatGPT for ENT trainees, so the responses do not offer an in-depth analysis of the modality of training or scientific production. A more specific survey with more articulate questions is needed to obtain more precise data.

Another relevant limitation is the absence of a control group. Our study focused solely on the use of ChatGPT, without a comparison group of ENT residents who do not use ChatGPT. Without such a comparison, it is challenging to draw conclusions about the true impact of ChatGPT on the education and research activities of ENT residents and whether it genuinely contributes to improved educational outcomes or research productivity. Our study aimed to investigate the general familiarity and the primary uses of ChatGPT in the educational training and scientific production of ENT Italian residents. However, further studies are necessary to fully understand its usefulness and advantages, including comparisons with the results of residents who do not use this tool.

Conclusions

Despite its potential value and efficiency as a tool for ENT trainees, ChatGPT is not yet widely recognised or extensively utilised in Italy, according to our data. ChatGPT can provide ENT trainees with swift access to study materials and research information via personal computers and smartphones. Our study reveals that ChatGPT is still relatively limited in its adoption among ENT residents for study and research purposes, given the small size of our sample and the numerous study limitations. These findings can serve as a valuable starting point for future, more extensive studies that delve deeper into the prevalence of ChatGPT and other AI usage and its actual influence on the training and scientific productivity of residents.

Scientific research is essential for the advancement of knowledge, and it is crucial for ENT trainees to be involved from the earliest years of training. Our data shows an overall relevant involvement of ENT residents in scientific production since their first year of residency, and thus ENT residents are actively engaging in research activities and contributing to scientific advancements in their field.

AI tools, like ChatGPT, can provide rapid and reliable access to the world of research and study, but one should always be aware of its limitations and possible mistakes due to its intrinsic limits. However, AI still cannot replace human thinking. To date, reliability of AI in scientific research has yet to be proven and it is essential to remain critical and thoroughly validate AI-generated results when used in scientific studies.

Acknowledgements

The authors would like to express their gratitude to all the residents of the ENT Italian schools who participated in the survey and provided valuable insights for the study. The authors would like to thank the experienced professional native-English speaking editor, Dr. Loulwa Sili, who proofread our manuscript.

Additionally, we would like to express our appreciation for the support of ChatGPT, an AI language model, during the composition of this manuscript. ChatGPT offered valuable recommendations to enhance the language and coherence of the manuscript.

Conflict of interest statement

The authors declare no conflict of interest.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author contributions

VD, CL: conceived the study; VD, CL, FDC: collected the data and wrote the paper; RN, VA, LS, DM: supervised the work and corrected the paper.

All the authors have read and approved the manuscript.

Ethical consideration

This is an observational study. The Provinces of Verona and Rovigo Research Ethics Committee has confirmed that no ethical approval is required.

Figures and tables

Figure 1.Distribution of residents by year of residency.

Figure 2.Distribution of residents based on their city of ENT residency.

Figure 3.A) Usefulness of ChatGPT in citation/references search on a scale from 1 to 5; B) Usefulness of ChatGPT in content search on a scale from 1 to 5; C) Usefulness of ChatGPT in language review on a scale from 1 to 5; D) Usefulness of ChatGPT in studying on a scale from 1 to 5.

Figure 4.Year of residency in which ENT residents started collaborating on scientific work.

Figure 5.Number of scientific works in which ENT residents are involved. The x-axis shows the number of scientific papers in which the trainees took part, and the numbers on the histograms indicate the number of trainees.

References

1. Temsah MH, Aljamaan F, Malki KH. ChatGPT and the future of digital health: a study on healthcare workers’ perceptions and expectations. Healthcare (Basel). 2023; 11:1812. DOI
2. van Dis EAM, Bollen J, Zuidema W. ChatGPT: five priorities for research. Nature. 2023; 614:224-226. DOI
3. Park I, Joshi AS, Javan R. Potential role of ChatGPT in clinical otolaryngology explained by ChatGPT. Am J Otolaryngol. 2023; 44:103873. DOI
4. Lum ZC. Can artificial intelligence pass the American board of orthopaedic surgery examination? Orthopaedic residents versus ChatGPT. Clin Orthop Relat Res. 2023; 481:1623-1630. DOI
5. Hoch CC, Wollenberg B, Lüers JC. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. 2023; 280:4271-4278. DOI
6. Dallari V, Liberale C, De Cecco F. Can ChatGPT be a valuable study tool for ENT residents?. Eur Ann Otorhinolaryngol Head Neck Dis. 2023;S1879-7296(23)00140-0. DOI
7. Lechien JR, Maniaci A, Gengler I. Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI). Eur Arch Otorhinolaryngol. 2024; 281:2063-2079. DOI
8. Chee J, Kwa ED, Goh X. “Vertigo, likely peripheral”: the dizzying rise of ChatGPT. Eur Arch Otorhinolaryngol. 2023; 280:4687-4689. DOI
9. Chiesa-Estomba CM, Lechien JR, Vaira LA. Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. Eur Arch Otorhinolaryngol. 2024; 281:2081-2086. DOI
10. Dallari V, Sacchetto A, Saetti R. Is artificial intelligence ready to replace specialist doctors entirely? ENT specialists vs ChatGPT: 1-0, ball at the center. Eur Arch Otorhinolaryngol. 2024; 281:995-1023. DOI
11. Shahidi S, Osborne MS, Jama GM. Experiences in academic publication among ENT trainees in the UK: results from a national survey. J Laryngol Otol. 2021; 135:396-402. DOI