Evaluation of the accuracy and robustness of large language models in dental examinations: sensitivity analysis of question format and evaluation parameters

INTRODUCTION

The evaluation of large language models on examination questions has become a standard way to estimate their clinical and educational competence, and the method has been applied across medicine, from the United States Medical Licensing Examination to national board examinations in many specialties. The appeal of the approach is that examination questions are a fixed, scored and clinically meaningful instrument, so a model's accuracy on them can be compared with a known passing standard and with the performance of human examinees, and a reader can therefore judge not only whether the model answered correctly but whether it would have passed, which is the question that a dental educator or a regulator would actually ask. The method also has a known weakness, which is that a single accuracy estimate is sensitive to the choice of questions, the prompt and the decoding settings, and a benchmark that reports one number without testing its robustness reports less than it appears to, because the same model could have scored differently under a different but equally defensible protocol.

The weakness is not a reason to abandon the method, but a reason to report the sensitivity of the estimate alongside the estimate itself, so that the reader can see the range within which the answer is stable and the dimensions along which it is not. This study adopts the examination-question method and, in place of a single estimate, reports the main result together with the sensitivity analyses that bound it, so that the reported accuracy is a range of defensible estimates rather than a bare number.

The dental setting is a natural and, so far, underused test of the method. Dentistry has its own examination instruments and its own knowledge base, and the performance of large language models on dental questions has been reported in a growing but heterogeneous literature. Lee et al.(1) and Lee et al.(2) evaluated ChatGPT, Gemini and DeepSeek on English-translated dental examination questions, Kim et al.(3) compared a large language model with medical students, and the systematic review of Dashti et al.(4) synthesized the accuracy of large language models on dental examination questions, while Ribeiro et al.(5) and Acar et al.(6) benchmarked the models against dental examinations directly.

The dental radiology and pediatric dentistry questions have been studied separately by Sözen et al.(7) and the pediatric dentistry benchmarks of Kayaci et al.(8) and Khanagar et al.,(9) and the evaluation of dental students by Carstea et al.(10) completes the picture. The models evaluated in these studies range from the proprietary general-purpose models to the open reasoning models, and the diversity of models is itself a reason to expect the accuracies to vary, which is why the robustness of any single estimate, rather than the estimate itself, is the more informative quantity. What this literature shares is that most studies report a single accuracy without the sensitivity analyses that would make the estimate robust, and the present study addresses that gap by making the sensitivity analyses the object of the study rather than an afterthought.

The gap matters because a single accuracy is not a property of the model alone but of the pairing between the model, the question set and the protocol, and two studies of the same model can report different accuracies without either being wrong, simply because their protocols differ. The sensitivity battery is the device that separates the model's true accuracy from the protocol's contribution, and its absence from most of the dental literature is the specific defect that the present study corrects.

The methodological choice made here, and justified against the alternatives, is a zero-shot benchmark with a fixed prompt and a fixed random seed, followed by a battery of sensitivity analyses. The zero-shot setting was preferred over few-shot prompting because it is the setting a practitioner would actually use, and it avoids the selection of exemplars that few-shot prompting requires, a selection that would itself become a source of variation that the study would then have to explain. The fixed seed was preferred over a new random sample because it makes the question set reproducible, and a benchmark whose questions cannot be reproduced cannot be verified.

The sensitivity battery was preferred over a single estimate because the whole point of the study is to measure how much the answer depends on the choices that a benchmark makes, and a study that reports a single number leaves those dependencies invisible. These choices are described in the methods and their consequences are quantified in the results, so that a reader who would have made a different choice can see, in each case, what difference the alternative choice would have made. The objective of this study is to estimate the zero-shot accuracy of a large language model on dental examination questions and to test the robustness of that estimate to question format, decoding temperature and sample size, so that the reported accuracy is a bound rather than a bare number, and the objective is served by reporting, alongside every main result, the specific sensitivity analysis that tests it.

METHODS

RESULTS

Main result

The model answered 70 of the 100 dental examination questions correctly, for an accuracy of 0.700 (95% CI 0.604 to 0.781), with 30 answered incorrectly. The width of the confidence interval reflects the sample size of 100, and a reader who needs a narrower interval would need a larger sample, which the present study's design did not pursue because the sensitivity of the point estimate, rather than the precision of a single number, was the object of the study from the outset. Under the specification of a higher decoding temperature of 0.7, the accuracy remained essentially unchanged at 0.690 (95% CI 0.594 to 0.772), so the main estimate was not an artefact of the low temperature chosen for the primary run, and the temperature change moved the estimate by only 0.010 in the point value. The sensitivity of the estimate to temperature is shown in (Figure 1), and the main and sensitivity estimates are summarized in (Table 1).

The main and sensitivity estimates are summarized in (Table 1).

Analysis	n	accuracy	95% CI
Main (temperature 0)	100	0.700	0.604-0.781
Single-choice	60	0.733	0.610-0.829
Multi-choice	40	0.625	0.470-0.758
Temperature 0.7	100	0.690	0.594-0.772
50-question subsample	50	0.700	0.562-0.809

Note: CI = confidence interval.

DISCUSSION

The principal finding is that the large language model answered dental examination questions with an overall accuracy of 0.700, and that this single estimate was robust to decoding temperature and to sample size but sensitive to the format of the question asked. The robustness to temperature and sample size is the more important of the two results, because it means the benchmark is not an artefact of the specific choices made in the primary run, and a reader can trust the 0.700 as a stable property of this model on this question set rather than as a fluke of the protocol or of a single run of the model.

The sensitivity to format is the more informative of the two results, because it shows that the model's apparent competence depends on the kind of question asked, and any single number would conceal that dependence. A reader who takes the 0.700 at face value learns less than a reader who also knows that the model scored 0.733 on single-choice questions and 0.625 on multiple-choice questions, and that the estimate would have been the same with half the questions, and the three numbers together are the finding rather than any one of them alone.

The robustness results can be stated in the language of the sensitivity analyses themselves. Across all scenarios, the accuracy ranged between 0.625 and 0.733, and the two scenarios that changed the sample or the temperature, the 50-question subset and the higher temperature, produced estimates that remained within 0.010 of the main estimate.

The format scenario was the only one that moved the estimate meaningfully, and it moved it by the mechanism that would be expected, because a multiple-choice question with several correct options is harder to answer with a single letter, and a model instructed to return one letter will underperform on a question that expects more than one. The sensitivity battery therefore does more than bound the main result; it explains where the main result comes from, and it does so without requiring a second model or a second question set, which is the economy that a well-chosen sensitivity battery provides.

The fact that the temperature change left the estimate nearly unchanged is itself informative, because it indicates that the model's answers are not being drawn from a wide distribution over alternatives but are concentrated on a single answer, which is the behavior expected of a model that knows the answer rather than one that is guessing among plausible options, and it distinguishes the present result from a benchmark in which a small temperature change would have scrambled the answers. The fact that the sample-size change left the estimate unchanged is similarly informative, because it indicates that the 100-question sample was not accidentally unrepresentative of the dental subject as a whole, at least in the aggregate, and it supports the use of the smaller sample in future replications where the cost of the full sample would be prohibitive.

The interpretation of the format difference deserves one further step, because it is the result most likely to be misread. A reader might conclude that the model is simply weaker on multiple-answer questions, but the more precise reading is that the benchmark, not the model, is the source of the difference, because the model was instructed to return a single letter while the multiple-answer questions expect more than one. The 0.625 on multiple-answer questions is therefore a lower bound on what the model could achieve with a prompt that permitted multiple letters, and the present study did not test that prompt.

The format sensitivity is reported as a finding about the benchmark's sensitivity, not about the model's ceiling, and the distinction matters for any reader who wants to compare the present estimate with a study that used a different prompt or a different scoring rule. The distinction also points to the single most useful extension of the present work, a prompt that permits multiple letters on multiple-answer questions, which would test whether the 0.625 is a floor or a ceiling.

The comparison with the dental large-language-model literature places the estimate in context. The systematic review of Dashti et al.( 4) synthesized the accuracy of large language models on dental examination questions, and the studies of Lee et al.,(1,2) Ribeiro et al.,( 5) Acar et al.( 6) and Carstea et al.( 10) each reported accuracies for dental questions in a similar range, with the exact figure depending on the model, the question set and the prompt. The present estimate of 0.700 is consistent with that literature, and the contribution of the present study is not a new accuracy but the demonstration that the accuracy is stable to the choices that most dental benchmarks leave unreported.

The DeepSeek-specific comparisons are the closest to the present study, because Lee et al.(1,2) included DeepSeek in their dental evaluations and Wang et al.( 11) compared DeepSeek with ChatGPT on Chinese medical questions, and both found DeepSeek's accuracy to be in the same band as the present estimate, which suggests that the present result is not peculiar to the particular question set used here. The comparison with the wider medical benchmark literature, from Babacan et al.( 12) and Zhang et al.(13) on medical school courses to the nursing, urology and neurology board examinations of Amankwaa et al.,(14) Eldaneen et al.(15) and Arslan et al.,(16) shows the same pattern, in which accuracies are reported without the sensitivity analyses that would make them comparable across studies.

The orthopaedic, urology, neurology, pediatric and dermatology benchmarks of Arı et al.,(17) Lu et al.,(18) Cho et al.,(19) Schönberg et al.(20) and the allergy and immunology comparison of Carroll et al.(21) each report a single accuracy for a single model and a single question set, and the Chinese national examination comparisons of Huang et al.(22) and the DeepSeek-versus-ChatGPT stability study of Wang et al.(11) are the exceptions that, like the present study, probe whether the estimate survives a change of setting.

The stability study of Wang et al.( 11) is the closest methodological cousin of the present work, because it too reported that the estimate remained stable across iterations rather than drifting, and the convergence of the two studies on stability, from different question sets and different models, is itself evidence that the robustness reported here is not an accident of the dental question set. The Polish specialization examination of Wojtas et al.,(23) the medical-genetics benchmark of Basdemirci et al.,(24) the PeruMedQA dataset of Carrillo-Larco et al.,(25) the image-based oral pathology benchmark of Watanabe et al.(26) and the multi-model pediatric comparison of Kafagi et al.(27) complete the picture, and together they establish that the field has measured accuracy widely but robustness narrowly, which is the gap the present study fills for dentistry.

The behavior of the model during the evaluation is worth one observation, because it bears on how the accuracy should be read. The model is a reasoning model, and it produced an internal chain of reasoning before each final answer, so the scored answer was the conclusion of a reasoning step rather than a direct retrieval.

This means the 0.700 is the accuracy of a model that reasoned its way to each answer, and a reader should not interpret it as the accuracy of a model that merely memorized the answers, because the reasoning step is what the model actually performed. The observation also has a methodological consequence, which is that the reasoning was not scored, and the study therefore cannot distinguish a correct answer reached soundly from a correct answer reached by a faulty route, a distinction that the reasoning analysis of Guo et al.(28) found to be important in medicine generally. The present study is therefore a measure of end-to-end answer accuracy, and its value lies in that measure's robustness rather than in any claim about the quality of the reasoning that produced the answers.

The comparison with studies that used a different method is the most instructive, because it reveals what the present zero-shot benchmark measures and what it misses. Studies that evaluated the model on clinical reasoning rather than on factual recall, such as Guo et al.(28) and Høj et al.,(29) found a larger gap between the model's knowledge and its reasoning, and Atlan et al.(30) reported that the model's performance was fragile to task reformulation, so an accuracy measured on a fixed question set can overstate the model's competence on a reformulated task.

The present study's format sensitivity, the drop from 0.733 to 0.625 between single- and multiple-choice questions, is a specific instance of the same fragility, and it suggests that a dental benchmark that reported only a single-answer accuracy would flatter the model. The present study is therefore best read as a lower bound on the model's competence, one that is robust but incomplete, and not as a claim about how the model would perform in a clinical or educational setting that requires reasoning, and a reader who needs that stronger claim should turn to the reasoning-based evaluations rather than to a benchmark of answer accuracy. The two kinds of study are complementary, and a complete picture of the model would combine the robust accuracy bound reported here with a reasoning-quality assessment reported elsewhere, neither substituting for the other.

The strengths and limitations of the study are best stated together, because they are the two faces of the same design choices. The strength is the reproducibility of the benchmark: a fixed seed, a fixed prompt, a fixed temperature and a deterministic scoring rule, all reported, so that the result can be reproduced and the sensitivity analyses re-run by any reader with access to the same dataset and model.

The corresponding limitation is that the benchmark measures one model, one question set and one prompt, and the generalizability of the 0.700 to other models, other dental question sets and other prompting strategies is not established by this study alone, a limitation that the dental literature shares and that the present study makes explicit rather than leaving implicit.

The use of the MedMCQA validation split, which is a public and widely used instrument, is a strength for comparability but a limitation for breadth, because the dental questions it contains may not represent the full range of dental examinations, and a national board examination may be more or less difficult than the validation split.

The absence of human examinees is a limitation, because the study cannot say how the model compares with dental students or practitioners, and the absence of a reasoning analysis is a further limitation, because the study measures answer accuracy rather than the quality of the reasoning behind the answer, so a correct answer arrived at by a wrong route is scored the same as a correct answer arrived at soundly.

These limitations bound the interpretation, and none of them changes the central finding that the estimate was robust to temperature and sample size and sensitive to format, because the robustness and the sensitivity were measured within the study itself rather than asserted about it.

The educational implication of the estimate is modest and specific, and it is worth stating because the dental literature sometimes over-reads a benchmark accuracy. An accuracy of 0.700 on examination questions does not license a conclusion that the model is ready to teach, to diagnose or to support clinical decisions, and the present study's own format sensitivity shows why: the model is weaker exactly where the questions are harder, and the benchmark measures answer selection rather than clinical judgment. What the estimate does support is a narrower use, namely as a study aid whose answers are correct more often than not and whose errors should be checked, and as a baseline against which later models can be compared.

A dental educator who uses the model in this way would be using it within the limits that the sensitivity analyses establish, and the present study's contribution is to have drawn those limits rather than to have inflated the number inside them. The limits are the more valuable part of the study for a practitioner, because a number without its limits invites overuse, and a number with its limits invites the cautious use that a benchmark of an educational tool should produce. The comparison with the educational benchmarks of Carstea et al.( 10) and the pediatric benchmarks of Khanagar et al.( 9) reinforces this reading, because those studies also concluded that the model's value is as a supplementary tool rather than as a replacement for human instruction, and the present study adds the sensitivity evidence that such a conclusion requires.

CONCLUSIONS

The large language model answered 100 dental examination questions with an accuracy of 0.700, and the estimate remained essentially unchanged under a higher decoding temperature and a halved sample, while it differed between single-choice (0.733) and multiple-choice (0.625) question formats, so the model's apparent competence is robust to the benchmark's sampling choices but sensitive to the kind of question asked. The principal contribution of the study is not the accuracy itself but the demonstration that it is a stable property of the model on this question set rather than an artefact of the protocol, and that a single bare number would have concealed the format sensitivity that the sensitivity analyses exposed.

References

1. 1 Lee Y, Huang C, Sun A, Chiang C. Performance of ChatGPT-4, Gemini, and DeepSeek-V3 on English-translated questions from the Taiwan national dental Technician Licensing examination over a three-week period. J Dent Sci. 2026;21(3):1511-1519. doi: 10.1016/j.jds.2025.07.3495
2. 2 Lee Y, Huang C, Sun A, Chiang C. Performance of ChatGPT-4, Gemini, and DeepSeek-V3 in answering the prompt-engineered questions from the Taiwan National Dental Technician Licensing Examination over a three-week period. J Dent Sci. 2026;21(3):1573-1582. doi: 10.1016/j.jds.2025.07.3486
3. 3 Kim K, Jeon H, Choi J, Lee D, Jeon Y, Shim B. Artificial intelligence in medical education: Performance of ChatGPT versus medical students on medical examination. Medicine (Baltimore). 2026;105(34):e50323. doi: 10.1097/md.0000000000050323
4. 4 Dashti M, Khosraviani F, Meyari A, Amirzade-Iranaq MH, Chaurasia A, Hefzi D, et al. Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis. International Dental Journal. 2026;76(4):109609. doi: 10.1016/j.identj.2026.109609
5. 5 Ribeiro CG, de Oliveira RG, Porto FR, Hespanhol FL, Ferraz Júnior AML. Artificial Intelligence in Dental Examinations: Performance Analysis of ChatGPT , Gemini, and Undergraduate Students. Eur J Dental Education. 2026:eje.70319. doi: 10.1111/eje.70319
6. 6 Acar NK, Sengul F, Celikel P, Erdogan AS. Evaluating the Accuracy of Large Language Models in Dentistry: A Multi‐Model Study Using Clinical Questions From Turkey's Dental Specialty Exams. Eur J Dental Education. 2026:eje.70295. doi: 10.1111/eje.70295
7. 7 Sözen E. Large language model accuracy in dental radiology: effects of cognitive complexity and content domain. Sci Rep. 2026;16(1):30024. doi: 10.1038/s41598-026-63160-4
8. 8 Kayaci ST, Ilhan HO, Tassoker M, Cap T, Demir H. Benchmarking GPT-5, Gemini 2.5 Pro, Grok 4, and other LLMs on pediatric dentistry questions from a dental specialization exam. Sci Rep. 2026;16(1):26603. doi: 10.1038/s41598-026-57392-7
9. 9 Khanagar SB, Al-Ehaideb A, Almutairi N, Alqahtani L, Maganur PC, Vishwanathaiah S, et al. Feasibility and exploratory assessment of large language models for pediatric dentistry queries: a comparative study. Front Oral Health. 2026;7:1813936. doi: 10.3389/froh.2026.1813936
10. 10 Cârstea AE, Vasilescu VG, Țâncu AMC, Didilescu AC, Pițuru SM, Imre M, et al. Artificial Intelligence in Evaluating Undergraduate Dental Students: Benchmarking AI Platforms Against Student Performance. Int J Prosthodont. 2026;0(0):1-23. doi: 10.11607/ijp.9742
11. 11 Wang Z, Qin Y, Wu J. Performance stability despite iteration: evaluating DeepSeek and ChatGPT on Chinese medical licensing examinations. Front Med. 2026;13:1874194. doi: 10.3389/fmed.2026.1874194
12. 12 Babacan S. Benchmarking Large Language Models ( LLMs ) Against Medical Students Using Psychometrically Evaluated Authentic Clinically Oriented Anatomy Multiple‐Choice Questions ( MCQs ) at the Apply and Analyze Levels of Bloom's Taxonomy. Clinical Anatomy. 2026:ca.70240. doi: 10.1002/ca.70240
13. 13 Zhang K, Yang W, Zheng W. Comprehensive Evaluation of Large Language Models on Four Core Medical School Courses: A Cross-Sectional Comparative Study. AMEP. 2026;Volume 17:1-10. doi: 10.2147/amep.s642751
14. 14 Amankwaa I, Odoom A, Kasim A, Kobiah E, Diebieri M, Boateng EA, et al. Performance of large language models on nursing licensure examinations: A systematic review and meta-analysis. Nurse Education Today. 2026;165:107154. doi: 10.1016/j.nedt.2026.107154
15. 15 Eldaneen M, Hendy S, Ramadan O, Nikolinakos P, Muneer A, Alnajjar HM, et al. Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study. JMIR Form Res. 2026;10:e100148. doi: 10.2196/100148
16. 16 Arslan İ, Terzi Kumandaş M, Arslan D, Aslan Koca K, Saltoğlu T, Çalışkan SA, et al. Humans vs. large language models in neurology board examination: performance, limitations, and reference reliability. Acta Neurol Belg. 2026;126(4):1401-1412. doi: 10.1007/s13760-026-03102-z
17. 17 Arı B. Evaluation of large language models in a national orthopaedic proficiency examination: Implications for health informatics and medical education. Health Informatics J. 2026;32(3):14604582261470637. doi: 10.1177/14604582261470637
18. 18 Lu H, Zhang Y, Wang Z, Zhao Y, Liu J, Qiu D, et al. Evaluation of ChatGPT-4o’s and DeepSeek R1’s responses to urological problems: a comparative study. BMC Urol. 2026;26(1):212. doi: 10.1186/s12894-026-02228-9
19. 19 Cho J, Sekaran A. Comparison of Large Language Model Performance on the United Kingdom Neurology Specialty Certificate Examination. Cureus. 2026;18(7):e112413. doi: 10.7759/cureus.112413
20. 20 Schönberg NKT, Deschler DJP, Hauer J, Zeumer M. A Comparative Evaluation of Large Language Models on Pediatric Board-Style Examinations. Hosp Pediatr. 2026;16(6):e417-e423. doi: 10.1542/hpeds.2025-009153
21. 21 Carroll M, Kentis S, Kareff H, Schechter C, Jariwala S. A Comparative Analysis of Large Language Model Performance on USMLE Step 1-Style Allergy/Immunology Questions: Evaluating Correctness and Consistency. Appl Clin Inform. 2026;17(04):754-762. doi: 10.1055/a-2946-7393
22. 22 Huang C, Sun Y, Liu W. A comparative study of the performance of different large language models in the Chinese National Pharmacist Licensing Examination. Front Med. 2026;13:1880914. doi: 10.3389/fmed.2026.1880914
23. 23 Wojtas J, Pak G, Górski WJ. Comparative Performance of Large Language Models in the Polish State Specialization Examination in Anesthesiology and Intensive Care Medicine. Cureus. 2026;18(7):e112868. doi: 10.7759/cureus.112868
24. 24 Basdemirci M, Tassoker M. Benchmarking the performance of large language models on medical genetics questions: a comparative evaluation of GPT-5.1, Gemini Pro 3, and Claude Opus 4.5. Front Med. 2026;13:1946188. doi: 10.3389/fmed.2026.1946188
25. 25 Carrillo-Larco RM, Lovón-Melgarejo J, Castillo-Cara M, Bravo-Rocca G. PeruMedQA: Benchmarking Large Language Models (LLMs) on Peruvian Medical Exams—Dataset Construction and Evaluation. MedSciEduc. 2026;36(4):1955-1964. doi: 10.1007/s40670-026-02692-w
26. 26 Watanabe H, Uehara O, Morikawa T, Kojima T, Suga T, Toyofuku A, et al. Performance of large language models on image-based oral pathology questions from the Japanese National Dental Examination. Journal of Dental Sciences. 2025;21(2):879-887. doi: 10.1016/j.jds.2025.08.037
27. 27 Kafagi AR, Atassi N, Kafagi AH. Performance of large language models (GPT-5.2, Gemini 3 Pro, Claude Sonnet 4.6 and Grok 4.1) on the Fellowship of The Royal College of Surgeons Urology Part A examination. BMJ Open. 2026;16(5):e108775. doi: 10.1136/bmjopen-2025-108775
28. 28 Guo Q, Du Y, Dai J, Lin W. Discrepancy between medical knowledge and clinical reasoning: evaluating large language models in prenatal diagnosis. Front Digit Health. 2026;8:1917573. doi: 10.3389/fdgth.2026.1917573
29. 29 Høj S, Jensen C, Meteran H. Clinical performance and practical limitations of large language models in respiratory medicine. Breathe. 2026;22(4):260125. doi: 10.1183/20734735.0125-2026
30. 30 Atılan AU, Çetın N. Beyond accuracy: an educational benchmarking study of task fragility and reasoning stability of large language models on dermatology board-style questions. Cutaneous and Ocular Toxicology. 2026:1-13. doi: 10.1080/15569527.2026.2733364