Initial validation of a script concordance test to measure the development of clinical reasoning among physical therapy residents

Main Article Content

Suzanne R. O’Brien
https://orcid.org/0000-0001-8965-178X
Naomi Dillon
Mikkel Linskey
Kevin Lagueras
https://orcid.org/0000-0002-4852-6090
Julie Uhl
Sarah Conroy
Alex Kiley
Nicole Lanzafame
Erin States
https://orcid.org/0000-0001-8891-2802
Emma Sturm

Abstract

Rationale: Clinical reasoning (CR) skills of physical therapy (PT) residents may be limited by a lack of clinical experience, and the complexity of neurologic and elderly patients requires individualized approaches for prescribing walking assistive devices (ADs). Script concordance tests (SCTs), which are case-based assessments that present information that the test taker must examine before answering, can measure the growth in CR. The purpose was to validate a SCT designed to measure CR growth in residents for walking AD prescription.


Methods: Using a prospective cohort design, a panel of 20 experienced practicing clinicians reviewed the test content and provided the correct answers. Neurologic and geriatric residencies were identified from the American Board of Physical Therapy Residencies and Fellowships database. Residency directors were informed of the study, and 27 of them provided access to their residents. Residents gave assent in month 1 and were followed through month 6. The SCT and instructions were emailed at both time points. In total, 27 month-1 tests and 20 month-6 tests were returned. Fleiss kappa was used to evaluate the consistency of the clinician panel’s and the residents’ responses at both time periods. Mean scores for the panel and the residents at months 1 and 6 were calculated. Testing for differences between the clinician panel and the residents at months 1 and 6 was assessed using a Mann-Whitney U test, and testing for differences between the residents at the same time points was assessed using a Wilcoxon signed rank test.


Results: Demographics of the clinical panel and residents are presented. Internal consistency was moderate, and chance agreement was fair in both groups. Mean test scores were 34.8 (4.3), 34.5 (4.1), and 36.3 (3.9) points for the panel, and residents at months 1 and 6, respectively. Concordance of choices on the SCT between the clinician panel and residents was similar at month 1 and grew more concordant by month 6.


Conclusion: The SCT did not differentiate between the practicing clinicians’ and residents’ CR skills to prescribe ADs. Further research comparing the CR of PT residents to PT clinicians in early practice should be undertaken since the residents’ responses were consistent with the practicing clinicians. The unique test construction or the clinician panel, whose members were not required to be board certified, may have introduced error.

References

1. Gilliland S. Clinical reasoning in first- and third-year physical therapist students. J Phys Ther Ed (2014) 28(3): 64–80. doi: 10.1097/00001416-201407000-00009

2. Gilliland S, Wainwright SF. Patterns of clinical reasoning in physical therapist students. Phys Ther (2017) 97: 499–511. doi: 10.1093/ptj/pzx028

3. Wan MSH, Tor E, Hudson JN. Construct validity of script concordance testing: progression of scores from novices to experienced clinicians. Int J Med Ed (2019) 10: 174–9. doi: 10.5116/ijme.5d76.1eee

4. Huhn K, Gilliland SJ, Black LL, et al. Clinical reasoning in physical therapy: a concept analysis. Phys Ther (2019) 99: 440–56. doi: 10.1093/ptj/pzy148

5. American Council of Academic Physical Therapy. Student readiness for the first full-time clinical experience. Available from: https://acapt.org/docs/default-source/public-docs/ksas-and-levels-of-competency-for-considering-student-readiness.pdf?sfvrsn=47da8bd8_2 [cited 1 October 2022].

6. American Board of Physical Therapy Residency and Fellowship Education. Quality standards for clinical residency and fellowship programs. 2022. Available from: https://abptrfe.apta.org/for-programs/clinical-programs/quality-standards-clinical [cited 29 September 2022].

7. O’Brien SR, Durr K, Laubisch E, et al. Every person is an individual: physical therapist clinical reasons used in inpatient rehabilitation for walking assistive device prescription in patients with stroke and brain injury. Disab Rehabil Assist Technol (2021) 16(1): 1–8. doi: 10.1080/17483107.2019.1647568

8. O’Brien SR, Barry M, Davidson E, et al. Physical therapist clinical reasoning in home care for walking assistive device prescription: a description of practice. Physiother Theory Pract (2023) 39(1): 80–8. doi: 10.1080/09593985.2021.1996495

9. Dory V, Gagnon R, Vanpee D, et al. How to construct and implement script concordance tests: insights from a systematic review. Med Educ (2012) 46(6): 552–63. doi: 10.1111/j.1365-2923.2011.04211.x

10. Fournier JP, Demeester A, Charlie B. Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak (2008) 8: 18. doi: 10.1186/1472-6947-8-18

11. Schmidt HG, Rikers RMJP. How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ (2007) 41: 1123–39. doi: 10.1111/j.1365-2923.2007.02915.x

12. Charlin B, Tardif J, Boshuizen HP. Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med (2000) 75(2): 182–90. doi: 10.1097/00001888-200002000-00020

13. Kelly W, Durning S, Denton G. Comparing a script concordance examination to multiple-choice examination on a core internal medicine clerkship. Teach Learn Med (2012) 24(3): 187–93. doi: 10.1080/10401334.2012.692239

14. Piovezan RD, Custodio O, Cendoroglo MS, et al. Assessment of undergraduate clinical reasoning in geriatric medicine: application of a script concordance test. J Am Geriatr Soc (2012) 60(10): 1946–50. doi: 10.1111/j.1532-5415.2012.04152.x

15. Esteves JE, Bennison M, Thomson OP. Script concordance test: insights from the literature and early stages of its implementation in osteopathy. Int J Osteopath Med (2013) 16(4): 231–9. doi: 10.1016/j.ijosm.2013.05.001

16. Lubarsky S, Dory V, Dugan P, et al. Script concordance testing: from theory to practice: AMEE guide no. 75. Med Teach (2013) 35: 184–93. doi: 10.3109/0142159X.2013.760036

17. Schmidt HG, Norman GR, Boshuizen HPA. A cognitive perspective on medical expertise: theory and implications. Acad Med (1990) 65(10): 611–21. doi: 10.1097/00001888-199010000-00001

18. Script concordance tests. University of Montreal School of Medicine. Available from: https://www.cpass.umontreal.ca/https://www.cpass.umontreal.ca/recherche/axes-de-recherches/concordance/sct/ [cited 11 August 2021].

19. Steinberg E, Cowan E, Lin MP, et al. Assessment of emergency medicine residents’ clinical reasoning: validation of a script concordance test. Western J Emer Med (2020) 21(4): 978–84. doi: 10.5811/westjem.2020.3.46035

20. Lineberry M, Kreiter CD, Bordage G. Threats to validity in the use and interpretation of script concordance test scores. Med Educ (2013) 47(12): 1175–83. doi: 10.1111/medu.12283

21. Erickson W, Lee C, von Schrader S. 2018 Disability status report: United States. Ithaca, NY: Cornell University Yang Tan Institute on Employment and Disability (YTI); 2021.

22. Hubbard Winkler S, Cowper Ripley D, Wu S, et al. Demographic and clinical variation in veterans health administration provision of assistive technology devices to veterans poststroke. Arch Phys Med Rehabil (2010) 91: 369–77. doi: 10.1016/j.apmr.2009.10.028

23. Marzolini S, Balitsky A, Jagroop D, et al. Factors affecting attendance at an adapted cardiac rehabilitation exercise program for individuals with mobility deficits poststroke. J Stroke Cerebrovasc Dis (2016) 25: 87–94. doi: 10.1016/j.jstrokecerebrovasdis.2015.08.039

24. Bland AC, Kreiter CD, Gordon JA. The psychometric properties of five scoring methods applied to the script concordance test. Acad Med (2005) 80: 395–9. doi: 10.1097/00001888-200504000-00019

25. Laerd Statistics. Fleiss kappa using SPSS statistics. Statistical tutorials and software guides. Available from: https://statistics.laerd.com/

26. Laerd Statistics. Mann-whitney U test using SPSS statistics. Statistical tutorials and software guides. Available from: https://statistics.laerd.com/

27. Laerd Statistics. Wilcoxon signed rank test using SPSS statistics. Statistical tutorials and software guides. Available from: https://statistics.laerd.com/

28. Lineberry M, Hornos E, Pleguezuelos E, et al. Exerts’ responses in script concordance tests: a response process validity investigation. Med Educ (2019) 53: 710–22. doi: 10.1111/medu.13814

29. Peyrony O, Hutin A, Truchot J, et al. Impact of panelists’ experience on script concordance test scores of medical students. BMC Med Educ (2020) 20(1): 313. doi: 10.1186/s12909-020-02243-w

30. Subra J, Chicoulaa B, Stillmunkès A, et al. Reliability and validity of the script concordance test for postgraduate students of general practice. Eur J Gen Practice (2017) 23(1): 209–14. doi: 10.1080/13814788.2017.1358709

Article Details

Keywords:
MeSH, clinical reasoning, self-help devices, internship and residency, author, ambulation, physical therapy
Section
Original Research Articles
How to Cite
O’Brien, S. R., Dillon, N., Linskey, M., Lagueras, K., Uhl, J., Conroy, S., Kiley, A., Lanzafame, N., States, E., & Sturm, E. (2023). Initial validation of a script concordance test to measure the development of clinical reasoning among physical therapy residents. The Journal of Clinical Education in Physical Therapy, 5. https://doi.org/10.52214/jcept.v5.9014