ORIGINAL RESEARCH ARTICLE

The relationship between physical therapy students’ communication skills during a standardized patient examination and professionalism domain performance on a first full-time clinical experience: an exploratory cohort study

Darryl Young^1*, Marissa Hanlon¹, Carol Recker-Hughes¹, Carol Sames¹, Kevin Neville¹ and Lauren Germain²

¹College of Health Professions, Department of Physical Therapy Education, SUNY Upstate Medical University, Syracuse, NY, USA;
²College of Medicine, Department of Public Health and Preventive Medicine, SUNY Upstate Medical University, Syracuse, NY, USA

Abstract

Purpose: Physical therapy education lacks validated tools for assessing student communication and professional behavior skills before clinical experiences (CE). The Gap Kalamazoo Communication Skills Assessment Form (GKCSAF) has been used in medical education to assess student communication skills. This study investigated: (1) the relationship between scores on a modified GKCSAF (mGKCSAF) during a standardized patient (SP) examination and professionalism domain scores on the Clinical Performance Instrument 2.0 (CPI) during a first full-time CE1 and (2) differences in mGKCSAF ratings across SPs, clinical instructors (CIs), and student self-assessments.

Methods: This retrospective study examined data from 73 first-year physical therapy students who completed a SP examination and were scored using the mGKCSAF before CE1. The mGKCSAF was scored by SPs, CIs, and the students themselves. At the end of CE1, students were assessed using the CPI. Spearman’s rank-order correlations were used to determine the relationship between mGKCSAF scores and five professionalism domain scores on the CPI. A one-way Analysis of Variance (ANOVA) was performed to compare mGKCSAF ratings across rater groups. Post hoc Tukey’s Honest Significant Difference (HSD) tests further explored differences among rater groups. G*Power analysis confirmed sufficient sample size.

Results: Small statistically significant correlations existed between mGKCSAF SP ratings and CI CPI ratings during CE1 in professionalism domain categories. Student mGKCSAF self-assessment scores were significantly lower than assessments by SPs and CIs.

Conclusion: The mGKCSAF may be an effective tool to assess physical therapy student communication skills before CE1 when it is scored by SPs.

Keywords: education; health occupations; social identification; assessment; standardized patients

 

 

Professional behaviors, including communication skills, play an important role in patient outcomes and have been identified as essential competencies required for physical therapy students before a first full-time clinical experience (CE1).^1,2 Patient care may be compromised if a student lacks the essential knowledge, skills, and professional behaviors required for effective patient interaction. Early identification of communication and professional behavior deficits is crucial, as these are often a primary reason for students’ struggles during CE1.³

The journey to developing essential professional behaviors and communication skills is unique to each learner. Some students struggle more along the journey than others for various reasons. Self-assessment and reflection are common teaching tools utilized to promote the development of these skills that can be enhanced by external feedback.^4,5,6 However, perfectionism and impostor syndrome, which are prevalent among Doctor of Physical Therapy (DPT) students,^7,8 can lead to inaccuracies in self-assessment and have been described as potential barriers to professional behaviors and identity formation.^9,10

Simulation-based education (SBE) is a teaching modality that provides students with access to the clinical environment and feedback on their communication skills and professional behavior.¹¹ Feedback from standardized patients (SPs) as part of a simulation experience has been described as a valuable tool for student professional development.^12,13 However, the delivery mechanism of that feedback is variable, with few available valid or reliable outcome measures.¹¹

Student professional behavior performance in the clinic was assessed using the American Physical Therapy Association’s (APTA) Clinical Performance Instrument (CPI) 2.0, prior to being updated to version 3.0. This validated measure of a PT student’s clinical performance assessed five professionalism domains: professional behavior, accountability, communication, cultural competence, and professional development.¹⁴ Three of these domains were included as reportable ‘red flag areas’ (professional behavior, accountability, and communication). Although the CPI is a commonly accepted and standardized assessment practice in the clinic, there are few tools for assessing student communication skills and professional behaviors before students enter the clinical environment.

In medical education literature, The Kalamazoo Consensus Statement established a framework for the development of communication curricula and assessment tools that have been used in undergraduate and graduate medical education.¹⁵ As part of that framework, the Gap-Kalamazoo Communication Skills Assessment Form (GKCSAF) was developed to assess essential communication skills using a 360-degree feedback model to be completed by multiple raters (clinicians, academic faculty [AF], patient/family) and by learners to provide an appraisal of their communication skills during real-time encounters with SPs.¹⁶ The tool has been demonstrated to have high levels of internal consistency (Cronbach’s alpha of 0.844) and good interrater reliability (Intraclass Correlation Coefficient [ICC] of 0.830) in studies of medical students, residents, and fellows.¹⁷

The competencies and exemplar sub-competencies provided in the GKCSAF incorporate communication and interpersonal skills recommended for PT students entering a CE1.² These skills are also identified in professional documents that guide DPT program curricula.^18,19,20 Communication is a key aspect of professional behavior and has been identified as a significant component of developing attributes associated with professionalism.²¹ The primary purpose of this retrospective study was to explore relationships between physical therapy student scores on a modified version of the GKCSAF (mGKCSAF) during a SP examination and CPI professionalism domain scores during CE1. A secondary aim was to explore whether there were statistically significant differences between student, SP, and Clinical Instructor (CI) ratings of communication and interpersonal skills using the mGKCSAF during a SP encounter.

Methods

Participants

This retrospective exploratory study used data collected from 73 first-year DPT students enrolled in a Physical Therapy Education Program (PTEP) over two consecutive years. These students were 65% female. SPs were paid and trained male actors from the community recruited by the institution’s SP program. The mGKCSAF was scored during a SP examination in the Spring of 2016 and 2017 and analyzed during the Spring of 2020. The CPI was scored in the Fall of 2016 and 2017 and analyzed following CE1 (an 8-week experience in either the outpatient, rehabilitation, or acute care setting) in the Spring of 2020. The study was deemed exempt from the institution’s IRB review.

The Integrated Standardized Patient Examination

The ISPE is a comprehensive simulated examination modeled after an assessment described by Panzarella and Manyon that focuses on the integration of knowledge, skills, and behaviors across the first-year curriculum of the DPT program.²² Students participated in the ISPE after three semesters of academic coursework and before CE1. The authentic patient encounter simulated a non-complex, musculoskeletal scenario typically encountered in an outpatient setting. Students were self-assessed and rated by AF, SPs, and CIs for a pass/fail grade.

The ISPE provides a formal assessment of student readiness through observing the student-SP encounter, which includes gathering history information, performing a limited physical examination (systems review, tests, measures), and providing patient education. During the 45-min encounter, observers viewed the simulation simultaneously using audiovisual technology. The AF observer completed a detailed Knowledge, Skills, and Abilities (KSA) rubric that conveys a student’s ability to perform foundational clinical skills, while the CI observer completed the mGKCSAF to assess critical communication skills throughout the examination.¹⁷

Scores from both the KSA and mGKCSAF rubrics informed faculty decisions regarding student readiness for CE1. Students who did not demonstrate adequate clinical skills were remediated prior to starting CE1. If students struggled with knowledge and psychomotor skills, they were asked to participate in small group remediation sessions and/or meet individually with their AF grader. If students struggled with communication skills, they reviewed their video performance, met with their AF grader, and were given the opportunity to apply their feedback in another SP encounter. Every student in this data set was ultimately successful in passing the ISPE (with/without remediation), and all successfully passed CE1 by achieving at least Advanced Beginner level scores in all domains on the CPI.

Modified Gap-Kalamazoo Communication Skills Assessment Form

The mGKCSAF was modified with author permission for use in this simulation. The GKCSAF includes language regarding communication with both the patient and family. The ISPE simulated a patient interaction without a family member present. To preserve face validity,²³ ‘and family’, language included in the GKCSAF was left out. The full mGKCSAF and permissions are described in Figure 1. A 5-point Likert Scale (poor to excellent) was used, where higher scores indicated stronger communication skills with a possible range of 9–45.

Fig. 1. Modified Gap-Kalamazoo Communication Skills Assessment Form. Adapted from: Essential Elements: The Communication Checklist, 2001 Kalamazoo Consensus Statement Group, and from the Gap Kalamazoo Communication Skills Assessment Form, both published in: Rider EA. Interpersonal Communication Skills: In: Rider EA, Nawotniak RH. A Practical Guide to Teaching and Assessing the ACGME Core Competencies, 2nd ed. Marblehead, MA: HCPRo, Inc., 2010. Used with permission.

This form was introduced to SPs, students, and CIs prior to the ISPE, and all groups were trained in its use. SPs participated in mGKCSAF training workshops as part of the University’s SP Program and had previous experience utilizing the tool in simulations across health profession programs within the University. PT students were introduced to the mGKCSAF during case-based role-playing exercises while gathering history information in a first-year, third-semester course. The role-playing scenarios promoted students’ familiarity with the tool and expectations for communication and interpersonal skills in a clinical environment. The student and SP completed the mGKCSAF immediately following the SP-student interaction. The mGKCSAF was introduced to the CIs in a workshop before the ISPE, where they watched recorded student–patient encounters and used the tool as a guide to provide feedback to the students.²⁴ The mGKCSAF was completed by CIs in real time while observing the student–SP interaction. Time constraints prevented AF from completing the mGKCSAF in addition to the detailed KSA rubric.

First full-time clinical experience final assessments – CPI

At the end of CE1, an 8-week experience, CIs completed a final assessment of DPT students using the CPI 2.0, an 18-domain checklist scored using a 6-point Likert scale with options ranging from ‘Beginning Performance’ to ‘Beyond Entry-Level Performance’. The CPI has been validated by Roach et al. and was used by PTEPs in the United States prior to the development of the revised CPI 3.0 in 2023.¹⁴ The CIs who completed the CPI on CE1 were not necessarily the same CIs who rated students using the mGKCSAF during the ISPE.

Data analysis

Statistical analyses were performed using IBM SPSS 27. Spearman’s rank-order correlations were used to determine whether mGKCSAF scores were correlated with clinical performance in CE1 as indicated by CPI professionalism domain scores. To account for multiple comparisons across the CPI domains, P-values were adjusted using the Benjamini-Hochberg (BH) procedure to control the false discovery rate.²⁵ Post hoc G*Power 3.1 analyses confirmed that the sample size (N = 73) was appropriate for the statistical tests being employed.²⁶ A one-way Analysis of Variance (ANOVA) was conducted to determine whether there were statistically significant differences in assessment scores assigned by three different rater groups: CIs, SPs, and the students themselves. Each participant earned scores from all three raters, yielding a paired design with 73 scores in each rater group. Assumptions of ANOVA were tested, including the homogeneity of variances using Levene’s test. The results indicated no significant violation of homogeneity (P = 0.46), supporting the use of ANOVA for this analysis. Post hoc comparisons were conducted using Tukey’s Honest Significant Difference (HSD) test to further explore pairwise differences among rater scores.

Results

The range, mean, and standard deviation of mGKCSAF ratings entered by the three rater types (Self, SP, and Cl) are shown in Table 1.

Table 1. Correlations between mGKCSAF ratings and CE1 final PT-CPI domain scores

Domain on CPI Form	Self/Clinic 1		CI/Clinic 1		SP/Clinic 1
	rs	P	rs	P	rs	P
CPI Professional Domain
Professional vehavior	0.02	0.86	0.08	0.50	0.27	0.02*
Accountability	0.04	0.76	0.13	0.26	0.26	0.03*
Communication	0.05	0.69	0.12	0.31	0.28	0.02*
Cultural competence	0.02	0.87	0.11	0.36	0.30	0.01*
Professional development	0.07	0.57	0.11	0.26	0.27	0.02*
*Correlation is significant at the 0.05 level (2-tailed). mGKCSAF, Modified Gap Kalamazoo Communication Skills Assessment Form; CI, clinical instructors.

GKCSAF by CPI domain

Spearman’s rank-order correlations revealed no statistically significant associations between Self or CI ratings of communication skills using the mGKCSAF and final CPI professionalism domain scores earned during CE1 (BH-adjusted P > 0.05; Table 1). However, SP ratings of communication skills on the mGKCSAF were positively associated with CPI scores earned during CE1 in all professionalism domains (BH-adjusted P < 0.05; Table 2).

Table 2. Descriptive statistics of mGKCSAF ratings by rater type

Rater type	N	Min.	Max.	Mean	SD
Clinical Instructor (CI)	73	10.50	45.00	35.21	6.35
Standardized Patient (SP)	73	16.00	45.00	36.33	5.55
Self	73	14.50	42.00	31.47	5.89
mGKCSAF, Modified Gap Kalamazoo Communication Skills Assessment Form.

GKCSAF student, SP, and CI ratings

The one-way ANOVA revealed a statistically significant difference in scores between rater groups, F (2, 216) = 13.42, P < 0.001 (Table 3). Tukey’s post hoc analysis indicated that while the scores entered by the CI and SP did not significantly differ (P = 0.49), the scores assigned by the student were significantly different from those given by either the CI (P < 0.001) or the SP (P < 0.001).

Table 3. Results of Analysis of variance analysis and post hoc testing

ANOVA analysis	Sum of squares	df	Mean square	F	Sig.
Between Groups	946.475	2	473.237	13.417	< 0.001*
Within Groups	7618.692	216	35.272
Total	8565.167	218
Tukey HSD analyses					95% Confidence interval
Assessor	Assessor	Mean difference	Standard error	Sig.	Lower bound	Upper bound
CI	SP	-1.12	0.98	0.49	-3.44	1.20
CI	Student	3.74	0.98	< 0.001*	1.42	6.10
SP	Student	4.86	0.98	< 0.001*	2.54	7.20
*< 0.05 ANOVA, Analysis of Variance; CI, clinical instructors; SP, Standardized Patient; HSD, Honest Significant Difference.

Discussion

This study sought to explore the usefulness of the mGKCSAF in providing feedback on student communication and interpersonal skills before CE1. The associations observed in this study between SP mGKCSAF scores and CPI professionalism domain scores suggest that early communication skills ratings from SPs may provide insights into later clinical performance, as assessed by CIs in the professionalism domains of the CPI. Similar findings have been published with medical interns for a simulation assessment known as an Objective Structured Clinical Exam (OSCE). Goldhamer et al. reported a relationship between GKCSAF scores on an OSCE and communication ratings on clinical performance evaluations. In that study, lower GKCSAF scores on an OSCE were associated with greater faculty communication concerns in the clinic, and non-faculty raters (other healthcare professionals and SPs) were more likely to identify communication deficits in the OSCE than faculty raters.²⁷

These findings are consistent with literature highlighting the value of SP feedback in the development of DPT student professional behaviors.^12,13 These studies have discussed the unique contributions that SP feedback offers students, such as a deeper understanding of professional behaviors in preparation for the clinic, student self-efficacy of professional behaviors, and the development of compassion and humility.^12,13 This external feedback is particularly helpful as students tend to be overly critical when self-assessing their communication skills, as observed in the ISPE.

Student mGKCSAF self-assessment scores were significantly lower than both the CI and SP mGKCSAF. These results are consistent with studies indicating a high prevalence of perfectionism and impostor syndrome among DPT students.^7,8 The impostor phenomenon impacts how ‘impostors’ assess themselves.²⁸ Impostors tend to be particularly critical of their performance and overestimate the number of mistakes they make when self-assessing.²⁸ As the prevalence of impostor syndrome has been reported to be as high as 74% of students enrolled in PTEPs in the United States,⁷ many of the students in this study likely experienced the phenomenon. This could account for the lower mGKCSAF scores observed in the student self-assessment when compared to the scores from the CI and the SP.

Feedback from the SPs using the mGKCSAF, in conjunction with activities associated with the ISPE (role plays in didactic course, review of recorded performance, individual remediation activities), may be useful to promote accurate student appraisal of strengths and gaps in their communication and interpersonal skills. Students who acknowledge areas in need of development prior to entering CE1 may be more likely to accept CI feedback and to implement plans to attain expected levels of professional performance. Integrating the mGKCSAF into first year simulation activities may support the development of essential professionalism competencies necessary to successfully complete early CEs.

Limitations and future studies

This study included quantitative data collected from a single DPT program and utilized a modified version of the GKCSAF. The impact of the modifications on the psychometric properties of the tool is unknown; however, the modifications, as described in the methods, were minor without altering the underlying construct. Final CPI scores were used as outcome variables and, in addition to preparation, likely reflect the impact of training and development while in a first full-time CE. In a system as complex as physical therapy education, direct attribution is difficult to establish.²⁹ When scored at the end of CE1, CPI scores would reflect didactic training as well as learning that occurred during CE1.

Occupational therapy has explored the reliability of the GKCSAF in their profession, but the reliability and validity of the tool among physical therapy students have not been published.³⁰ Future studies are needed to determine the validity and reliability of the tool in SBE for DPT students and to investigate whether similar correlations are seen with CPI 3.0. Qualitative studies examining the comments at the end of the mGKCSAF would provide additional insights into the correlation between SP mGKCSAF scores and CPI scores. In addition, further investigation of how perfectionism and impostor syndrome impact student self-assessment and readiness for clinical practice should be explored.

Conclusion

The mGKCSAF, integrated into the ISPE, offers a valuable means to assess student proficiency in essential communication skills, highlighting both strengths and areas for improvement in preparation for CEs. Early identification allows for targeted remediation prior to a student’s first clinical placement, enhancing their success in the clinic. SP ratings of student performance using the mGKCSAF are associated with CPI professionalism domain scores on CE1. Notably, students often underrate their communication skills compared to SP and CI assessments. The mGKCSAF warrants further exploration as a standardized tool to provide feedback on physical therapy students’ communication skills before their initial CE.

Ethical approval

The protocol Correlations between pre-clinical assessments of communication skills and clinical workplace based assessments, 1760683-1, has been verified by the SUNY Upstate IRB as Exempt according to category 4ii.

References

1.	Søndenå P, Dalusio-King G, Hebron C. Conceptualisation of the therapeutic alliance in physiotherapy: is it adequate? Musculoskelet Sci Pract (2020) 46: 102131. doi: 10.1016/j.msksp.2020.102131
2.	Timmerberg JF, Dole R, Silberman N, et al. Physical therapist student readiness for entrance into the first full-time clinical experience: a Delphi study. Phys Ther (2019) 99(2): 131–46. doi: 10.1093/ptj/pzy134
3.	Silberman N, LaFay V, Hansen RL, et al. Physical therapist student difficulty in clinical education settings: incidence and outcomes. J Phys Ther Educ (2018) 32(2): 175–82. doi: 10.1097/JTE.0000000000000046
4.	May WW, Morgan BJ, Lemke JC, et al. Model for ability-based assessment in physical therapy. J Phys Ther Educ (1995) 9(1): 3–6. doi: 10.1097/00001416-199501000-00002
5.	Jette DU, Portney LG. Construct validation of a model for professional behavior in physical therapist students. Phys Ther (2003) 83(5): 432–43. doi: 10.1093/ptj/83.5.432
6.	Hayward LM, Blackmer B. A model for teaching and assessing core values development in doctor of physical therapy students. J Phys Ther Educ (2010) 24(3): 16–26. doi: 10.1097/00001416-201007000-00003
7.	Young A, Handlery K, Kahl D, et al. A survey of the prevalence of impostor phenomenon among US entry-level Doctor of Physical Therapy students. J Phys Ther Educ (2024) 38(1): 19–24. doi: 10.1097/JTE.0000000000000313
8.	Richardson MV, Miller H, Papa E, et al. Perfectionism, stress, and the entry-level Doctor of Physical Therapy student: a cross-sectional, observational study. J Phys Ther Educ (2022) 36(1): 9–16. doi: 10.1097/JTE.0000000000000213
9.	Stelling BE, Andersen CA, Suarez DA, et al. Fitting in while standing out: professional identity formation, imposter syndrome, and burnout in early-career faculty physicians. Acad Med (2023) 98(4): 514–20. doi: 10.1097/ACM.0000000000005049
10.	Thomas M, Bigatti S. Perfectionism, impostor phenomenon, and mental health in medicine: a literature review. Int J Med Educ (2020) 11: 201. doi: 10.5116/ijme.5f54.c8f8
11.	Stockert B, Silberman N, Rucker J, et al. Simulation-based education in physical therapist professional education: a scoping review. Phys Ther (2022) 102(12): 133. doi: 10.1093/ptj/pzac133
12.	Donaldson MB, Tyler K, Carroll A. The effect of standardized patients for physical therapy students on behaving and communicating as a professional: a systematic review. Phys Ther Rev (2022) 27(6): 464–76. doi: 10.1080/10833196.2022.2141039
13.	Riopel MA, Litwin B, Silberman N. Promoting professional behaviours in physical therapy students using standardized patient feedback. Physiother Can (2019) 71(2): 160–7. doi: 10.3138/ptc.2018-0
14.	Roach KE, Frost JS, Francis NJ, et al. Validation of the revised physical therapist clinical performance instrument (PT CPI): version 2006. Phys Ther (2012) 92(3): 416–28. doi: 10.2522/ptj.20110129
15.	Makoul G. Essential elements of communication in medical encounters: the Kalamazoo consensus statement. Acad Med (2001) 76(4): 390–3. doi: 10.1097/00001888-200104000-00021
16.	Rider EA, Nawotniak RH. Interpersonal and communication skills. In: Rider EA, Nawotniak RH, eds. A practical guide to teaching and assessing the ACGME core competencies. 2nd ed. Chicago, IL: HCPRo, Inc; 2010, pp. 1–37.
17.	Peterson EB, Calhoun AW, Rider EA. The reliability of a modified Kalamazoo Consensus Statement Checklist for assessing the communication skills of multidisciplinary clinicians in the simulated environment. Patient Educ Couns (2014) 96(3): 411–8. doi: 10.1016/j.pec.2014.07.013
18.	American Physical Therapy Association. A normative model of physical therapist professional education: version 2004. Alexandria, VA: American Physical Therapy Association; 2004.
19.	American Physical Therapy Association. Code of ethics for the physical therapist (HOD S06-20-28-25). Available from: https://www.apta.org/siteassets/pdfs/policies/codeofethicshods06-20-28-25.pdf [cited 19 June 2024].
20.	American Physical Therapy Association. Core values for the physical therapist and physical therapist assistant (HOD P09-21-21-09). Available from: https://www.apta.org/apta-and-you/leadership-and-governance/policies/core-values-for-the-physical-therapist-and-physical-therapist-assistant [cited 19 June 2024].
21.	Brown T, Yu ML, Etherington J. Are listening and interpersonal communication skills predictive of professionalism in undergraduate occupational therapy students? Health Prof Educ (2020) 6(2): 187–200. doi: 10.1016/j.hpe.2020.01.001
22.	Panzarella K, Manyon A. Using the integrated standardized patient examination to assess clinical competence in physical therapist students. J Phys Ther Educ (2009) 22(3): 24–32. doi: 10.1097/00001416-200810000-00004
23.	American Psychological Association. APA dictionary of psychology. n.d. Available from: https://dictionary.apa.org/face-validity [cited 26 November 2024].
24.	Recker-Hughes C, Dungey J, Miller S, et al. A novel approach to clinical instructor professional development: a multi-session workshop with application of skills in a student standardized patient exam. J Phys Ther Educ (2015) 29(1): 49–59. doi: 10.1097/00001416-201529010-00009
25.	Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Series B Stat Methodol (1995) 57(1): 289–300. doi: 10.1111/j.2517-6161.1995.tb02031.x
26.	Faul F, Erdfelder E, Buchner A, et al. Statistical power analyses using G*Power 3.1: tests for correlation and regression analyses. Behav Res Methods (2009) 41(4): 1149–60. doi: 10.3758/BRM.41.4.1149
27.	Goldhamer ME, Cohen A, Brooks M. Use of an objective structured clinical exam (OSCE) for early identification of communication skills deficits in interns. Med Teach (2018) 40(1): 40–4. doi: 10.1080/0142159X.2017.1387646
28.	Sakulku J. The impostor phenomenon. Int J Behav Sci (2011) 6(1): 75–97.
29.	Mayne J. Addressing attribution through contribution analysis: using performance measures sensibly. Can J Program Eval (2001) 16(1): 1–24. doi: 10.3138/cjpe.016.001
30.	Fan SC, Tsai ST, Wang YC, et al. Reliability of the Gap–Kalamazoo communication skills assessment form in occupational therapy. BJOT (2024) 87(7): 424–33. https://doi.org/10.1177/03080226241239574