Secondary Logo

Data-driven Decision Making in Doctor of Physical Therapy Curricula Part I

Program-level Analysis

Roller, Dawn, DPT; Wininger, Michael, PhD; Leard, John, EdD, PT, ATC; Crane, Barbara, PhD, PT, ATP/SMS

Journal of Physical Therapy Education: December 2018 - Volume 32 - Issue 4 - p 368–375
doi: 10.1097/JTE.0000000000000064
Method/Model Presentation
Free
SDC

Background and Purpose. We describe a data-driven internal curricular review process for professional (entry-level) physical therapist education programs. Using National Physical Therapy Exam (NPTE) performance as an outcome measure, we test the ability of individual courses, as well as summary grade point averages (GPAs), to identify students at risk for failure in the first attempt of the NPTE.

Methods. National Physical Therapy Exam score was modeled in a simple univariate linear regression on 2 sets of predictor variables: individual course final grades and summary GPAs. First-time NPTE performance was configured in 2 measures: a continuous measure of exam score and a dichotomous pass–fail index. For NPTE score as a continuous measure, linear regression was used to test for association. For pass–fail, we tested for optimal grade or GPA benchmark by systematic thresholding across a range of numerical grades (70–95%) or grade points (1.5–4.0). Methods were demonstrated in the setting of a dataset collected from a single graduate cohort (N = 36), among whom 26 first-time NPTE scores were available. Exploratory analyses tested the sensitivity of this analysis to datasets containing letter grades, and repeatability across 5 cohorts.

Outcomes. Grade distributions in individual courses (n = 12) were weak to moderate predictors of NPTE score (range: 0.00 < R 2 < 0.60); curricular GPA distributions (n = 6) were moderate to fair predictors (0.4 < R 2 < 0.7). Overall undergraduate GPA and GPA in prerequisite science courses were found to not predict first-time NPTE score whatsoever. Optimal grade criterion within the classes ranged from 79 to 93 points; optimal GPA criteria ranged from 2.6 to 3.4. At-risk students were identifiable as early as the end of the first semester. Model consistency varied somewhat when replicated across 5 cohorts but showed general consistency in maintaining relative strengths of measured associations. This method showed robustness to conversion to letter-grade format.

Discussion and Conclusion. Risk for failure in first-attempt NPTE is a parameter that can be measured, and at-risk students can be identified early in their physical therapist education programs. Internal policy making should be guided by program experience and use data at the per-course or per-semester level.

Dawn Roller is an assistant professor in the Department of Rehabilitation Sciences at the University of Hartford.

Michael Wininger is an assistant clinical professor in the Department of Biostatistics at the Yale University, is an assistant professor in the Department of Rehabilitation Sciences at the University of Hartford, and is a statistician of medicine, Cooperative Studies Program in the Department of Veterans Affairs.

John Leard is an assistant professor in the Department of Rehabilitation Sciences at the University of Hartford.

Barbara Crane is a professor in the Department of Rehabilitation Sciences at the University of Hartford, 200 Bloomfield Avenue, West Hartford, CT 06117 (bcrane@hartford.edu). Please address all correspondence to Barbara Crane.

The authors declare no conflicts of interest.

This study was reviewed and approved by the University's Institutional Review Board.

Received October 15, 2017

Accepted June 04, 2018

Back to Top | Article Outline

BACKGROUND AND PURPOSE

The principal goal of formal physical therapist education is to ensure that graduates are capable of competent, ethical, autonomous, and contemporary physical therapy practice.1–3 Nationally, the standard to practice as a Physical Therapist is to achieve the passing score on the national licensure examination. Thus, it should be the goal of any professional (entry-level) Doctor of Physical Therapy (DPT) Program to develop a curriculum that supports passing the National Physical Therapy Exam (NPTE) and to develop mechanisms for identifying students at risk for exam failure. Additionally, the Commission on Accreditation in Physical Therapy Education requires that all programs maintain at least an 80% average pass rate on the NPTE over a 3-year period and require that programs collect outcome data that justify curriculum support this standard.3 But program review is a complex and multifaceted process; there are no specific guidelines, and there is a need for more published studies documenting an effective method for internal review.

There are various ways to test for relationships between program performance and NPTE outcome: standard correlation or linear regression techniques allow for probing the relationship between 2 numerical values (say, grade point average [GPA] and NPTE score)2,4–7; logistic regression supports testing for dichotomous outcomes (ie, pass-fail).8,9 While we are aware of one study where both linear and logistic regression techniques were employed, the results are inconsistent between the 2 approaches, creating need for additional replication.10 Grade point average, reported in various forms (eg, mean undergraduate, core science course, first-year professional), has been found to be one of the most consistent variables associated with success on the NPTE.1–7,10

Beyond methodology, there is the open question of when a risk assessment is supported. Kosmahl2 found a positive relationship between first-year Professional GPA and NPTE success; however, they suggest seeking additional information on other factors that contribute to predicting success on this exam. Because foundational courses typically occur in the first year of study, the review of individual course data may provide additional insight to early detection of NPTE success.5 However, there is a natural expectation that the early coursework, being so distant from the NPTE, may be less correlated.11

The purpose of this study was to assess the predictive relationship between program-specific performance markers and first-time NPTE performance in both the continuous and the dichotomous view, with a particular interest in assessing whether associations emerge early in the program. Our two approaches were: 1) GPA or course grade, and NPTE score, all as continuous variables and 2) GPA or course grade, thresholded systematically, and NPTE thresholded at pass–fail (600 points). We ask: what are the most robust predictors of NPTE failure risk, and how early in a student's education can risk be estimated accurately. The goals of this study are to 1) facilitate a data-driven approach to program review, 2) demonstrate the analysis and disseminate findings from our own internal review on data collected from a representative sample of students, and 3) identify key opportunities and limitations to physical therapist education curriculum review in facilitation of replication by others. We seek to add to the literature base, contributing to the broader conversation about how to approach rigorous program review and facilitating early identification of at-risk students.

Back to Top | Article Outline

METHODS

Data Collection

This study presents an analysis of academic performance data from a single professional (entry-level) program cohort, admitted to a DPT program. The program is a 3-year, full-time curriculum with a “summer-start”; thus, the first year is organized into a summer, fall, and spring semesters. The cohort studied started in June 2012 and graduated in May 2015.

All data were stripped of personally identifiable information; all activities performed in this study were conducted after review by the University's Institutional Review Board. The study was approved as exempt and did not require informed consent of the students as the data analyzed were not identifiable.

Final-grade data from 12 curriculum offerings (24 credits) were compiled (Table 1). These represent a subset of the 57 courses (111 credits) available to students in the program. We purposely limited our scope to those courses, which we anticipated would have broad topical overlap with the NPTE and served as foundational courses in the curriculum. Highly specialized courses and clinical rotations were intentionally discarded from the analysis set.

Table 1

Table 1

Back to Top | Article Outline

Grade Point Data

Grade point averages were calculated in eight ways: 1) total cumulate GPA at the time of program entry (“general GPA”), 2) GPA in the foundational science courses taken in undergraduate curriculum (“science GPA”), 3–5) GPA in summer, fall, and spring semester of year I (GPA-I, GPA-II, and GPA-III), 6 and 7) cumulative GPA after fall and spring semester of year I (GPA-I/II, GPA-I/II/III), and 8) final GPA at the time of graduation from the DPT program. Grade point averages at the time of program entry (general GPA and science GPA) were extracted from Departmental records. We note that while data points related to academic performance before admission do not reflect on the DPT curriculum, these parameters nevertheless have potential as markers for NPTE performance.7 Curricular GPAs were calculated using only those courses included in this analysis, and other courses that may have been taken concurrently were not included in these GPA calculations. In this way, these GPAs do not necessarily reflect the GPA as might appear on a student's transcript but reflect the averaged grade-point of course final grade weighted by credit count. Whereas course grades were recorded on a 0–100 scale, they were converted to GPA via conventional transformation: division by 10 and subtraction of 5.5, that is, 3.0 equates to 85%.

Back to Top | Article Outline

National Physical Therapy Exam Score Prediction

We measured the association between academic performance variables and first-time exam score using a univariate linear model, regressing NPTE score as a continuous variable against either GPA or final course grade, both also as a continuous variable. Both the significance of the predictor (P-value) and the coefficient of determination (R 2) were retained, as well as the slope and intercept of the regressor. Students whose exam score or final grades were missing from the dataset were removed on a course-by-course basis. For maximally generalizable analysis, we purposely did not filter for outliers. In the interest of maximal interpretability, we did not consider higher order (nonlinear) regressions, or multivariate regressions. In total, 20 separate univariate regressions were performed (8 GPA predictors and 12 course grade predictors).

Back to Top | Article Outline

National Physical Therapy Exam Outcome Prediction

In converting NPTE score to a dichotomous response variable (pass–fail), we considered employing a logistic regression,8,10 but believed this to add only minimal value beyond linear regression. Rather, we identified an opportunity to systematically test for the optimal parameter value by converting the predictor (course grade or GPA) also into a dichotomous variable (above or below a threshold). Because sample size was small and there was a relatively infrequent occurrence of course failure and/or NPTE failure, we used Fisher's Exact test to determine relative risk (RR) of first failure on the NPTE.4 For predictor variable (course grade or GPA), a range of thresholds were probed for suitability for predicting NPTE “pass/fail”. The contingency table was set up in a 2 × 2 format (supra-threshold vs sub-threshold; NPTE pass vs NPTE fail). For each course, cutoff grades of 70, 71, 72, …, 93, 94, 95 were tested as the course pass–fail criterion; for each GPA, cutoff GPA values of 1.5, 1.6, 1.7, …, 3.8, 3.9, 4.0 were tested as the GPA criterion. The P-value for the Fisher's exact test was extracted for all 26 thresholds. The threshold grade yielding the single lowest P-value across this set was retained, as were the parameters from the exact test. Students whose NPTE score was not known were discarded from the analysis. Our aim in this analysis was to measure both the P-value and the relative risk of course failure as a marker for NPTE failure, defined as follows:

where outcomes are phrased in Course-Exam, that is, supraFail = number of students completing the class above the specified threshold grade and also failing the NPTE on first attempt. Two exemplar contingency tables from DPT500 are shown in Table 2.

Table 2

Table 2

Back to Top | Article Outline

Secondary Analyses

Although a full dataset of grades and GPA values was not available for multiple years, there were 5 variables for which 5 full cohorts' data were available: course grades for 2 classes, as well as undergraduate science GPA, undergraduate overall GPA, and the final GPA at time of graduation. To assess robustness of the proposed modeling paradigm, regression analysis was replicated for each cohort (25 total regressions). Additional cohorts tested were from students who graduated in 2010, 2011, 2013, and 2016; data were incomplete in years 2012 and 2014.

The regression was performed on the raw grading data, that is, numerical score on a 100-point scale. However, many universities, including our own, record grades in a letter system. Because the letter system loses resolution by grouping students together, we were concerned that the regression would lose accuracy in the setting of letter grades, compared with use of numeric grades. To test the sensitivity of the regression to the change in grading scheme, we remapped continuous numerical course grade data in the 2015 cohort according to a template employed throughout the University and within the Department (Table 3).

Table 3

Table 3

For each course in the 2015 cohort dataset, course grades were remapped and regressed linearly for their goodness of fit in association to the NPTE score. Again, R 2 was taken as the single representative parameter of interest.

Back to Top | Article Outline

OUTCOMES

Descriptive Statistics

Among the 12 classes, the average grade was 86.6 ± 3.0 points (min: 82.3 points; max: 91.7 points); the average range in grade distribution was 20.2 ± 6.7 points (min: 13.1 points; max: 36.3 points). The average low grade was 77.3 ± 4.2 points (min: 71.9 points; max: 83.8 points), the average high grade was 97.6 ± 4.0 points (min: 94.2 points; max: 109.2 points).

Among the GPA upon program entry: general GPA was 3.4 ± 0.2 and science GPA was 3.3 ± 0.2. Among the curriculum GPAs: GPA-I (10 credits) was 3.0 ± 0.5, GPA-II (7 credits) was 3.5 ± 0.3, and GPA-III (7 credits) was 2.9 ± 0.4; among the running-average GPAs: GPA-I/II was 3.2 ± 0.4 and GPA-I/II/III was 3.1 ± 0.4.

Among the 36 students in the cohort, 26 authorized release of NPTE data to the Department. Of these 26 students, 17 (65%) passed the exam on first attempt; average first-attempt score was 641.3 ± 61.3 (range: 546–771). Of those who did not pass the exam in first attempt, the average margin of failure was 22.3 ± 13.2 points (range: 13–54 points).

Back to Top | Article Outline

National Physical Therapy Exam Score Prediction

Course final grades yielded generally low-accuracy predictions of the NPTE first-time score. Although the P-values were low, the coefficients of determination were moderate at best—minimum: P = 3.2 × 10−6, maximum: P = .90, with 11 (91.7%) courses below P < .05, 10 (83.3%) courses below P < .01, and 6 courses (50%) below P < .001. The R 2 values ranged from 0.00 to 0.60, with 5 courses yielding R 2 > 0.4 (Table 4). Exemplar regressions are shown in Figure 1.

Table 4

Table 4

We summarize our linear regressions in Table 4. Slopes among courses with significant associates to the NPTE performance varied from 5 NPTE points per course grade point to 10 NPTE points per grade point, meaning that an increase in course grade by 1 percentage point predicts an increase in NPTE score by between 5 and 11 points. We note that negative intercepts are not unexpected, particularly for courses with steep slopes.

National Physical Therapy Exam score prediction was much stronger for curricular GPAs: P < .001 in all 6 graduate GPA measures, with 0.45 ≤ R 2 ≤ 0.68. Grade point average upon entry was not a significant predictor of NPTE score and yielded P values not less than .05. We note that a significant P-value is not particularly informative about a course's predictive utility; R 2 is the much more robust measure.12

Back to Top | Article Outline

National Physical Therapy Exam Outcome Prediction

For each of 12 courses, 26 different grade thresholds were measured for predictive value in identifying at-risk students for first-time failure on the NPTE. One example is shown in Table 5.

Table 5

Table 5

We observe that for many grade thresholds, including that associated with the lowest P-value, the confidence interval contains infinity. This is an expected outcome for contingency tables containing cells with no elements as division by zero is a mathematically undefined operation, which in many computing environments is short-handed as infinity. In particular, it is ideal to have to have zero students who are represented in the off-diagonal elements. Off-diagonal elements are defined as the students who are above the course grade threshold and fail the exam (SupraFail) or students who fall below the course grade threshold and pass the exam (SubPass); refer to sample contingency tables (above) for representative examples.

Across the 12 courses, we find that the optimum grade threshold varies from 79 to 93 points, and that the predictive sensitivity of detecting NPTE failures ranges from low (2 of 8) to high (all 8). For 2 courses (DPT500 and DPT504), the P-value was significant at P < .05, but the RR is reported as “infinity.” This is an expected outcome that would result from either of the following quantities yielding zero: either SubFail + SubPass (ie, there were no students below the threshold course grade) or SupraFail (ie, there were no students passing the course and subsequently failing the exam). In both cases here, the latter explains the undefined RR (Table 6).

Table 6

Table 6

P-values ranged from P = 3.4 × 10−5 to P = .22, with calculable RR ranging from 0.4 to 16. Among the 7 GPAs tested for utility in predicting NPTE outcome, all 5 curriculum GPAs yielded significant P-values and with RR ranging from 4.7 (GPA-II) to 9.3 (GPA-I). Neither Program-entry GPA yielded P < .05. We note that the optimal GPA cutoffs varied widely, whereas optimal thresholds varied from 2.6 (GPA-III) to 3.4 (GPA-II). However, this agrees with our findings related to the threshold course grades presented in Table 6.

Back to Top | Article Outline

Secondary Analyses

When replicating the linear regression across 5 variables for which there were multiple cohorts' data, we found moderate variability, but could not identify systematic trends. In Table 7, we report raw R 2 values from these regressions.

Table 7

Table 7

These data are shown in Figure 2 for ease of visualization.

Although some variables (science GPA and final GPA) varied within narrow ranges, others (course grades in DPT608 and DPT609 and overall GPA upon admission) were widely scattered. Uniformly, the science GPA was a poor predictor of NPTE performance across all 5 cohorts.

Separately, we found very high agreement between goodness of fit as extracted from linear models fitted to either raw numeric grade data or on course performances converted to letter grades; these data are summarized in Table 8.

Table 8

Table 8

These data correlated with ρ > .9, indicating that the approach described here works equally well for either grading system (Figure 3).

Back to Top | Article Outline

DISCUSSION AND CONCLUSION

Main Findings

The primary aim of this study was to test whether performance in any single course or any single milestone GPA showed utility in predicting performance on the NPTE, either in first-time score or in first-time pass/fail outcome. Our primary findings were that many predictor variables showed moderately good prediction of NPTE score for many courses (Table 4). Our findings in the prediction of pass/fail were more varied. Some predictors showed negligible association to exam outcome while others showed very promising association (Table 6). In the case of outcome prediction, the optimal grade threshold varied substantially by course. Our findings are fairly robust across multiple years: predictors that are strong, weak, or moderate tend to be consistently strong, weak, or moderate, with some fluctuation (Table 7). Finally, we find that these approaches appear to work well for letter grades; therefore, access to raw numerical grade is not strictly necessary.

Back to Top | Article Outline

Comparison

There is a challenge in comparing the performance of our approach versus those of others, given that there are only a handful of studies assessing the relationship between NPTE performance and academic performance. Methods vary, and there are challenges in assessing comparability of samples across studies. In a previous study of linear correlations between academic predictors and NPTE score, a substantial relationship has been reported: 47% of explained variance in NPTE score attributable to GPA and Clinical Performance Instruments.2 Elsewhere, only modest associations between NPTE score and core course GPA (R 2 = 0.42) and first-year GPA (R 2 = 0.12) were found, and the usefulness of GPA as a predictor appeared to decrease after year 1.6 Here, we found explained variance of 37.4 ± 17.6% (range: R 2 = 0.00 – 0.60) in single courses and 60.3 ± 8.8% (range: 0.45–0.68) in various GPA benchmarks. However, we found that preadmission variables were very weak predictors of NPTE success. This mirrors the findings of others,7 although some studies have found useful predictors through Graduate Record Examination scores and behavioral interviews.13

We observe that another study has shown that weakly significant findings generated in a linear regression disappear when replicated via logistic regression.10 Similarly, we found that some courses that were weakly significant in linear regression lost statistical significance in the Fisher's exact test: DPT501, DPT504, DPT505, and DPT511 (compare Table 4 vs Table 6). This may reflect a loss of statistical power when converting a continuous variable (NPTE score) to a dichotomous variable (NPTE outcome).14 The decision of whether to analyze these variables as continuous or dichotomous variables is nuanced and may impact the outcome of an analysis, its interpretation, and/or its ease of implementation.15 Extended discussion of the relative merits of these approaches is beyond the scope of this report. However, we appreciate both the ease of interpretation afforded by logistic regression (pass vs fail is the “bottom-line”) and the broader utility of linear regression, which provides guidance even in years when historical data contain only a few NPTE failures.

Back to Top | Article Outline

Design Considerations

Although we had access to multiple graduation cohorts of data, there were several reasons for intentionally selected a single year's dataset. Our dataset was most complete for the 2015 academic year; all other datasets had substantial missing data due to either incomplete grading or large proportion of exam passes (2016 cohort: 97% first-time pass rate, ie, one student not passing on first attempt) or inconsistent in accessible record keeping (years before 2015). Moreover, analysis of a single cohort avoids “batch effects” due to evolutions in course content or instructor, variability in admissions, etcetera. Although we believe multi-cohort datasets could add substantial statistical power to this analysis, the need to control for these confounding variables would make the analysis more difficult to implement and interpret. For this demonstration of approach, a single-cohort analysis is ideal. Our data integrity is especially high: grade information was collected directly from the course instructors; NPTE data were collected directly from the Federation of State Boards of Physical Therapy; demographic variables are reviewed for accuracy institutionally. Our primary outcome of first performance on NPTE is ideal because the ultimate pass rate in NPTE is high, with few failures, which compromises statistical power. Finally, we note that while adding a logistic regression to this study would facilitate connection to previous studies, we felt that it added only incrementally to linear regression and that the systematic Fisher exact test offered much greater perspective.4

Back to Top | Article Outline

Limitations

Our study is limited by the lack of a recommended framework for programmatic review. Cook et al4 recently reported on a possible association between undergraduate GPA and first-time pass rate; however, previous studies have shown much less robust prediction (R 2 < 0.4) between graduate GPA and NPTE outcomes than we did.6 In non-PT domains, for example, Nursing, Medicine, and Psychology, we find the predictive power of undergraduate GPA is mixed,16–19 although comparing results between professions is complicated because of the varying intervals between degree matriculation and licensure exam.

Regarding the present study, we acknowledge the limitations of sample size. We anticipate that this will be a common barrier for most programs wishing to implement this analysis and that the nature of this limitation will vary by program; for example, some programs will have a limited number of retrospective cohorts for analysis, some programs take in a larger class size per year; programs with a particularly high first-time pass rate will have difficulty obtaining reasonable heuristics of NPTE performance due to lack of sample contrast. Furthermore, our study considers only those students successfully completing the program. Students who depart the program without completing their doctoral degree were censored from this dataset due to incomplete course completion information and lack of NPTE results. Similarly, we suspect a reporting bias among those students who elect not to share their NPTE scores with their alma mater. Thus, we recognize the inherent limitations to prospective monitoring for at-risk students caused by these missing data.

We note further that while the primary outcome of our study is first-time pass rate of NPTE, and our main dataset (graduating class of 2015) showed excellent prediction in this setting, this is partly a reflection of a cohort where there was adequate sample of students requiring additional attempts at the NPTE. In contrast, among the graduating class of 2016, all students (minus one) passed the exam on their first attempt. This is an inherent confounder of this type of analysis. For cohorts with high pass rates, where contingency tables would have low cell values, a more appropriate selection of outcome measure would be the raw regression of predictor against the NPTE score. In our exploratory analysis of replication of model fit, we find fair agreement between the 2015 and 2016 cohorts among most predictors, suggesting that the approach of regression is useful year-to-year.

Finally, we observe that the modeling performed here is univariate, meaning that a single variable (NPTE score or outcome) was regressed against a single predictor (course grade or GPA). While there is certain explanatory value in adding additional factors (demographics, other descriptors related to educational history, and past academic performance), it is simply not feasible to further stratify an analysis containing only 26 students. We speculate that adding multiple terms to a given regression may add robustness to our single-term models (for instance, regression of NPTE score against performance in 2 classes), a multivariate regression study requires additional analytical techniques (collinearity analysis, feature selection, and possibly data transformation) that would far exceed the scope of this article and may decrease the accessibility of this method to a broad audience. Nevertheless, we believe there is excellent opportunity to expand this work.

Back to Top | Article Outline

Implications

Our analysis of a partial curriculum grade set has revealed several critical findings. First, we have determined that both individual foundational courses (range: 0.00 < R 2 < 0.60, Table 4) and curricular grade points (0.4 < R 2 < 0.7) have substantial ability to predict NPTE scores through simple linear regression without additional covariates. We note that the 2 GPAs calculated at the time of program entry show modest predictive utility: R 2 < 0.4. We further show that at-risk students can be identified by benchmarking performance within distributions of single course final grades, semester-wise GPAs, and cumulative GPAs (Table 6). Perhaps, the most profound outcome of this study is that at-risk students can be identified with high likelihood as early as the first semester. In our case demonstrated by completion of DPT508 with a grade below 83 points, which was associated with first-time NPTE failure with RR = 10.9 (P < .001; Table 6) and a GPA <3.0 at the end of the first summer semester yielded an outcome prediction RR = 9.3 (P < .01).

It is important to note that while our results are promising (many R 2 > 0.5 and significant RR scores), there is yet substantial variance in the NPTE performance that remains unexplained. While an extended discussion on this matter is beyond the scope of this article, we suggest that one important variable not accounted for here may be time to exam completion. Obviously, there is a tradeoff between recency of academic training versus depth of independent professional practice. We suggest that there may be substantial merit to further study the association between NPTE performance and timing of exam attempt relative to graduation. We encourage others who wish to replicate this study to include other factors in exploration.

This work was inspired by ambition to identify students at risk for failing the NPTE, with an interest in providing early intervention. Rapid and accurate identification of at-risk students is critically important to DPT Programs. Early intervention provides the best opportunity for successful remediation or advising of student who should consider withdrawal from the program given the high cost of tuition and potential long-term consequences of student loans.16 We feel it is prudent to consider every early performance marker as a candidate for prediction. Thus, while not a curriculum item, per se, we considered GPAs at the time of program entry to be worthy of inclusion in this study. Given their poor yield (Tables 4 and 6), we can now assert—at least for this dataset—that preadmission GPA is not a useful predictor of NPTE performance and that screening efforts are better directed toward the courses and graduate GPA thresholds.

Also noteworthy is the volatility of thresholds across the Program: cutoffs within the Program curricula range between 79% and 93%, and GPA thresholds range from 2.6 to 3.4, without any indication of systematic trend. This variability makes programmatic policy making a complicated enterprise. For example, this study provides evidence that simplistic performance criteria, for example, “Must pass each class with a minimum grade of 80 points” or “Must maintain a cumulative GPA of 3.0” may be suboptimal and impractical for their intended purpose of filtering truly at-risk students. It is not uncommon for 2 courses to have substantially different class averages or for a single course to vary year-to-year. One way to accommodate this variability is to consider thresholds that are specific to each course or based on a percentile. For instance, in DPT500, the optimal cutoff for predictive utility was found to be 84 points (Table 5). This corresponds to 14 students (supra-threshold) versus 12 students (sub-threshold); all 14 students passed the NPTE, whereas 9 of the 12 failed. It may be that instead of adopting a fixed-grade threshold at 84 points, it is preferable to set that risk marker at bottom 46% of the class (corresponding to 12 out of 26). Clearly, policy making is optimized when using multiple datasets.

In this study, we present a simple, reproducible analysis of student academic records as predictors of NPTE performance. Because we demonstrate these analyses primarily in the setting of a single cohort from a single DPT program, specific conclusions about which classes or GPAs (or which thresholds therein) are most predictive is not supported. This would require intensive review of a complete dataset comprising many cohorts, with the caveat that results will likely vary year-to-year. However, we urge that other programs consider incorporating a similar data-driven approach to identifying specific classes or GPA criteria with predictive utility for their own program. Furthermore, there is considerable opportunity for others to test the year-to-year variability in these benchmarks and the impact of missing data. We encourage others to replicate our approach and to publicly disseminate their findings.

Back to Top | Article Outline

REFERENCES

1. Jewell DV, Riddle DL. A method for predicting a student's risk for academic probation in a professional program in allied health. J Allied Health. 2005;34:17–23.
2. Kosmahl E. Factors related to physical therapist license examination scores. J Phys Ther Educ. 2005;19:52–56.
3. Mohr T, Ingram D, Hayes S, Du Z. Educational program characteristics and pass rates on the National physical therapy examination. J Phys Ther Educ. 2005;19:60–66.
4. Cook C, Engelhard C, Landry MD, McCallum C. Modifiable variables in physical therapy education programs associated with first-time and three-year National Physical Therapy Examination pass rates in the United States. J Educ Eval Health Prof. 2015;12:44.
5. Meiners KM, Rush D. Clinical performance and admission variables as predictors of passage of the National physical therapy examination. J Allied Health. 2017;46:164–170.
6. Dockter M. An analysis of physical therapy preadmission factors on academic success and success on the national licensing examination. J Phys Ther Educ. 2001;15:60–64.
7. Thieman TJ, Weddle ML, Moore M. Predicting academic, clinical, and licensure examination performance in a professional (Entry-Level) Master's degree program in physical therapy. J Phys Ther Educ. 2003;17. https://www.questia.com/library/journal/1P3-576098241/predicting-academic-clinical-and-licensure-examination.
8. Utzman RR, Riddle DL, Jewell DV. Use of demographic and quantitative admissions data to predict performance on the National Physical Therapy Examination. Phys Ther. 2007;87:1181–1193.
9. Adams CL, Glavin K, Hutchins K, Lee T, Zimmermann C. An evaluation of the internal reliability, construct validity, and predictive validity of the physical therapist clinical performance instrument (PT CPI). J Phys Ther Educ. 2008;22:42–50.
10. Vendrely AM. An investigation of the relationships among academic performance, clinical performance, critical thinking, and success on the physical therapy licensure examination. J Allied Health. 2007;36:e108–123.
11. Spiegelhalter DJ. Probabilistic prediction in patient management and clinical trials. Stat Med. 1986;5:421–433. .
12. Brase CH, Brase CP, Kupresanin J. Student Solutions Manual: Understanding Basic Statistics. Boston: Brooks/Cole, Cengage Learning; 2013.
13. Hollman JH, Rindflesch AB, Youdas JW, Krause DA, Hellyer NJ, Kinlaw D. Retrospective analysis of the behavioral interview and other preadmission variables to predict licensure examination outcomes in physical therapy. J Allied Health. 2008;37:97–104.
14. Deyi BA, Kosinski AS, Snapinn SM. Power considerations when a continuous outcome variable is dichotomized. J Biopharm Stat. 1998;8:337–352.
15. Zhao L, Chen Y, Schaffner DW. Comparison of logistic regression and linear regression in modeling percentage data. Appl Environ Microbiol. 2001;67:2129–2135.
16. Donnon T, Paolucci EO, Violato C. The predictive validity of the MCAT for medical school performance and medical board licensing examinations: A meta-analysis of the published research. Acad Med. 2007;82:100–106.
17. Seldomridge LA, DiBartolo MC. Can success and failure be predicted for baccalaureate graduates on the computerized NCLEX-RN? J Prof Nurs. 2004;20:361–368.
18. Yu LM, Rinaldi SA, Templer DI, Colbert LA, Siscoe K, Van Patten K. Score on the Examination for Professional Practice in Psychology as a function of attributes of clinical psychology graduate programs. Psychol Sci. 1997;8:347–350.
19. Stroman L, Weil S, Butler K, McDonald C. The cost of a number: Can you afford to become a surgeon? Bull R Coll Surg Engl. 2015;97:107–111.
Keywords:

Curriculum review; Exam; Prediction; Risk; Students

Copyright2018 (C) Academy of Physical Therapy Education, APTA