Skip to main content
SpringerLink
Log in

The Reliability and Validity of Using Regression Residuals to Measure Institutional Effectiveness in Promoting Degree Completion

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

A relatively simple way of measuring institutional effectiveness in relation to degree completion is to estimate the difference between an actual and predicted graduation rate, but the reliability and validity of this method have not been thoroughly examined. Longitudinal data were obtained from IPEDS for both public and private not-for-profit 4-year institutions (n = 1496). Hierarchical panel regression was used to predict 4- and 6-year graduation rates based on structural, demographic, financial, and contextual attributes. A direct effects model yielded effectiveness scores that were highly correlated between consecutive data years (r = 0.65–0.80), which indicated acceptable to good test–retest reliability. A test of convergent validity indicated that effectiveness scores were positively associated with students’ perceptions of a supportive campus environment (r = 0.32–0.45). A test of discriminant validity revealed relatively small correlations between effectiveness scores and institutional attributes, such as educational expenditures (r = 0.07–0.16). The modeling of interaction effects in relation to institutional type marginally improved the validity of effectiveness scores among public but not private institutions. The results suggest that correct model specification can yield residual scores that reliably and validly measure institutional effectiveness in promoting timely degree completion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. The New Economy Index reflects economic performance in the domains of knowledge jobs, globalization, economic dynamism, the digital economy, and innovation capacity (Atkinson and Andes 2010).

  2. While both the ACT and SAT are used to assess academic preparedness, differences in their content permit only a rough approximation of concordance (ACT 2012).

  3. The NSSE custom analysis service does not permit researchers to examine potential sources of correlation attenuation. In order to examine this possibility, data were collected from 53 public institutions that had publicly accessible data from the 2004 NSSE administration. After removing two high-leverage outliers, the correlation between the NSSE SCE scale and the institutional effectiveness scores was very strong, r = 0.73.

References

  • ACT. (2012). ACT-SAT concordance. Retrieved from http://www.act.org/aap/concordance/.

  • Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through college. Washington, DC: U.S. Department of Education.

    Google Scholar 

  • American Association of State Colleges and Universities. (2002). Accountability and graduation rates: Seeing the forest and the trees. Retrieved from http://archive.sheeo.org/access/Graduation%20Rate%20Brief%20-%20AASCU.pdf.

  • Astin, A. W. (1993). What matters in college: Four critical years revisited. San Francisco: Jossey-Bass.

    Google Scholar 

  • Astin, A. W. (2006). Making sense out of degree completion rates. Journal of College Student Retention: Research, Theory & Practice, 7(1–2), 5–17.

    Google Scholar 

  • Astin, A. W., & Oseguera, L. (2005). Degree attainment rates at American colleges and universities. Los Angeles: Higher Education Research Institute, UCLA.

    Google Scholar 

  • Aten, B. H., Figueroa, E. B., & Martin, T. M. (2012). Regional price parities for states and metropolitan areas, 2006-2010. Retrieved from http://www.bea.gov/scb/pdf/2012/08%20August/0812_regional_price_parities.pdf.

  • Atkinson, R. D., & Andes, S. (2010). The 2010 state new economy index. Retrieved from http://www.itif.org/files/2010-state-new-economy-index.pdf.

  • Bailey, T., Calcagno, J. C., Jenkins, D., Leinbach, T., & Kienzi, G. (2006). Is student-right-to-know all you should know? An analysis of community college graduation rates. Research in Higher Education, 47(5), 491–519.

    Article  Google Scholar 

  • Bailey, T., & Xu, D. (2012). Input-adjusted graduation rates and college accountability: What is known from twenty years of research? Retrieved from http://www.hcmstrategists.com/contextforsuccess/papers/LIT_REVIEW.pdf.

  • Berger, J. B., & Milem, J. F. (2000). Organizational behavior in higher education and student outcomes. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. XV, pp. 268–338). New York: Springer.

    Google Scholar 

  • Campbell, C. M., & Cabrera, A. F. (2011). How sound is NSSE? Investigating the psychometric properties of NSSE at a public, research-extensive institution. The Review of Higher Education, 35(1), 77–103.

    Article  Google Scholar 

  • Chen, R. (2012). Institutional characteristics and college student dropout risks: A multilevel event history analysis. Research in Higher Education, 53(5), 487–505.

    Article  Google Scholar 

  • Choy, S. (2002). Nontraditional undergraduates: Findings from “The Condition of Education, 2002.” Retrieved from http://files.eric.ed.gov/fulltext/ED471077.pdf.

  • Corman, H., & Davidson, P. K. (1984). Economic aspects of post-secondary schooling decisions. Economics of Education Review, 3(2), 131–139.

    Article  Google Scholar 

  • Dougherty, K. J., Natow, R. S., & Vega, B. E. (2012). Popular but unstable: Explaining why state performance funding systems in the United States often do not persist. Teachers College Record, 114, 1–41.

    Google Scholar 

  • Field, K. (2013, August 22). Obama plan to tie student aid to college ratings draws mixed reviews. Chronicle of Higher Education. Retrieved from http://chronicle.com/article/Obama-Plan-to-Tie-Student-Aid/141229/.

  • Furr, M. (2011). Scale construction and psychometrics for social and personality psychology. Thousand Oaks, CA: SAGE Publications.

    Google Scholar 

  • Gansemer-Topf, A. M., & Schuh, J. H. (2006). Institutional selectivity and institutional expenditures: Examining organizational factors that contribute to retention and graduation. Research in Higher Education, 47(6), 613–642.

    Article  Google Scholar 

  • Goenner, C. F., & Snaith, S. M. (2004). Accounting for model uncertainty in the prediction of university graduation rates. Research in Higher Education, 45(1), 25–41.

    Article  Google Scholar 

  • Gordon, J., Ludlum, J., & Hoey, J. J. (2008). Validating NSSE against student outcomes: Are they related? Research in Higher Education, 49(1), 19–39.

    Article  Google Scholar 

  • Hemphill, J. F. (2003). Interpreting the magnitudes of correlation coefficients. American Psychologist, 58(1), 78–80.

    Article  Google Scholar 

  • Horn, L. (1999). Stopouts or stayouts? Undergraduates who leave college in their first year. Retrieved from http://nces.ed.gov/pubs99/1999087.pdf.

  • Horn, A. S. (2013). The selection of peer states for performance benchmarking in higher education. Retrieved from http://www.mhec.org/sites/mhec.org/files/20130429peer_state_selection_working_paper.pdf.

  • Horn, L., & Carroll, C. D. (2006). Placing college graduation rates in context: How 4-year college graduation rates vary with selectivity and the size of low-income enrollment. Retrieved from http://nces.ed.gov/pubs2007/2007161.pdf.

  • Humphreys, D. (2012). What’s wrong with the completion agenda—And what we can do about it. Retrieved from https://www.aacu.org/liberaleducation/le-wi12/humphreys.cfm.

  • Kamens, D. H. (1971). The college” charter” and college size: Effects on occupational choice and college attrition. Sociology of Education, 44, 270–296.

    Article  Google Scholar 

  • Kelchen, R., & Harris, D. (2012). Can value-added methods improve the measurement of college performance? Empirical analyses and policy implications. Retrieved from http://www.hcmstrategists.com/contextforsuccess/papers/KELCHEN_HARRIS_PAPER.pdf.

  • Kuh, G. D., Cruce, T. M., Shoup, R., & Kinzie, J. (2008). Unmasking the effects of student engagement on first-year college grades and persistence. The Journal of Higher Education, 79(5), 540–563.

    Article  Google Scholar 

  • McCormick, A. C., Pike, G. R., Kuh, G. D., & Chen, P. D. (2009). Comparing the utility of the 2000 and 2005 Carnegie classification systems in research on students’ college experiences and outcomes. Research in Higher Education, 50(1), 144–167.

    Article  Google Scholar 

  • McGuinness, A. C. (2011). The states and higher education. In P. G. Altbach, P. J. Gumport, & R. O. Berdahl (Eds.), American higher education in the twenty-first century (pp. 139–169). Baltimore: Johns Hopkins University Press.

    Google Scholar 

  • Michell, J. (2003). Epistemology of measurement: The relevance of its history for quantification in the social sciences. Social Science Information, 42, 515–534.

    Article  Google Scholar 

  • Monks, J. (2000). The returns to individual and college characteristics: Evidence from the National Longitudinal Survey of Youth. Economics of Education Review, 19(3), 279–289.

    Article  Google Scholar 

  • Muthén, B., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P. Marsden (Ed.), Sociological methodology (pp. 216–316). Cambridge: Blackwell.

    Google Scholar 

  • National Survey of Student Engagement. (2007). Reliability: 2007 Institution-level temporal stability. Retrieved from http://nsse.iub.edu/pdf/psychometric_portfolio/Reliability_TemporalStability_Institution-Level_2007.pdf.

  • National Survey of Student Engagement. (2012). Promoting student learning and institutional improvement: Lessons from NSSE at 13. Bloomington, IN: Indiana University Center for Postsecondary Research.

    Google Scholar 

  • NCHEMS. (2013). Workforce conditions. Retrieved from http://www.higheredinfo.org/.

  • Nunnally, J. C. (1988). Psychometric theory. Englewood Cliffs, NJ: McGraw-Hill.

    Google Scholar 

  • Oseguera, L. (2005). Four and six-year baccalaureate degree completion by institutional characteristics and racial/ethnic groups. Journal of College Student Retention, 7(1–2), 19–59.

    Article  Google Scholar 

  • Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students. San Francisco: Jossey-Bass.

    Google Scholar 

  • Pike, G. R. (2013). NSSE benchmarks and institutional outcomes: A note on the importance of considering the intended uses of a measure in validity studies. Research in Higher Education, 54(2), 149–170.

    Article  Google Scholar 

  • Pike, G. R., Kuh, G. D., McCormick, A. C., Ethington, C. A., & Smart, J. C. (2011). If and when money matters: The relationships among educational expenditures, student engagement and students’ learning outcomes. Research in Higher Education, 52(1), 81–106.

    Article  Google Scholar 

  • Pike, G. R., Smart, J. C., Kuh, G. D., & Hayek, J. C. (2006). Expenditures and student engagement: When does money matter? Research in Higher Education, 47(7), 847–872.

    Article  Google Scholar 

  • Porter, S. R. (2000). The robustness of the graduation rate performance indicator used in the U.S. News & World Report college rankings. The Case International Journal of Educational Advancement, 1(2), 145–164.

    Google Scholar 

  • Porter, S. R. (2011). Do college student surveys have any validity? The Review of Higher Education, 35(1), 45–76.

    Article  Google Scholar 

  • Price, D. V., & Tovar, E. (2014). Student engagement and institutional graduation rates: Identifying high-impact educational practices for community colleges. Community College Journal of Research and Practice, 38(9), 766–782.

    Article  Google Scholar 

  • Raghunathan, T. E., Rosenthal, R., & Rubin, D. B. (1996). Comparing correlated but nonoverlapping correlations. Psychological Methods, 1(2), 178.

    Article  Google Scholar 

  • Rhee, B. (2008). Institutional climate and student departure: A multinomial multilevel modeling approach. The Review of Higher Education, 31(2), 161–183.

    Article  Google Scholar 

  • Ryan, J. F. (2004). The relationship between institutional expenditures and degree attainment at baccalaureate colleges. Research in Higher Education, 45(2), 97–114.

    Article  Google Scholar 

  • Scott, M., Bailey, T., & Kienzl, G. (2006). Relative success? Determinants of college graduation rates in public and private colleges in the US. Research in Higher Education, 47(3), 249–279.

    Article  Google Scholar 

  • Smart, J. C., Ethington, C. A., Riggs, R. O., & Thompson, M. D. (2002). Influences of institutional expenditure patterns on the development of students’ leadership competencies. Research in Higher Education, 43(1), 115–132.

    Article  Google Scholar 

  • Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87, 245–251.

    Article  Google Scholar 

  • Stratton, L. S., O’Toole, D. M., & Wetzel, J. N. (2005). A multinomial logit model of college stopout and dropout behavior. Retrieved from http://anon-ftp.iza.org/dp1634.pdf.

  • The National Center for Public Policy and Higher Education. (2008). Measuring Up 2008: The national report card of higher education. Retrieved from http://measuringup2008.highereducation.org/print/NCPPHEMUNationalRpt.pdf.

  • Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition. Chicago: The University of Chicago Press.

    Google Scholar 

  • Titus, M. A. (2004). An examination of the influence of institutional context on student persistence at 4-year colleges and universities: A multilevel approach. Research in Higher Education, 45(7), 673–699.

    Article  Google Scholar 

  • Toutkoushian, R. K., & Smart, J. C. (2001). Do institutional characteristics affect student gains from college? The Review of Higher Education, 25(1), 39–61.

    Article  Google Scholar 

  • Volkwein, J. F., & Grunig, S. D. (2005). Resources and reputation in higher education: Double, double, toil and trouble. In J. C. Burke & Associates (Ed.), Achieving accountability in higher education: Balancing public, academic, and market demands (pp. 246–274). San Francisco: Jossey-Bass.

    Google Scholar 

  • Volkwein, J. F., & Tandberg, D. A. (2008). Measuring up: Examining the connections among state structural characteristics, regulatory practices, and performance. Research in Higher Education, 49(2), 180–197.

    Article  Google Scholar 

  • von Hippel, P. T. (2007). Regression with missing Y’s: An improved strategy for analyzing multiply imputed data. Sociological Methodology, 37, 83–117.

    Article  Google Scholar 

  • Webber, D. A., & Ehrenberg, R. G. (2010). Do expenditures other than instructional expenditures affect graduation and persistence rates in American higher education? Economics of Education Review, 29(6), 947–958.

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to acknowledge the helpful comments of Ernest Davenport as well as the able assistance provided by Allison BrckaLorenz.

Author information

Authors and Affiliations

  1. Midwestern Higher Education Compact, 105 Fifth Avenue South, Suite 450, Minneapolis, MN, 55401, USA

    Aaron S. Horn

  2. Chungbuk National University, E1-1-301, Naesoo Dong Ro 52, Seowon Gu Chungjoo, Chungbuk, 361-763, South Korea

    Giljae Lee

Authors
  1. Aaron S. Horn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giljae Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giljae Lee.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Horn, A.S., Lee, G. The Reliability and Validity of Using Regression Residuals to Measure Institutional Effectiveness in Promoting Degree Completion. Res High Educ 57, 469–496 (2016). https://doi.org/10.1007/s11162-015-9394-7

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-015-9394-7

Keywords

Search

Navigation