When examining the effect on the labor market of post-secondary education in a particular state, researchers often use administrative outcome data – statewide statistics that look at university graduates and the effect on their employment. Yet when those graduates leave the state, their outcomes are not recorded. This attrition creates a bias in the research.
In their working paper, “Attrition from Administrative Data: Problems and Solutions with an Application to Postsecondary Education,” published with the National Bureau of Economic Research, Kevin Stange, associate professor of public policy, and Andrew Foote, senior economist at the U.S. Census Bureau, explores the potential magnitude of that bias.
“We find that out-of-state migration is particularly problematic for high-earners, flagship graduates, and certain majors. Consequently, the effect of graduating from a flagship university is 10% higher than one would estimate using in-state earnings exclusively, though the extent of bias differs substantially across contexts.” they write.
The researchers begin by examining how significant out-of-state migration is for recent college graduates, which is important to track since many state merit-aid programs aim to retain talented alumni.
“We find that migration out of state is considerable, approaching 30% even among graduates attached to the labor force,” they conclude. “Furthermore, it is not ignorable, as mobility is higher for students at the higher end of the earnings distribution, for certain majors, and for certain institutions.”
Stange and Foote then break down this migration-induced bias by three categories: college selectivity, major, and attainment. When it comes to college selectivity, the researchers find that graduates at the most selective universities do earn more than those attending non-flagship universities, but state records understate this pattern.
Looking at differences in college majors, they find that, “in-state earnings mischaracterize the earnings premium associated with different majors due to differential out-of-state migration.” And, finally, regarding attainment, Stange and Foote find “that the earnings gain associated with obtaining a degree in Career Technical Education (CTE) at a public two-year college is 0.01 log point higher than in-state earnings would suggest.”
After diving into the labor market consequences of college choices, Stange and Foote provide researchers with four lessons about addressing bias arising from attrition:
- “Bias is reduced when the sample is conditioned on having positive observed earnings.”
- “A sensible test of the potential for bias is whether the rate of in-state earnings being observed differs between treatment and control groups.”
- “The Lee (2007) bounding approach is likely inappropriate in this setting given the failure of the monotonicity assumption and bounds are wide and uninformative.”
- “Self-employment is not a major source of differential attrition or bias in the three applications we examined.”
The authors conclude with describing the implications of migration for states. They find that states disproportionately lose their highest-paid alumni and an ability to follow those students out of state is a barrier to evaluating state merit-aid programs' ability to retain talent. Stange and Foote also find that as a result of systematic differences in rates of out-migration, earnings estimates published by states tend to understate earnings differences between institutions and fields.
“While most authors have recognized the potential for bias, prior work has not had access to data that would permit them to directly test the extent of bias,” Stange and Foote conclude. “This study takes advantage of a unique match between postsecondary records from five states with administrative records nationally, permitting us to quantify the extent of bias due to out-of-state migration among U.S. college graduates.”
The working paper was featured in The Hechinger Report:
Payoff for state flagships is 10 percent larger than published data indicate, The Hechinger Report, August 15, 2022
