Faulty Chronicle of Higher Education Survey Deserves More Scrutiny

As a person who appreciates insights into the thinking of various actors in higher education, I get excited whenever a newspaper invests its resources in fielding a large survey of college faculty, administrators, students, or those who hire them.

Thus, I was initially excited to see the results from the latest survey conducted by the Chronicle of Higher Education and America Public Radio's Marketplace, on how employers view the skills and talents of recent college graduates.  Reading through the results, one began to think the story contained provocative information on business's view of online education, for example.

But then I got to the story's end, and stopped cold. There lay the description of survey methodology: The findings on these pages come from a survey developed, fielded, and analyzed by Maguire Associates Inc., a higher-education consulting firm, on behalf of The Chronicle and American Public Media's Marketplace. Maguire invited 50,000 employers to participate in the study. Experience.com, a career-services consultancy, helped develop the sample by providing a contact list of employers that recruit recent college graduates. The survey was conducted in August and September 2012. There were 704 responses.

First, that's the sum total of all details we're given about the sample.  We know nothing about the characteristics of the sampling frame or the respondents.  In other words, we have no idea whether these employers are representative of specific sectors, specific states, etc.  And equally importantly, while the author didn't do the math for us, a simple calculation reveals a response rate of just 1.4%.

This is insanely low.  So low, in fact, that I don't think the results should be published at all-- let alone featured on the Website.  And if they are, a major banner is needed proclaiming at the start of the story, "PROCEED WITH CAUTION!"  I said the same thing regarding an Inside Higher Ed survey last year, and that one had a response rate of 7%.

Yes, I understand that surveys are expensive to field and really costly to do well, but at the very minimum the public deserves far more clarity and transparency in reporting on methods than the Chronicle has provided here. In addition, there are clear best practices for fielding web surveys (which I have to assume this was) and it sure doesn't seem like Maguire Associates followed them.  For example, Paul Umbach of North Carolina State wrote a nice piece specifically for those conducting higher education research that lays out the issues point by point. Given that low response rates may lead to sample bias (affecting generalizability), low power, and inaccurate estimations, the researchers should have randomly sampled from the target population only enough people to have sufficient power-in other words, going after a fraction of those 50,000 with solid methods instead of surveying all of them with weak technique.  And the overarching goal should have been to achieve a decent rate of response to reduce the possibility of a non-representative sample-- in this case, the response rate is pathetically low (this is as low as I think I've ever seen) and no information on representativeness is provided.  This doesn't seem uncommon for Maguire Associates-- this 2011 survey report says nothing about their sampling or response rate, nor does this one from 2010, but this one from 2005 and this one from 2008 provide at least some of the information that Chronicle readers deserve.

It's unethical to publish this information like this. So I recommend that the Chronicle do what I asked Inside Higher Ed to do-- take it down! And consider working with a more credible survey firm next time, please.

The MOOC Industrial Complex

These days, every education reform movement seems to generate profit for multiple partners.  Take No Child Left Behind, the latest testing and accountability regime. As many scholars have documented, billions of dollars have flowed to corporations providing the tests, textbooks and "supplementary education services" required by that federal policy.  Advocates say this is appropriate since it means the market is functioning freely to provide high quality services, while critics note that absent government regulation (carefully limited under the law) public goods are quickly becoming private ones.

In recent blogs about MOOCs, I questioned their business model, asking why supposedly cash-strapped universities (like mine) would choose to engage with them when there is no evident monetary return?  I received little response from MOOC advocates on that question. But the answer is becoming increasingly clear.

Many universities have stated that MOOCs are the kind of innovative activity that donors would like to support, and that we would gain new donors if we engaged in creating them.  University foundations, including UW-Madison's, seem quite confident in this-- to the point that they are putting in their own dollars for the initial investments to get MOOCs off the ground.  While it's possible that this is sheer altruism, it's doubtful-- their leaders always expect a return on investment.

Where will it come from?

Why, the industry that supports the MOOCs of course.  For example, today's New York Times  documents the growing testing industry associated with online proctoring, helping colleges "keep an eye" on online test-takers. Unfortunately that article failed to investigate the money involved in this effort, focusing instead on the quality of the services provided to prevent cheating. This is unfortunate, since tracking the dollars created by the work of educators, ultimately benefiting the corporate bottom line is exactly what the "paper of record" should be doing.

What other entities will benefit from the instructional activities provided by public and non-profit universities?  Write in and let's start a list.  And while we're at it, let's think hard about the likelihood that these corporations will eventually constitute those "benevolent new donors" our universities are counting on.


ps. I keep meaning to direct my readers to this excellent commentary in the Chronicle of Higher Education. I highly recommend reading it in full.