How to Evaluate a MOOC

Since we're already doggie-paddling around the deep-end of the MOOC phenomenon, it seems high time to begin thinking about how to assess MOOCs.  This is far from straightforward, since they are not typical courses and might best be conceived of as extension or continuing education activities.

Very few colleges and universities are revealing their assessment practices for MOOCs; Duke is one exception.  This report explains that selective university's experiences with one course, and demonstrates how their administrators and faculty thought about measuring success. In my view, some of it is good, and some of it is exactly what not to do.

So here is my overview of an initial evaluative framework for thinking about whether or not a MOOC was a "success." I'm posting this with the explicit intention of generating discussion to refine this framework and build it out, so please add your two cents.

The framework has three components: an assessment of costs expended to generate and implement the MOOC, an assessment of its intended impacts in multiple domains, and as assessment of unintended consequences.  Only by considering all three can one make an informed decision about whether a MOOC was worthwhile, yet reporting on MOOCs suggests that many institutions are focusing on only one or two of the three.

1. COSTS
1. How many total hours of effort were required to build the MOOC?
1. How many of these were on the part of the faculty member?
2. How many involved TA time?
3. How many involved staff time?
2. What technological costs were incurred?
1. How much was spent to get the MOOC off the ground?
2. What are the regular maintenance costs?
3. What are the costs of upgrading the MOOC?
3. How many hours of administrative time were devoted to meetings about the MOOC? 
1. How much time and money is spent advertising the MOOC?
2. How much time will be required to regulate it?
3. How much time will be required to assess its outcomes?
2. IMPACTS
1. On Students.  
1. How many students initially enrolled in the MOOC?
2. How many of these were students were not already enrolled in the university?
3. How many countries are represented among MOOC users?
1. How many of these countries were previously not served by the university, or represent new relationships?
2. How many of these countries are home to under-educated populations?
4. How many of these students had no college experience prior to the MOOC?
5. How many of these students already held a college degree of any kind?
6. What percent of the total initial enrollment completed the first week of the course?
7. What percent of the total initial enrollment completed the first assessment?
8. What percent of the total initial enrollment completed the entire course?
2. On Faculty.  
1. What was the professor's level of satisfaction with the course, as compared to satisfaction of teaching a usual university course?
2. What new contacts for the professor resulted from the MOOC, related to his/her research agenda?
3. How did teaching the MOOC inform the professor's research agenda?
4. How did teaching the MOOC change how the professor approaches his/her regular university courses?
5. Is there any evidence that the professor and/or his/her research gained more recognition in the community where MOOC students came from? (e.g. media mentions, etc)
3. SIDE EFFECTS
1. How many course releases (from campus teaching) were required for the professor to teach the MOOC, and how many students on average are taught per course (typically)?
2. How many press mentions occurred regarding the MOOCs?
1. How many were positive?
2. How many were negative?
3. What did state legislators say about the MOOCs?
1. How many felt the MOOC was a welcome activity?
2. How many felt the MOOC was an unwelcome activity?
4. What is the dollar value of the activities incurred by the MOOC contracted out to the provider rather than performed "in-house"?

As I said, this is just a rough start.  Write in, and help me flesh this out!

Nick Jones: Candidate for UW-Madison Chancellor

This post is the second in a series of four.

It's hard to say much about Nick Jones, candidate for chancellor of UW-Madison, because hardly anyone seems to know who he is.  He's spent most of the last 30 years at a single, very elite private institution-- Johns Hopkins University-- where he's currently the dean of the Whiting School of Engineering.  About 10 years ago, Jones left Hopkins about about two years to work at the University of Illinois at Urbana-Champaign, but quickly returned. Apart from engaging in various engineering-related activities, he doesn't seem to have done much in higher education leadership.

That said, Jones is well-liked by those in the Hopkins community, and appears to be good at fundraising. Very nice.

But this is a case where having been a dean and the product of one institution may simply not be enough. He talks in terms of sports metaphors when describing his current job, noting that  "I’m an offensive lineman. Basically, I run interference for my faculty. At the end of the day, it’s the faculty who are on the field, trying to get stuff done and put together creative ideas. My job is to facilitate and make that happen. I have to get out in front and clear the path so that they can do what they do. I help get any impediments out of the way, such as administrative ones, and bring resources to the table to help make their aspirations come true."  Well frankly, being chancellor at Madison is going to require dealing with many more types of impediments and resources will be hard to come by-- and the job is first and foremost to help students, not faculty.

Johns Hopkins University is, to be quite frank, not a training ground for leading UW-Madison at all. It's small, very privileged, and well-known for being incredibly siloed. Jones seems quite nice, but frankly unprepared for this work.

Kim Wilcox: Candidate for UW-Madison Chancellor

PREFACE:

There are four candidates for UW-Madison Chancellor. This week I will profile each of them, contributing information gleaned from "off list" discussions and sleuthing. As I noted in my last couple of blogs, unfortunately that sort of due diligence was not undertaken by the search firm.

I'm doing this in the spirit of sifting and winnowing, with an eye towards helping us identify the candidate who best suits UW-Madison with its many strong traditions-- foremost among them our tradition of shared governance. I hope you will join me in that spirit, refraining from engaging in name-calling or sheer speculation, while sharing any useful information you may have, using the comments function on this blog.

Until December 2012, Kim Wilcox was the Provost and Vice President for Academic Affairs at Michigan State University, where he also served as a professor in the Department of Communicative Sciences and Disorders, a member of the MSU Foundation board of directors, and on the board of directors of the Spectrum Health – MSU Alliance Corporation. 

Earlier in his career, Wilcox was dean of the College of Liberal Arts and Sciences and vice provost for general education coordination at the University of Kansas. He was born in Michigan, and received his bachelor’s degree in audiology and speech sciences from Michigan State University, and his master’s and doctorate from Purdue University, both in speech and hearing science.

This is not Wilcox's first attempt at a chancellor position. Last year, he was up for the position at the University of Hawaii-Manoa, and according to some sources earlier he was considered for a similar job in Minnesota, in a search he pulled out from. It seems he's been looking to leave his current job for awhile.

While also a public university, Michigan State is a very different academic environment from UW-Madison, most critically with regard to its top-down style of governing. The president and provost hold a great deal of power, and make nearly all campus decisions with the deans--and without consultation with the faculty. This sort of background is not an asset when it comes to preparing for the chancellorship of UW-Madison. 

Some colleagues find Wilcox to be the ultimate technocrat, citing his love of metrics and accountability, and noting that under his leadership faculty governance is an "empty enterprise." Others who are more fond of him seem to generally be students and faculty who've known no other approach to decision-making, and are pretty happy to be "allowed a voice" in decisions.

One of the key issues on which people speak most vehemently about Wilcox is in regards to his efforts to restructure academic departments. Here is a video of Wilcox -- around the 6 minute mark, he announces his suggestion for consolidation of some departments within a college, “reducing the number from 13 to 6.”  These efforts mainly focused on the liberal arts, the Classics in particular-- which he apparently proposed eliminating--but I'm also told he would have eliminated Geology if a big donor hasn't intervened.  Commentary on discussion boards tend to center on these issues-- for example see this message board and this reaction to the Classics decision.

When it comes to student enrollment, it has been said that “under Wilcox's direction, MSU has grown its enrollment to more than 49,000 students while raising the academic credentials of the entering class, increasing the percentage of students from underrepresented groups, decreasing the average time to degree, increasing the graduation rate for undergraduates and decreasing the percentage of students graduating with any accumulated debt” However, while an examination of MSU's data digest (h/t to that helpful grad student, you know who are you!) supports most of these claims, it's also worth noting that during Wilcox's tenure, the racial/ethnic gap in graduation rates widened by 40%. The reliance on adjuncts at the university also increased, as did the concentration of racial/ethnic minorities in that group of faculty. Overall, I find little evidence that Wilcox has systematically worked to address issues of achievement gaps at MSU, a major issue on that campus and ours.

To conclude: while I have little doubt that the search and screen committee found much value in Wilcox's portfolio of materials, I have serious concerns about his ability to adapt and thrive in our campus governance system. I am also quite unsure that he will treat the issues of diversity that we face on this campus with the seriousness of purpose that they deserve. For this reason, at this point he is not my choice for chancellor.

Postscript: Wilcox was up for the position of chancellor at the University of Wyoming and did not get it.

Postscript:

Just received this email about Wilcox from MSU colleagues-- seems important enough to share:

There are quite a few serious "negatives" here that anyone who might be in a position to have influence should know about. One point, for example, is the statement Wilcox made to the NYT Education Life section that appeared on Dec. 29, 2009: "Kim Wilcox, the provost and vice president for academic affairs at Michigan State, notes that universities, his included, used to offer majors in elocution and animal husbandry. In a major re-examination of its curriculum, Michigan State has added a dozen or so new programs, including degrees in global studies and, in response to a growing industry in the state, film studies. At the same time, it is abandoning underperformers like classical studies: in the last four years, only 13 students have declared it their major." As if his condescension weren't bad enough, the statement he made here was simply false. We graduated five to six majors per year with an average of 25-26 majors on the books each of the four years he cites. This, with three regular faculty members. The stated reason for cutting programs, especially in our case, was the budget, but, again, the numbers simply weren't there. There were also serious violations of governance and procedures when Classics was eliminated.