Open UToronto MOOC Initiative: Report on First Year of Activity
August 2013
Laurie Harrison
[ Download the PDF Version of this Report ]
The University of Toronto took the lead among Canadian universities, beginning our exploration of the Massive Open Online Course (MOOC) arena in the summer of 2012. With little research or data available on this emergent phenomenon, a strategy of learning through the experience of design and implementation was adopted in order to inform our understanding of the potential impact both within our institution and beyond. Under the umbrella of the broader Open UToronto initiative, the following overarching goals were identified:
◦ Contribute to the education community and the broader public through sharing of institutional expertise and open curriculum content
◦ Explore and evaluate a range of pedagogical approaches and open course platforms
◦ Leverage use of open educational resources through integration into University of Toronto degree program courses
◦ Showcase the University of Toronto’s capacity as a leading institution for teaching, learning and research
◦ Advance global innovation in online learning through development of new course structures, methods and instructional strategies
The University of Toronto currently has partnerships with two MOOC platform providers. In July 2012 an agreement was put in place with Coursera, an education company that provides a specialized MOOC platform at no cost to the university. A second partnership arrangement was undertaken with the EdX initiative, a non-profit created by founding partners Harvard and MIT. In both cases the learning environment is hosted by the partner organization, and course content design and instruction is the responsibility of the University of Toronto.
To date, seven Coursera MOOCs have been offered across a range of discipline areas. These short, fully online “coursettes” ranged from 4 to 8 weeks in length, and have attracted more than 500,000 registrants before, during and after the scheduled sessions. Seven more MOOCs, including new EdX-hosted offerings are currently in the design phase.
The following is a summary of MOOCs that have been offered, scheduled or are in development.
MOOCs Offered
|
Description |
Platform |
Sessions |
|
|
Learn to Program: The Fundamentals Jennifer Campbell and Paul Gries Department of Computer Science |
Coursera |
Oct. 2012 |
Aug. 2013 |
|
Neural Networks for Machine Learning Geoffrey Hinton Department of Computer Science |
Coursera |
Oct. 2012 |
|
|
The Social Context of Mental Health and Illness Charmaine Williams Factor-Inwentash Faculty of Social Work |
Coursera |
Jan. 2013 |
June 2013 |
|
Aboriginal Worldviews and Education Jean-Paul Restoule Ontario Institute for Studies in Education |
Coursera |
Feb. 2013 |
|
|
Learn to Program: Crafting Quality Code Jennifer Campbell and Paul Gries Department of Computer Science |
Coursera |
March 2013 |
|
|
Statistics: Making Sense of Data Alison Gibbs and Jeffrey Rosenthal Department of Statistics |
Coursera |
April 2013 |
|
|
Steve Joordens Psychology – UT Scarborough |
Coursera |
May 2013 |
|
MOOCs Scheduled in Next Six Months
|
Description |
Platform |
Sessions |
|
Behavioral Economics in Action Dilip Soman Rotman School of Management |
EdX |
Oct. 2013 |
|
Bryan Karney Faculty of Engineering |
EdX |
Oct. 2013 |
|
Death 101: Measuring Global Causes of Death Prabhat Jha Dalla Lana School of Public Health |
EdX |
May 2014 |
MOOCs Currently In Proposal Development
|
Description |
Platform |
Sessions |
|
BioInformatic Methods I BioInformatic Methods II Nicholas Provart Department of Cell and Systems Biology |
Coursera |
TBC |
|
The Logic of Business: Building Blocks for Organizational Design Mihnea Moldoveanu Rotman School of Management |
EdX |
TBC |
|
Library Advocacy Wendy Newman iSchool |
EdX |
TBC |
Patterns of Activity:
Of the total 366,424 registrants during Coursera MOOC sessions completed to date, 207,566 accessed video content, thus 57% may be considered to be active learners. Many of these learners were curious about the subject area, browsed for information on particular topics or explored the MOOC format in general, taking advantage of the ease of registration to participate to the level appropriate to their needs and interests. The aggregate completion rate for Coursera MOOCs offered to date is 8% for all registrants, and 17% for active learners, which is well within the normal range for this mode of delivery. A total of 6 Coursera MOOCs are currently complete.
|
MOOC Activity Summary Data for Completed Scheduled Sessions – July 25, 2013 |
||||||
|
Activity and Completion Statistics |
total registrants |
active learners* |
posts in discussion |
successfully completed |
percentage registrants who completed |
percentage active learners who completed |
|
Complete |
|
|
|
|
|
|
|
Learn to Program: The Fundamentals
|
80000 |
75450 |
23055 |
8240 |
10% |
11% |
|
Neural Networks for Machine Learning
|
49550 |
15903 |
5192 |
1398 |
3% |
9% |
|
The Social Context of Mental Health and Illness
|
23491 |
8193 |
13289 |
1423 |
6% |
17% |
|
Aboriginal Worldviews and Education
|
20966 |
8860 |
34112 |
3381 |
16% |
38% |
|
Learn to Program: Crafting Quality Code
|
53974 |
17224 |
4709 |
3352 |
6% |
19% |
|
Statistics: Making Sense of Data
|
62488 |
36356 |
5224 |
2825 |
5% |
8% |
|
Total |
366424 |
115042 |
207566 |
17794 |
8% |
17% |
* an active learner is person who watched at least one video
An unexpected ongoing pattern of activity is learner registration and use of MOOC materials after the scheduled session is completed. An additional 136,008 users have registered and are seen to be active in Coursera MOOCs after the close of scheduled MOOCs. Additional research is needed to understand the implications of this activity and is being taken up by faculty researchers.
|
MOOC Summary Data During Archive Period – July 25, 2013 |
|||
|
MOOC archive |
registration at end of session |
total registration to date |
# archive users |
|
Learn to Program: The Fundamentals |
80000 |
139852 |
59852 |
|
Neural Networks for Machine Learning |
49550 |
77464 |
27914 |
|
The Social Context of Mental Health and Illness |
23491 |
34060 |
10569 |
|
Aboriginal Worldviews and Education |
20966 |
30056 |
9090 |
|
Learn to Program: Crafting Quality Code |
53974 |
76945 |
22971 |
|
Statistics: Making Sense of Data |
62488 |
68100 |
5612 |
|
Total Archive Users |
|
|
136008 |
Research Program
To further our understanding of these new online course formats, Online Learning Strategies (OLS), together with the MOOC instructors, have developed an extensive research program around MOOCs and inverted (flipped) classrooms. Some of the topics being investigated include:
- Student demographics and learning goals/approaches
- Range of pedagogical approaches that are available to instructors
- Patterns of engagement and learning among participants.
The faculty members contributing to this research program to date are:
- Paul Gries and Jennifer Campbell (Learn to Program; Crafting Quality Code)
- Alison Gibbs and Jeffrey Rosenthal (Statistics: Making Sense of Data)
- Steve Joordens (Introduction to Psychology)
- Charmaine Williams (The Social Context of Mental Health and Illness)
- Jean-Paul Restoule (Aboriginal Worldviews and Education)
Two of these courses (Statistics: Making Sense of Data, and Introduction to Psychology) have received special support from the Gates Foundation to research their MOOCs. The Statistics MOOC is also being used in another Gates funded project, a multi-campus study of MOOCs as a deeply integrated instructional resource for blended learning at universities across the Maryland system.
Current Data Analysis Activities and Studies
Quantitative Data Analysis: The Coursera platform provides rich data on learner interaction with the courses, and we will complement this learning analytics data with data on student demographics and learning intents from intro- and exit-surveys. The OLS is coordinating individual research projects by the different PIs, and facilitating access to data from Coursera, as well as compiling general demographic data now available. See Appendix A: Demographic Report on University of Toronto Coursera MOOCs.
Qualitative Analysis of Instructor Experience and Design Outcomes: In addition to the research on learners, and student interactions with courses, we are also interested in instructors, their experiences and attitudes, and the process of constructing courses. OLS has launched a separate research project, supervised by Dr. Carol Rolheiser from the Center for Teaching Support and Innovation (CTSI), which aims to interview all UofT MOOC instructors about their experience of building and running a MOOC, as well as analyzing the course designs and pedagogical approaches of the various courses.
A full institutional research report is anticipated in August 2013. As well, research studies being undertaken by individual faculty members are forthcoming in the near future. Online Learning Strategies is facilitating a MOOC research group that is working collaboratively on joint studies, development of protocols for ethics review and discussion of data management and analysis strategies.
Design Process:
Given that both our Coursera and EdX partnerships are institutional initiatives, each of the MOOCs proposed for development at the University of Toronto is first reviewed by the divisional Dean, and also endorsed by the Office of the Provost. Advice on governance and process is provided by the Open UToronto Advisory Committee. A workflow for developing an open online course was developed and published in February of 2013. This documentation along with additional resources to support planning and design is available on the Open UToronto web site:
◦ Overview of Approaches to Open Course Content
◦ MOOC Resourcing and Planning Guidelines
◦ MOOC Design and Development Guidelines
◦ MOOC vs. Degree Courses – FAQs
Our design process relies heavily on a team-based process. Typical teams include the instructor(s), educational technology leads, librarians, AV technicians and TAs. While development teams meet regularly to discuss design and production plans, we have also held regular “round table” meetings where all teams could meet together to discuss ideas and progress.
Of particular interest was the EdX Design Workshop day, when design teams gathered at CTSI)for a day of learning, collaborative planning and discussion of instructional challenges in the MOOC environment. Two guests from EdX visited from Boston for the day to provide coaching on use of the EdX tool kit and provide advice on MOOC video production strategies. The event was loosely based on the Course Design Institute model that is offered annually by CTSI. The EdX MOOC cohort is using and adapting well-established design theories and models to the MOOC environment. These include:
Wiggins, G. P., & McTighe, J. A. (2005). Understanding by design. ASCD.
Akyol, Z., & Garrison, D. R. (2013). Educational Communities of Inquiry: Theoretical Framework, Research and Practice. IGI Global.
◦ Best Practice Models for e-learning
Staffordshire University. (n.d.). Best Practice Models for e-learning. Retrieved July 31, 2013, from http://learning.staffs.ac.uk/bestpracticemodels/
Each MOOC team is responsible for the design and structure of content and learning activities, specific to their discipline and learner needs. In each of our MOOCs, the learning design undertakings extend beyond provision of lecture material. Individualized activities, discussion frameworks, assessment processes and community building have been the focus of exploration, pushing the boundaries of constraint within the large scale of the MOOC environment.
Inverted Classroom Design
Several of our MOOCs have been, or will be used as curriculum content for use in redesign of University of Toronto degree courses for the inverted classroom or “flipped” model. These include:
◦ Learn to Program: The Fundamentals
◦ Learn to Program: Crafting Quality Code
◦ Neural Networks for Machine Learning
◦ Statistics: Making Sense of Data
◦ Behavioral Economics in Action
The inverted classroom design strategy leverages the MOOC platform to deliver course content in advance of scheduled classes. As a result, the amount of time spent on active learning strategies during scheduled meeting time can be increased. A typical format involves viewing instructor-created videos and completing online tasks or quizzes assigned as homework. Students arrive prepared to apply their new knowledge by solving problems or doing practical work together in the classroom, guided through the process by the instructor. Other possible benefits are use of the materials prior to a course commencing, to ensure all students are familiar with prerequisite content and skills, or during a course for remediation on any challenging concepts or lessons.
Three of our faculty members in the Faculty of Arts and Science have research studies underway that are specifically focused on learning more about the potential of the inverted classroom model. Our involvement in the development of MOOCs has sparked broader interest in both the potential of online instructional design strategies as well as innovative approaches to campus-based degree course curriculum.
Next Steps:
We are moving from the early pioneering phase into our second year of exploration of the potential of the MOOC model. Our faculty and teams are well informed by our experience to date, but all agree that we have much to learn within the fast-shifting MOOC landscape. We are now entering a period where MOOCs are being offered for a second time, with adjustments and adaptation based on student and instructor feedback. Six new MOOCs are in the design phase, with registrations rolling in for the most recently scheduled sessions. Research teams are engaged in data gathering and analysis. As we move into this next period, our strategy going forward has three underlying principles:
◦ Learn about the potential of emergent online models from the leading edge of design and instruction innovation.
◦ Provide design support and team-based development strategies to build capacity and ensure the quality of our online learning initiatives.
◦ Engage in research and evaluation activities to gather evidence and inform our future planning related to MOOCs and online learning.
Acknowledgements:
This report reflects the collective achievement of all those involved in supporting design and implementation of our MOOCs at the University of Toronto. Colleagues whose continuing commitment to ensuring the highest quality experience for our learners include our award-winning faculty, instructional design teams, library colleagues, educational technology professionals, teaching assistants, advisory committee members and many others who have contributed to our success.
Appendix A: Demographic Report on University of Toronto Coursera MOOCs
Prepared June 16, 2013 by Stian Haklev
Background
In order to support MOOC-related research and evaluation processes, the Office of Online Learning Strategies is involved in administrating access to detailed user data sets from Coursera and cleaning it for analysis. The data is being stored centrally and made available for a range of purposes, including Institutional planning processes. It will also be used in a research study among five MOOCs that launched between January and May 2013. The course instructors are all co-PIs on the ethics proposal, and will share the analytics data. For a full description of activities visit: Open Utoronto MOOC Research and Evaluation.
The data
In addition to the intro- and exit-surveys, for students who choose to respond, we also have access to learning analytics data for all students in the chosen courses, including when students log on, their activity, their success at quizzes, etc. The data is anonymized, but we are able to link intro- and exit-surveys with student activity for each individual student. The co-PIs are planning to explore the relationships between different variables, but for this initial institutional report, we are simply reporting some of the key variables from the intro-survey.
The survey
A generic intro- and exit- survey was developed, and offered to students in all courses (with some additional questions and categories for specific courses). This survey, with adaptation as appropriate, will be used in all future UofT MOOCs. Due to the timing of the ethics proposal, the survey was offered to participants in The Social Context of Mental Health and Illness, and Aboriginal Worldviews and Education quite late in the course, which can explain the low response rate.
MOOC Courses
|
Course name |
Instructor(s) |
Enrolment |
Response rate |
|
The Social Context of Mental Health and Illness |
Charmaine Williams
|
22.500 |
6% |
|
Aboriginal Worldviews and Education |
Jean-Paul Restoule |
20.600 |
8% |
|
Learn to Program: Crafting Quality Code |
Paul Gries Jennifer Campbell |
54.700 |
17% |
|
Statistics: Making Sense of Data |
Allison Gibbs Jeffrey Rosenthal |
51.500 |
34% |
|
Introduction to Psychology |
Steve Joordens |
60.700 |
32% |
Tables reflecting summary data collected intro surveys for these five MOOCs are provided below. For more information contact Stian Haklev, Institutional Researcher for the Open Utoronto initiatives.
User Categories
Education Levels
Age Groups
Why Students Enrolled
Gender
Level of English
