Focus Session 2: Quality, Effectiveness, and Affordability of Online Instruction

Summary of Participant Input from Focus Session 2: Quality, Effectiveness, and Affordability of Online Instruction

Question 1: How should we assess and measure the effectiveness and quality of instruction within digital learning environments?

Participants strongly endorsed basing assessments on students’ achievement of the course’s learning outcomes, noting that this isn’t always the case with traditional courses. Participants were emphatic that performance-based assessments need to be implemented far all classes, not just for digitally enhanced ones. Assessments should also measure the efficacy of particular software tools and specific pedagogical practices.

Formative evaluations of the course that occur as the course is in progress should be used to adapt the course to learners’ needs. Course evaluations should include both student input and peer evaluation by faculty. Instructors should have timely access to evaluations. Evaluations should explicitly consider the course’s mode of delivery and technologies. 

Student performance should be assessed frequently, so that students have a good sense of where they stand in the course. 

Be sure that students are aware of the course’s learning objectives before they register and are reminded of them while taking the course.  Learning objectives should be included in course syllabi.

Courses should also be assessed longitudinally with respect to the longer term performance of students (e.g., How did students fare in the next course in sequence?) and of the course over time (e.g., Has student performance in the course improved or worsened over the past five terms?).

Performance of learners in online classes should be compared to that of students in traditional sections to ensure that both types are accomplishing the learning objectives effectively.

Courses may have additional metrics for success than learning objectives.  Is the course helping students to graduate quickly?  Are students withdrawing from the course at a high rate?  Is the course attracting an adequate number of students?

Recommended practices should be researched-based and could come from a standard rubric, such as Quality Matters, or a homegrown one. Participants were emphatic that Quality Matters was an option worth exploring for Purdue to promote quality in course design.

Question 2: What strategies should we use to encourage and support the adoption of high impact practices in digital education?

Participants emphasized the need for a centralized resource, a “hub,” for digital education where resources were aggregated and accessible.  This hub should promote not only best-practice pedagogies but also a robust research initiative in teaching and learning to establish Purdue as a leader in this area.  It should make faculty aware of what is going on in digital education at Purdue and elsewhere.

Build a community of instructors and scholars with experience in infusing digital technologies into their courses who could encourage and support other faculty, probably in their discipline areas, to engage in this arena.  They could share and showcase their experience with effectively incorporating digital technology to groups of faculty.  

Make the identification and implementation of online best practices a cooperative, campus-wide initiative.  Once we have identified a set of best practices and incentivized their use across the campus, we must also recognize that practices will vary by the discipline area and accommodate those differences appropriately.

Faculty who do good work in digital education should be incentivized by recognition in P&T decisions, awards, and a high campus profile.  Financial incentives for faculty and departments should also be part of the incentive package.

We should plan to overcome barriers to adoption of best online practices.  For example, reassure faculty that their value to the university will be enhanced by adoption of digital best practices (they won’t render themselves obsolete) and their intellectual property will be protected.  To address faculty’s academic integrity concerns, we should explore online proctoring and authentication services.  Likewise, we need to properly support faculty on accessibility compliance so that they do not feel vulnerable on that count. Also, we should promote digital education as an integral part of the Purdue educational experience so that parents and students will not feel online classes are substandard.

Other participant suggestions included

  • Incentivizing participation in digital education by offering online courses at a reduced rate
  • Making faculty training in online instruction a requirement before teaching online
  • Addressing scale issues by using more continuing lecturers
  • Adopting course templates that have some best practice features built in

Question 3: How do we make DE affordable and sustainable, including costs for students, and costs of instructional development and implementation?

Participants noted that the university’s online goals must be aligned with a sustainable business model.  If online learning is priced higher than classroom-based learning, students will opt out.  If digital learning is financially disadvantageous to departments, they will opt out. Costs should be transparent to departments.

Several participant groups discussed e-books as a potential cost savings for students.  However, it was noted that e-books eliminate the possibility of selling books at the end of term and that students don’t like reading from a screen in long stretches. It was also suggested that access to e-books can be limited by firewalls, especially internationally.  Working with librarians to create “lib guides” can be a cost-effective alternative to e-books.

It was noted that some online tools, such as homework systems, actually create additional costs for students.  Should in-house development of such tools be considered?  It was also suggested that training students on navigating the LMS and other digital tools could reduce time and frustration on their part.

Centralization was also presented as a means of achieving cost savings.  Services related to online instruction and course development should be centralized.  A central unit should negotiate the purchase/licensing of online tools to ensure best prices. A central unit could negotiate with publishers for best prices and flexibility.

Keep the online development process as simple as possible for faculty. Create a ‘one-stop shop” for the resources.  Ensure that minor changes to the course can be made simply and quickly. Maintain a repository of digital content so that faculty do have to “re-invent the wheel.”

Other cost savings mechanisms considered were contracting with private providers for some services and using student workers for some tasks (e.g., video editing) in the development process.

Participants also suggested revenue enhancements that could offset costs.  These included

  • Attracting students from other institutions to Purdue online courses
  • Ensuring that our online degree programs are competitively priced
  • Creating new pathways for credentials that are responsive to the needs of the job market (i.e., micro-credentials)

Videos

Chantal Levesque-Bristol
Director, Center for Instructional Excellence

How Would You Describe Quality Matters?

Suzanne Ahlersmeyer
Instructional Designer

Articles

OLC Quality Framework

Source: Online Learning Consortium (OLC) website

The online Learning Consortium (formerly Sloan-C) has been a leading organization promoting the incorporation of technology into higher education.  In this excerpt from their website OLC describes its vision of quality in online learning.   OLC says“…learning effectiveness, access, scale (capacity enrollment achieved through cost-effectiveness and institutional commitment), faculty satisfaction, and student satisfaction… [are the] OLC’s Five Pillars of Quality Online Education, the building blocks which provide the support for successful online learning. The intent of the quality framework, which is always a work in progress, is to help institutions identify goals and measure progress towards them.”

Using Quality Matters TM (QM) to Improve All Courses

Author: Diane L Finley
Source: Journal of Teaching and Learning with Technology
Published: December 2012

Quality Matters is one of the most widely used rubrics for quality in online learning. In this article a faculty member explains the process using Quality Matters in her own institution, shares her experience improving her own online courses, and describes what her students are now able to do better while navigating the course and interacting with peers.

Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies

Authors: Barbara Means, Yukie Toyama, Robert Murphy, Marianne Bakia, Karla Jones
Source: US Department of Education Report
Published: September 2010

Since the 1990’s several meta-analyses of research studies on the effectiveness of digitally-enhanced learning have been conducted.  These have typically shown that learners in online courses learn as well as those in traditional classroom settings.  We have included the Executive Summary of a study by the Department of Education which suggests that in some circumstances online learners do slightly better than their counterparts in traditional classes.  The summary indicates some pedagogies that have a significant effect on learning and some audiences for whom online learning is more effective.  You can read the entire report here [https://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf].

Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial

Authors: William G. Bowen, Matthew M. Chingos, Kelly A. Lack, Thomas I. Nygren
Source: Journal of Policy Analysis and Management
Published: October 2013

William Bowen, former president of Princeton, has been studying whether technology can bend the expense curve for higher education and thereby make it more affordable.  He has written widely on this subject and we have selected an early study of his looking at online education at six public institutions.  In the first section of this article Bowen et.al.  present their findings that online students at the institutions learn as efficiently as traditional students.  The section we recommend discusses the possible impact of online learning in cost reduction and what a complete cost analysis would have to account for.  The link below is to the entire article; in the navigation bar on the right, click on “Costs and Potential Benefits” to jump to the most relevant material.

Want to Contribute?

If you have articles or videos that would enrich our discussion of digital education, please send them to ap4de@purdue.edu. Submitted materials will be reviewed before being posted on this site.

Note: Please indicate if you're a student, faculty, staff, or alumni in the subject of the email to ap4de@purdue.edu.

Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, (765) 494-4600

© 2015 Purdue University | An equal access/equal opportunity university | Copyright Complaints | Maintained by Digital Education

Trouble with this page? Disability-related accessibility issue? Please contact Digital Education at ap4de@purdue.edu.