Coping with the Complexity of Learning Analysis
Educators have a lot of data at their disposal: student grades, demographics, communications, course surveys, and learning analytics. And while all this information can seem overwhelming at times, careful instructional analysis can yield great results.
To improve student outcomes and retention, several of our design faculty were recruited for a data-driven review and diagnosis of courses in which students were struggling. The course team needed to
- Gather relevant course data.
- Analyze the data to determine why students were struggling.
- Propose improvements for student success, including advanced design and teaching methods.
While the overall process was straightforward, analyzing the data presented a real challenge. Our team had to process an enormous amount of information: program assessments, grades, faculty observations, course evaluations, student interactions, student submissions, LMS captured data, learning support data from the Student Learning Center or the Library, etc. Some data points were only loosely related, and some contradicted each other.
As Reeves & Herdberg (2003) argued, “decision informed by sound evaluation are better than those based on habit, ignorance, intuition, prejudice, or guesswork”. The purpose of collecting information is to support instructional decision making. Depending on the specific goal, we can organize data in a more meaningful and effective way. As the result of our project, I came up with a learning analysis model which has the following three dimensions:
- Analytic-Quantitative: This dimension, consisting of quantitative data, aims to explore what works by testing our hypotheses. For example, in a particular course, we would examine data to see whether face-to-face and online students perform differently. We collect data such as course grades, assignment or test scores, which indicate to what extent students are learning, and data on learning analytics, which are captured in our learning management system (LMS), such as access frequency and type of the activities.
- Interpretive-Qualitative: This dimension, consisting of qualitative data, explores phenomena related to instruction. For example, how do students react to the content, activity, and interactions designed for in a particular course? We collect this kind of data from students’ course evaluation surveys, instructor interviews, and students’ reflection papers.
- Connoisseur-Appraisive: Last but not least, the third dimension focuses on the unknown and creative aspects of course designs and implementations, or as Verganti suggested, the design-driven innovations (2009). We aim to discover how students handle learning and how their motivation, cognition, and social interactions could and should evolve in the course. Eisner argued the critical importance to “avoid reductionistic thinking that impoverishes our view of what is possible”. Following the Eisner model, we emphasize the creative and critical aspects of teaching and learning. This dimension is often obtained through expert reviews. For example, instructional design experts examine both the instructional alignments among objectives, instruction, and assessment and creative or lack of creative components in a course.
Table 1. A three-pronged learning analysis model
|Analysis goals||Analysis methods||Data collection techniques|
|Determining whether the course elements work or not||Experimental.
Correlational using descriptive and inferential statistics
grades, learning analytics, etc.
|Portraying how the course works or not||Interviews, survey, documents||Course evaluation, reflection paper, instructor interview|
|Discovering creative or critical elements||Educational criticism and connoisseurship||Expert review, panel discussion|
I believe that the three-dimensional model facilitates a holistic approach for instructional analysis. The three dimensions support and are complementary to each other. Findings from the first dimensions guide the investigation of the second dimension. Analyses from the second dimension shed light on inquiries in the third dimension.
This course diagnostic experience reminds me an age-old question, design: Art or Science? If the first two dimensions, i.e., analytic-quantitative and interpretive-qualitative, are considered more science-based, then the connoisseur-appraisive dimension is more art-based. The latter is more dependent on whether the reviewer can imagine the learning experience holistically and dynamically and whether the reviewer can generate a solution creatively. So, I feel like my post and the model proposed above have opened a debate between science and art. Where do you stand: science or art? How can the science dimension and art dimension of learning analysis support and validate each other throughout the analysis and design processes?
Reeves, T. C. & Hedberg, J. G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications.