Assurance of Learning (AoL) Through Business Simulations

Published on 31 May 2022

"I wanted a simulation; I wanted active learning. Today students want a multi-media, interactive experience; they don't want to be taught the way we were taught." Dr Joel Rudin

On Wednesday, May 11, we held a webinar together with guest speaker, Professor and Assistant Chair, Dr Joel Rudin from Rowan University. Hosted by Edumundo's own, Leon Lloyd, we discussed the importance of providing Assurance of Learning (AoL) in US Higher Education.

In this blog article, we provide a recap of the key highlights and insights presented and discussed during the webinar including examples of participants' questions at the end.

What is Assurance of Learning (aoL)?

"'Assurance of learning' means different things in different contexts but above all it's a data-based process that demonstrates learning improvement."

Assurance of Learning, defined by Joel in the above quotation, is a term used across many accrediting agencies and forms a key criterion in how they accredit institutions.

Agencies such as AACSB want to see institutions' Assurance of Learning plans, what data they've obtained. and how they've used that data to demonstrate improvement. Joel explained one part of their accreditation process:

"Once every five years they send a team of three people, these are Deans of Business from other universities to review the evidence of learning acquisition in their school's programs."

The process emphasises the collection of data at the core of AoL and typically follows three steps which Joel outlined on his first slide:

"They don't expect perfection and 100% grades. They want to see proof of improvement. When you are proving improvement [in that particular subject] they then want you to focus on another subject to focus on improving that."

Proving Assurance of Learning is typically done via the collection of quantitative data over the course of a learning exercise. How and what data is collected requires some experimentation with various tests and formats.

Where does the Data Come from?

"We teach in a variety of ways so why not assess in a way that's aligned to that style of teaching."

Joel explained that data to prove learning improvement has evolved throughout the years, starting initially with 'Nationally standardized tests' before moving on to 'multiple-choice tests in capstone courses' and then currently using a variety of different tests across the same variety of courses.

"About 20 years or so ago, there was a preference from agencies such as AACSB for institutions to use 'Nationally Standardized Tests' but there was some quality issues here. They were very expensive and not very effective. We then moved to a commonly used 'multiple-choice tests' for capstone courses. These are better because their faculty-driven, adapted to the particular courses students have studied previously. They are more tailored but the problem is they're harder to achieve improvement. We then moved on to a variety of different assessments: we teach in a variety of ways so why not assess in a way that's aligned to that style of teaching."

Once Joel's faculty had agreed upon the use of different tests tailored to individual courses, Joel began looking at ways in which business simulations could provide this 'loop', demonstrating improvement in students' learning and knowledge acquisition through quantitative data.

Business Simulations and Assurance of Learning - Dr Joel Rudin's Approach

"The nice thing about business simulations for Assurance of Learning is that simulation providers keep a record of what happens during the exercise: not only who finished first and who finished last but what decisions they made. I can compare and contrast which teams made the optimal decision to improve their company [in the simulation]."

Joel explained two approaches through the use of simulations to prove learner improvement:

Decisions - what percentage of individuals/teams made "good" decisions for their business?

This way data is collected and measured to demonstrate improvement based on the level of "good", informed decisions made. This is based on students' (in their teams) use of applying strategic models, concepts, and market research in their rationale for what decisions they made during each round of the simulation.
Results - what percentage of individuals/teams made decisions that yielded good results?

This way data is collected and measured based on the amount of points awarded in the simulation exercise. Points are awarded for increasing their stock market price with their overall points being outlined in the simulation's balanced scorecard. Results-based is currently the approach Joel is using with the ProSim Advanced simulation.

"They [Accrediting Agencies] want some successes but primarily they want to see measurable efforts to prove assurance of learning; they're not after perfection."

Used on Joel's MBA class 'Operations Analytics' the simulation was rolled out in-class across ten groups of students, taking part in a total of seven rounds of the simulation. Based on the weekly insight reports, provided Edumundo's Operations Team, Joel was able to identify after each week what the results were on the simulation's stock market and balanced scorecard.

After the simulation exercise, Joel collected a total of 70 results of which 57 (81%) showed an increase in stock market value with the remaining 13 (19%) showing a decrease in stock market value. This data is then used to prove Assurance of Learning during accreditation exercises and provides a benchmark for future simulation exercises to monitor improvement increases or decreases.

The Q&A Session

After the discussion on Assurance of Learning, the webinar culminated with a Q&A session where participants were able to ask questions to Joel and Leon. Here were a few examples of the questions asked:

"How do you overcome the barriers for learners to work in teams?"
Joel: "I assign them to groups. My experience in the working environment is that I never had the choice of who I worked with so I randomly assign students to groups to provide a more realistic experience."
"How did you state/measure the goal in your AoL report?"
Joel: "The AoL report is not due for a couple of months. However, it's going to be based on teaching students to make effective strategic decision making."
"Are the algorithms deciding on the value of market research open and available or can students intuitively figure them out?"
Leon: "On market research specifically, it's based on the weighted investment students think is right for their business. However they don't know how much value that research is until they make the investment."

A big thank you to all our participants and Dr Joel Rudin for being our guest speaker. The session illuminated key aspects of Assurance of Learning and how it's interpreted by key Accrediting Agencies such as AACSB.

If you are interested in finding out more about our business simulations, click on the button below otherwise sign-up for updates on future webinars further down!