A veteran who led a maintenance crew. A medical assistant with her GRE. A retail worker returning for her degree after an early setback. Each one brings real, hard-won skills to college, and traditional assessments rarely capture them. WGU Labs is working to change that. Next week at AIME-Con, we'll share how.
From here to there: Our AIME-Con preview
WGU Labs had three presentations accepted to this year's AI in Measurement and Education Conference (AIME-Con) in Pittsburgh. The conference convenes leaders across psychometrics, natural language processing, education, and learning analytics — Labs researchers Megan Imundo and Kjorte Harra and learning designer Lesley Reilly among them.
If you're not going to be there, this post is your preview.
Our presentations detail different aspects of Current Skills Validation (CSV), a WGU Labs system that meets students where they are in a way that mainstream assessments often don't: it validates the skills students pick up through both formal education and non-instructional settings. Think military service, caregiving, work experience, or even self-directed learning.
Because once we meet students “here,” with a thorough understanding of the skills they already have, we can better help them get “there,” developing the skills they need to reach their academic and professional goals.
A team of AI agents, each with one assessment goal
Megan Imundo | "A multi-agent architecture for valid, reliable, and scalable skills assessment" | Tuesday, October 6, 11 a.m. ET
Mainstream assessment exists in a fixed, calibrated form; CSV challenges that with a dynamic, multi-agent architecture that builds on adaptive technologies and harnesses the latest capabilities of AI. As Imundo writes, “LLMs alter the economics of assessment.”
Within the CSV system, each agent focuses on a different aspect of validation, allowing for real-time interplay as it adjusts to each student’s needs and proficiencies. This kind of braided collaboration means that no two students move through CSV in the same way. The multi-faceted architecture is also key to iterating: the team can test and tweak each agent separately.
Imundo’s presentation and accompanying paper explain the three-pronged architecture in detail. Together, the agents in skills framework, generated assessment, and evaluation of learner responses generate five types of questions grounded in learning science principles. For example, multiple choice questions are grounded in the Dreyfus model of skill acquisition; short answer items in the transfer of learning; open-ended prompts, cognitive load theory; reflection prompts, metacognition; role-play, situated cognition. (Reilly, for her part, has been wrestling with how to design effective AI role-plays for over a year now. She wrote up that process here.)
Borrowing quality control from the factory floor
Kjorte Harra | "Off the assembly line: Applying manufacturing benchmarks to real-time assessment generation" | Wednesday, October 7, 11 a.m. ET
Harra pulled from an unexpected source — the manufacturing industry — to create a new benchmarking method. She’ll explain how her team borrowed and adapted acceptance-sampling logic and applied it across CSV experiments with the WGU Student Insights Council (SIC).
When the system says 'you're ready'
Lesley Reilly | "Is this a test? Learner experience and control of the formative-summative shift" | Wednesday, October 7, 3:45 p.m. ET
Then Reilly is slated to talk about covert vs. overt shifts from formative to summative assessment. Her team did testing and interviews to see what happens to students’ senses of autonomy and control when the system moves them from one assessment mode to the other. The team talked with SIC students about how much they trusted the results, and whether the CSV system gave feedback that aligned with their experiences.
All three AIME-Con presentations reflect the iteration and benchmarking process currently underway with CSV. You can read even more about that process in Test-taking Savvy Isn’t Knowledge.
To follow CSV as it develops, or if you’re interested in supporting or partnering this work, contact us at info@wgulabs.org.
