Open edX Analytics: Evaluating the Impact of Open Courses
Measuring the impact of online education goes far beyond counting enrollments and completion rates. In this presentation at MIT, Esteban Etcheverry — co-founder of Aulasneo — shares a framework for evaluating the real impact of open courses using Open edX analytics and Panorama. The approach combines quantitative and qualitative data to paint a complete picture of educational outcomes.
The traditional approach: quantitative data only
Traditionally, open courses — especially those offered by NGOs or government agencies — are evaluated through quantitative metrics: enrollment numbers, completion rates, and quiz scores. These numbers provide a snapshot but don't reveal the full story. They lack the depth needed to truly understand the educational experience and its broader impact on learners' lives.
Aulasneo's approach: merging quantitative and qualitative analysis
Leveraging Panorama, our flexible Open edX analytics engine, Aulasneo has pioneered a method that combines hard data with personal stories and feedback at scale. By doing so, we achieve a richer and more nuanced picture of educational outcomes.
Our method uses a three-layered framework:
1. Setting clear goals
Understand your audience, decide what you want to measure, and define how you'll measure it. Clear objectives shape the direction of all data collection and analysis.
2. Implementing three surveys
We deploy a structured set of three surveys at different moments:
- Initial student profile survey: taken upon course entry to understand the background and expectations of learners before they start.
- End-of-course survey: taken upon completion to capture immediate reactions, feedback, and perceived learning outcomes.
- Follow-up survey: taken between one month and one year after completion to gauge long-term impact and how learners applied their new knowledge in real-world scenarios.
3. Integrating advanced forms with Open edX analytics
To capture rich qualitative data, we integrate LimeSurvey — an open-source forms engine — with Open edX platform. These forms include open-ended questions and connect directly to Panorama for comprehensive analysis.
A real-world application: Cap-Net UNDP
In collaboration with Cap-Net UNDP, a global agency delivering open courses on water management, we put this approach into action:
The initial survey is positioned as a unit in the first section of the course, ensuring learners provide profile data right after registration. The end-of-course survey captures both qualitative and quantitative feedback immediately after completion. The follow-up survey invites students to share their journey through open-ended questions — revealing how knowledge translated into real-world action.
Open edX analytics at scale
All collected data feeds into custom Panorama dashboards: a Student Profile Dashboard, an End-of-Course Dashboard, and a 12-Month Post-Course Dashboard. Together, these give course teams and organizational leaders a 360-degree view of educational impact — from who their learners are, to what they learned, to how it changed their professional practice.
Want to measure the real impact of your Open edX courses?
Explore Panorama Analytics