Upgrade took 1 hour. Preparation took 2 months. Lessons learned from a migration.

An Open edX migration across nine major releases: 500GB MongoDB, squashed data migrations, videos in GridFS, charsets, Meilisearch and a one-hour cutover window.

Open edX is a good example of a system of mid to high complexity: front end, back end, relational db, non-relational db, search db, memory db, live platform with thousands of users and gigabytes of content. This is the story and lessons learned of how we migrated one of our latest clients, who was nine releases behind, to our service and upgraded to the latest version.

Context

Our client had a self-hosted, local installation of Open edX, in one virtual server in AWS. They were in Lilac, i.e., nine major releases behind Ulmo, the latest at that time. They wanted to move to the latest version to overcome some technical issues, and we convinced them to also migrate to our SaaS offering, so they shouldn’t be concerned anymore about upgrading.

Open edX versions

Open edX names major releases with tree names in alphabetic order. Lilac is the 12th major release, and at migration time, Ulmo was the latest, the 21st. Lilac was launched by mid 2021, and Ulmo by end of 2025. Since then, Open edX releases two major versions per year, and three or four minor updates in between. Keeping up to date is important for many reasons:

  • Security updates: old versions may have known vulnerabilities that hackers can use to break your site.
  • End of support: Not only does the main platform support only the latest version, but so do its dependencies.
  • Infrastructure obsolescence: Databases also evolve in time. Old versions may become unavailable, or cloud providers may charge extra costs for using obsolete versions.
  • Bug fixes: Bugs are fixed in new versions; you don’t get your code patched without an upgrade
  • New cool features: Last but not least, newer versions incorporate features you will love.

Why upgrading is difficult

We’ve seen many sites that are stuck in old versions. So you might think “why don’t you just upgrade?” There are many reasons that make upgrading your system to the latest version difficult.

  • Fear of spoiling a production setup. Especially when you self-host a deployment, touching the core of your production system can be risky. The fear of leaving an unusable service with hundreds of clients complaining may discourage any system operator from making big changes.
  • Upgrade complexity. Migrating a production site requires planning. Backing up all data, preparing a migration script, having a rollback plan, and defining a service time window are only a few of the mandatory activities that you should consider before migration. And bad things happen during migration. If you don’t have experience migrating sites, you might feel overwhelmed by the implications.
  • Infrastructure upgrades. Usually major releases require upgrading infrastructure components, especially databases. This can be a tricky part and must be done in coordination with the system upgrade.
  • Integrations. If your system is connected to others, like a CRM, payment gateway, ERP, etc, the APIs may change. You have to consider how to adapt integrations to the new versions.
  • Code customizations. Especially when adopting open source software like this, it is tempting to customize it by changing the code base. When you upgrade from the official repositories, all your changes will be lost unless you rebase them on top of the new code. Sometimes it is a simple task, generally it’s a hard task (as it requires some rework and new testing), and in many cases it is impossible without re-writing the whole customization.
  • Data migrations. Data definitions change from version to version. When you have production data you cannot just create the new database: you have to move your data across these changes. Advanced programming frameworks (like Django in Open edX) manage it with a process called migrations. Due to the nature of this process, and the fact that in each major release the data migrations are squashed (compressed), you cannot skip major version upgrades. This means that if you are nine versions behind, you must do nine upgrades to reach the latest.

Self-hosted vs. SaaS

When you self-host your system, you have to take care of everything. Check my article about choosing between self-hosting and SaaS (https://medium.com/@andres_81160/how-to-choose-between-saas-and-self-hosted-lms-078c2798ab31 ).

How I plan upgrades

I believe that good planning is the key to a successful migration. So I prefer spending as much time as necessary planning carefully how the migration will be, instead of failing migration windows or having the site down for days.

This case is especially comprehensive, as it included both moving the self-hosted site to our SaaS, and at the same time upgrading nine major releases. So let’s split the process.

Migration plan

Moving from self-hosting to SaaS means changing everything. Planning carefully is key to a successful migration. These are the main steps to make it.

Planning phase

Without stopping the site, enough time before the migration target date, we do all the preparation steps.

  • Create the new site from scratch, in a temporary domain, in the latest version
  • Move non-relational data. This means dumping the original db, transferring the data, and then loading the target db. This database can be huge.
  • Move relational data. The same, but for the relational database.
  • Move media files (this includes profile images, SCORM packages and student assignment submissions)
  • Run data migrations
  • Adapt customizations and theme
  • Test the site

Each of the steps above exposes challenges. So I take note of all the issues and make sure all are resolved before the cut date. It’s important here to have the customer test the temporary site for all the features that are critical for their business.

Running this process also serves to measure the time it takes in each step, and plan for alternatives. Moving data and running migrations can take time, and it must be measured.

On the migration window, we just:

  • Stop the original site, so there are no users modifying the databases until we finish the move. Some down time is expected here because we are moving infrastructure. A simple upgrade without infra moving can be done without downtime. If you wonder if it’s possible to do the move without downtime, the answer is yes, but it is much more complex and is justified only in very mission-critical sites, which is out of the scope here.
  • Move relational and non-relational data again (as it might have changed from the planning phase)
  • Move media files again if they changed
  • Fix data issues identified in the planning phase
  • Run all intermediate data migrations (except the last)
  • Initialize the last version (which implies the last data migration)
  • Point the DNS to the new infrastructure

If the planning phase was carefully followed, then the migration should take no more than one or two hours. Of course things happen, but when the critical issues are managed, any unexpected issue can be easily handled during the migration window.

Upgrade

Upgrade is part of the migration plan. But there are many things to keep in mind.

Data migrations

Non-relational data is not a problem. As these types of databases do not have a fixed schema (table definitions, data types, etc.) MongoDB can survive upgrades without structural changes. So you don’t have to worry about it. The problem is MySQL.

The data migrations process handles the changes in the relational database structure in a way that it can be done with a live database without losing data stored. This includes adding or deleting tables, adding or deleting columns, or changing data types. These migrations are stored in files with complex SQL instructions, which take a long time to execute. They accumulate over time with each release.

As each major release can have hundreds of migration files, after a few releases it can take too long to run them all. Sometimes a release creates a table that will be deleted a few releases after, so many changes are even unnecessary. That’s why developers squash migrations in some major releases. This means that when you install Ulmo (the 21st release) for the first time, you only run the latest data migrations, and not all the migration history since the first release.

Installing from scratch is ok. Upgrading one major release is ok. But you cannot skip a release upgrade, as the migrations squashed in the middle release will be lost and it will break the migration sequence. That’s why you have to migrate major releases one by one.

The challenge is that you would have to set up intermediate versions that you would never use after the migration. And there are some old versions that are just impossible to set up, mostly because the image will not compile as some dependencies may not be available anymore.

How I solved it

It’s not necessary to launch a full working Open edX to run migrations. E.g., you don’t need the front end components, or background tasks running. You just need a minimal image to run the migration command for two services: LMS and CMS. So I created a Docker image for each release, from Koa to the latest, and fixed all dependency issues so I can just run the migration process.

So the process is quite simple:

  • Upload the data to the database (only MySQL)
  • Launch the next version docker container (if the customer was in Lilac, I run Maple)
  • Connect it to the db
  • Run the migration command for the LMS and the CMS
  • Stop the container
  • Repeat with the next version

MySQL version

In Palm, Open edX moved from MySQL 5.7 to 8. It sounds overwhelming, but it is not. If the target site is already in MySQL 8, the dump from 5.7 will work without problems.

Character sets and data collation

A small caveat: data collation and charset. Character set defines how each character will be represented internally inside the database. In Redwood, Open edX moved from utf8mb3 to utf8mb4. The “mb” means multibyte, and indicates how many bytes are used to represent a character. “utf8mb4”, with four bytes per character, allows special characters like emojis inside data fields.

Collation is how characters are ordered when sorting data (e.g., we might want letters sorted like a, A, b, B, c, C instead of a, b, c, A, B, C). It doesn’t sound like a big deal, but it must be aligned with the character set. A different character set or collation in linked fields across tables can break a data migration.

The solution is to update the charset and collation as a first step. Fortunately you can safely migrate the charset upwards (mb3 to mb4) without losing data, so it’s safe and recommended to do it first.

Do not underestimate this step. Updating all tables' charset can be the most time-consuming task in the migration process.

MongoDB: the elephant in the elevator

In Open edX MongoDB stores the course structures and content, and it can grow large in big sites.

When I started planning for the migration and asked the client to send me the dump of their data, they took days to do it. When I received the dump, I knew why: the MongoDB was over 500GB in size. Yes, more than half a terabyte.

Ok, this can be normal if there are many courses, and there were more than one thousand of them. But once I had the data, I checked why it was so large, and the answer came immediately: videos.

What you are not told about file storage

Every course in Open edX has a section called File uploads. You can basically upload any file and it gives you a link you can use inside the course structure for whatever you want.

Any site operator would assume that these files go to a typical storage in a file system or something similar. They do not. They are stored in MongoDB GridFS as binary data in a collection called fs.chunks.

This approach has the benefit that course files move with the course when you export and import, or when you re-run course editions. But it has some drawbacks.

  • When you use the absolute link to the file, it will point to the original course version, not the imported or re-run. If you change the file in the original course, you may lose access in the new one.
  • The file gets duplicated in every import or re-run.
  • Viewing or downloading the file implies loading the database.
  • It can make the MongoDB huge, which impacts backups and data movements.

So I recommend using file uploads for small files, or images to include as part of the content. For big files or reusable content, it is better to have a dedicated storage outside the platform, and insert the links.

And definitely not for videos. Videos are big files and require high throughput. MongoDB is not designed to serve videos.

So where to host course videos?

It is surprising that, given that video is a core component of MOOCs, there is no definitive guide to store and manage videos. There are two main strategies for storing and serving videos: as a file object, or streaming. When you store videos as a file, you can store them in a regular storage like S3 and optionally set up a CDN in front. Streaming is much better, as it can offer dedicated features like adaptive quality.

Video deserves a dedicated analysis. So to keep it simple:

  • Never use the Open edX files feature to store videos
  • Use a video service like YouTube, Vimeo or VdoCipher to stream videos
  • Or store them in S3 or similar storage with an optional CDN in front.

How we fixed it

This is what took up most of the two months of preparation. We made a joint effort in which the client created a script to extract the video links of all courses, removed duplicates, and uploaded all videos to the video streaming platform. When done, they gave me the list of videos with the new URL.

Then I created a script to modify the video URL in each course, and delete the video from the MongoDB.

Yet we still had to do a lot of manual work. There were many duplicated videos, orphaned blocks, and unused files across many courses. All of them had to be worked out manually.

After all this work, not only is the database much smaller, but the videos also display much better.

Forum

In the Sumac release, the Forum application was (thankfully) fully refactored from Ruby + MongoDB to Python + MySQL.

There is a process to migrate the data from MongoDB to MySQL. The challenge is that after Sumac, the settings to connect that process to the old data are no longer present and they have to be patched manually.

As a bonus, here is a small plugin that lets you run the migration command after Sumac:

from tutor import hooks as tutor_hooks
tutor_hooks.Filters.ENV_PATCHES.add_item(
  (
      "openedx-common-settings",  """
FORUM_MONGODB_DATABASE = 'cs_comments_services'
"""
  )
)

Meilisearch

Starting from the Teak release, Open edX replaced ElasticSearch with Meilisearch. The migration is quite straightforward and is managed by the initialization step.

The challenge that we faced was that our Meilisearch server was a small one. It was big enough for production, but got throttled when it had to reindex more than one thousand courses. As the reindex process is part of the initialization pipeline, it got stuck. To work around this in the maintenance window, I had to manually initialize the plugins and later resume reindexing the courses in a larger Meilisearch instance.

E-commerce integration

The customer has an e-commerce site that used custom code inside Open edX to get additional user information to process the order. We reviewed the process and helped them refactor the integration to use standard endpoints to get the information they needed without customizing the Open edX codebase.

PDF certificates

Surprisingly, PDF certificate generation code was completely deprecated in 2021, despite the preference of many institutions to issue credentials in this format.

Our client had developed a custom PDF certificate generation, which relied on a custom component inside Open edX to trigger the certificate issuance.

We recommended the use of webfilters instead, and they were able to refactor the microservice they had to keep the functionality.

Webfilters can be enabled using our tutor-contrib-webhooksplugin, which is open and free to use. This way we removed the last code customization from the platform.

Theme

Thanks to our Branding application (https://branding.aulasneo.com ), the customer can apply their own styles (logos, fonts, custom CSS) by themselves without having to ask for support. The changes can be applied immediately without needing to rebuild Docker images or restarting the service.

Branding is free to use in any Open edX site, self-hosted or hosted by any provider.

As a small bonus, if you want to change the legend of the login screen, here is a tutor plugin that achieves it easily (you will need to rebuild the MFE image after enabling it):

version: 1.0.0
name: change_authn_legend
patches:
  mfe-dockerfile-pre-npm-build-authn: |
    RUN sed -i \
      -e "s/defaultMessage: 'Start learning'/defaultMessage: '<new first line>'/" \
      -e "s/defaultMessage: 'with {siteName}'/defaultMessage: '<new second line>'/" \
      src/base-container/components/default-layout/messages.js

Another tip: if you want to hide the “Help” link in the learner dashboard, add this plugin that leverages the MFE slots:

version: 1.0.0
name: hide_learning_help
patches:
  mfe-env-config-runtime-definitions: |
    addPlugins(config, 'org.openedx.frontend.layout.header_learning_help.v1', [{
      op: PLUGIN_OPERATIONS.Hide,
      widgetId: 'default_contents',
    }]);

SCORM

SCORM is an old technology, yet widely used in learning environments. It relies heavily on JavaScript and iframes, which offer many challenges, especially when implemented in scalable infrastructures with dedicated object storage.

Thanks to our Tutor plugin for SCORM in S3, which is open and free to use, this is not a problem. They even fixed a long-standing issue they had with some SCORM modules that didn’t work in the old setup and now they do in the new one.

Conclusion

Thanks to the careful planning and the experience migrating and supporting Open edX, our client was able to migrate and upgrade to the latest version with minimal down time. The new deployment offers a number of benefits.

They now have a fully managed Open edX as a service with dedicated expert support. The platform is always in the latest stable version, with security patches applied, cool new features enabled, and supported components. Videos now don’t overload the database and offer adaptive streaming without delays or glitches. The codebase is kept as the official standard: no custom code to rebase on future upgrades. We have optimized the setup based on best practices from our ten years of experience. The deployment is on a robust, resilient and scalable infrastructure. Instead of a monolithic site, the application runs on Kubernetes and is connected to redundant databases. All infrastructure costs are included in the service fee.