Bootstrap Migration Guide

Overview

Bootstrap is a third-party UI library used throughout HQ. It's essentially a set of stylesheets and javascript widgets. This means that migrating from one version to the next involves the following kinds of work:

• Styling updates: This can be as simple as replacing a renamed CSS class, which is often handled automatically by our migration tool. It can also mean updating HQ stylesheets to work harmoniously with Bootstrap's new/updated styles.
• HTML updates: Some of the new/updated Bootstrap styles correspond to HTML changes. This affects simple widgets like dropdown menus, which may now need classes to be applied differently - for example, the children of dropdown menus now need to use a new dropdown-item class. Our migration tool detects widgets that need to be updated, although a developer is needed to implement the changes. The tool can automatically make other HTML-related changes, such as renaming attributes like data-toggle that have been renamed in the new version.
• JavaScript updates: HQ uses a number of Bootstrap-provided interactive widgets, such as modal dialogs. The APIs for accessing these widgets have changed, so the calling code needs to be updated. There are also a few non-Bootstrap widgets that depend on a specific version of Bootstrap and need to be migrated to a new version. Our migration tool detects usage of widgets that need to be updated, but a developer is needed to make the changes.

Because the HQ codebase is large, we are taking an incremental approach to this migration. For organizational purposes, we are executing and tracking this migration primarily on an app-by-app basis.

Multiple versions of Bootstrap cannot coexist on the same page. The @use_bootstrap5 decorator marks individual views as migrated to Bootstrap 5. This decorator sets a use_bootstrap5 attribute on the request that's used to set up the relevant Bootstrap environment.

This means that files that are required by both migrated and unmigrated pages need to be "split" into two versions: one to include on Bootstrap 3 pages and another to include on Bootstrap 5 pages. This applies largely to code that is fundamental to HQ's UI and used throughout the platform, such as the hqwebapp app, the analytics JavaScript libraries, etc.

It can also be helpful to "split" files in this way while migrating an app. This allows Bootstrap 5 changes to be developed, reviewed, and merged in small pieces, before switching the Bootstrap version used in production. It is a more complex process that adds more code during the migration, although most of this code is automatically generated by the migration tool. In these cases, once the app is fully migrated, the Bootstrap 3 files can be deleted.

To support this approach, we have tooling that facilitates "splitting" and "unsplitting" files: copying files, moving them, and updating references to them. This tooling also performs the simpler migration updates, such as renaming CSS classes. For updates that it cannot handle automatically, it adds a TODO comment to the relevant code.

Once the entire platform is fully migrated to Bootstrap 5, any remaining Bootstrap 3 code can be removed: the Bootstrap 3 versions of files in hqwebapp and similar apps, the @use_bootstrap5 decorator, any code related to request.use_bootstrap5 or window.USE_BOOTSTRAP_5, the Bootstrap 3 stylesheets and JavaScript library, and so forth.

Step 0: Prepare Local Environment

If you haven't done so, you will need to install Dart Sass to compile scss files. The recommended method is with homebrew brew install sass/sass/sass, as described in DEV_SETUP.md.

Step 1: Announce Migration

This spreadsheet is a high-level tracker of which apps have been migrated. It contains notes on the approximate size of different apps (in terms of how many views they contain) and on inter-app dependencies.

The migration tool supports two general workflows for migrating an app: one based on splitting all files into Bootstrap 3 and Bootstrap 5 versions, and one based on migrating files "in place." The splitting files approach is necessary for apps that are dependencies for other apps, which will need to support both Bootstrap 3 and Bootstrap 5 for a while. It is optional for small, independent apps.

Before you begin the migration of an app, please announce your intention to do so in #gtd-dev. It is helpful to be aware of other front-end development in the app. If the app has ongoing front-end development, consider delaying either that development or the migration, to reduce code conflicts. After announcing your intent to migrate the app, please update the app's migration status.

Step 2 (small apps): The "No Split" Process

For small apps that are not depended on by other apps, the migration tool supports a streamlined process that does not require splitting files into a Bootstrap 3 and Bootstrap 5 version. This is a good option for apps with a small number of views, as it involves fewer pull requests and avoids the overhead of splitting files. This approach does the entire migration in a single pull request, which makes it unlikely to be a good choice for apps with 10+ views.

To begin a "no split" migration, first lint the app's JavaScript and make a pull request for any linting changes. Then, to begin the actual migration, run:

./manage.py migrate_app_to_bootstrap5 <app_name> --no-split

This will iterate through all the templates and javascript files in <app_name>, applying all automated fixes and adding TODO comments for fixes that aren't automated. After staging the changes to each file, you will have the option to auto-commit those changes.

For each migrated template, the command will wait for you to apply the @use_bootstrap5 decorator to the relevant views (see Migrating Views) before moving on to the next template.

Once all the files are migrated in-place, go through the TODO comments, which reference helper documents in the changes_guide directory.

After all TODOs are addressed, test your work: load the page and test out any interactivity. Most migrations should also go through QA. Migrations can skip QA if they make relatively superficial changes and are in low-risk areas.

Also mark the migration as "complete" as part of your pull request. This updates a status file where we programmatically track the status of the overall platform migration. To do so:

./manage.py complete_bootstrap5_migration <app_name>

You're done! Ignore the rest of this document.

Step 2 (large apps): Auto-Migrate & Split Files

For large migrations, the first step is to split affected template and javascript files into bootstrap3 and bootstrap5 versions. This is an automated process that uses a management command to find and replace the straightforward changes, while flagging more complex changes to be addressed later.

Once the migration changes are staged, the file and split into the two bootstrap versions, with the migration changes saved to the bootstrap5 version. The two files are then saved to separate bootstrap3 and bootstrap5 directories under the original parent directory.

The command will then find references to the original filepath and replace them with the filepath to the bootstrap3 version.

Pre-Work: Lint JavaScript

Pre-work: Your life will be easier if you lint the app's JavaScript before the migration script duplicates every js file. This is usually not arduous. To find lint errors, make sure you have eslint installed locally and then run:

npx eslint corehq/apps/<app_name>

If you can, please cherry-pick these lint commits into a separate PR and merge those changes before creating the PR to split the files.

Step 2.1A: Migrating Large Apps

Please set aside focused time to PR these changes as soon as possible to avoid any migration conflicts or missed renamed references, especially in parts of the codebase that are undergoing frequent front-end changes. This initial "split file" Pull Request should only contain changes automatically made by the management command. Additional changes should be made in subsequent PRs. Here is an example Pull Request of this step.

Use the management command below with <app_name> being the name of the application you would like to migrate:

./manage.py migrate_app_to_bootstrap5 <app_name> --skip-all

For instance, if you want to migrate corehq.apps.reminders, you would use reminders as <app_name>.

The option --skip-all is optional, but recommended unless you have a particularly tricky app as it speeds up the migration process by auto-committing the split files changes for each template and javascript file in the app. If you use --skip-all, you are still responsible for changes made by the tool, so it's a good idea to review the auto-generated commits in GitHub or your favorite diff tool.

If --skip-all is not used, the management command will iterate through all template and javascript files in the app. You have the option to review and accept each change line-by-line or review the diff for each file at the end.

Regardless of what route you take, code will also be updated with TODO comments noting areas that need attention and could not automatically be updated. These comments reference helper documents in the changes_guide directory.

Step 2.2: Configure & Update Diffs

Since a large app migration usually takes several weeks, we need to ensure that any changes made to bootstrap3 templates during this time are kept in sync with their bootstrap5 counterparts. We do this by saving diffs of the split files and running tests against them to ensure diffs stay the same. Once changes are made to both split files, the diffs can be regenerated so that tests continue to pass.

As migrate_app_to_bootstrap5 iterates through each template and javascript directory, it will create new bootstrap3 and bootstrap5 directories. For instance app_manager/base_template.html now becomes app_manager/bootstrap3/base_template.html and app_manager/bootstrap5/base_template.html.

When new split directories are created, please ensure the paths to these directories are added to bootstrap5_diff_config.json and commit those changes.

For most applications, you can just run the following command to automatically update bootstrap5_diff_config.json:

./manage.py build_bootstrap5_diffs --update_app <app_name>

Once those changes (if any) are committed, please run the following management command and commit the results:

./manage.py build_bootstrap5_diffs

Step 2.3: Verify References

Right before submitting the "split files" Pull Request, it is important to rebase the latest master branch into your working branch and run the following command:

./manage.py migrate_app_to_bootstrap5 <app_name> --verify-references

This will iterate through all the split bootstrap3 files in the app and ensure that references to the old file (without bootstrap3/ in the filepath) are updated to the bootstrap3 path. The reason why we do this right before making the pull request is so that any changes merged into master since the split files process began are checked for references so that we can avoid any 404s or 500s.

After verifying references, it might be necessary to run build_bootstrap5_diffs again before finally opening the Pull Request.

Step 2.4: Re-Migrate Files

There is a chance that during the time your split-files PR was open, someone might have added a new template or javascript file, or made a change to one of the bootstrap3 versions of an already split file. If you merge master into your split files branch and find that the diffs have diverged for a file, please run the following command to re-migrate the bootstrap3 version of that file.

./manage.py migrate_app_to_bootstrap5 <app_name> --filename <file_path>

In this usage, <file_path> can either be the file name of the diverged file or the relative path. For example, if builds/bootstrap3/edit_menu.html diverged, you can specify edit_menu.html or builds/bootstrap3/edit_menu.html as the <file_path>.

This command will then re-apply the migration changes to the bootstrap3 file and then save those changes to the bootstrap5 version. You can then commit those changes and then run build_bootstrap5_diffs to rebuild the diffs.

Step 2B: Migrate Stylesheets

Some larger applications, like the App Builder and Web Apps, have their own stylesheets that add or modify styles on top of the main stylesheet. Since Bootstrap 5 has moved to using SASS / SCSS instead of LESS, less files that are part of the app undergoing a migration will need to be migrated from less to scss.

The process for this step in the migration is not automated due to the rarity and complexity of this requirement. You can begin this process by creating parallel scss files for all the existing less files. The contents can be copied over from less to scss.

The syntax will need to be updated to use SCSS style variables and mixins. Here is a guide to help with those changes. Changes that are common in HQ code include:

• Variables and string interpolation

  // LESS
  @thing: 10px;
  .@{my-prefix}-kind-of-text: 10px;
  height: calc(~"100vh - 65px");
  
  // SCSS
  $thing: 10px;
  .#{$my-prefix}-kind-of-text: 10px;
  height: calc(100vh - 65px);  // tildes typically weren't necessary

• Mixins

  // LESS
  .my-mixin(@param1, @param2) { ... }
  .my-mixin(170px, 115px);
  
  // SCSS
  @mixin my-mixin { ... }
  @include my-mixin(170px, 115px);

• Removing bootstrap 3 browser support mixins, including .transition, .box-shadow, and border radius mixins.

  // LESS
  .box-shadow(0 0 10px 0 rgba(0, 0, 0, 0.2));
  
  // SCSS
  box-shadow: 0 0 10px 0 rgba(0, 0, 0, 0.2);

• Updating responsive styling that depends on screen widths:

  // LESS
  @media(min-width: @screen-xs) { ... }
  @media(max-width: $screen-sm-max) { ... }
  
  // SCSS
  @include media-breakpoint-up(sm) { ... }
  @include media-breakpoint-down(lg) { ... }

  Note that the screen size categories have changed from bootstrap 3 to 5, so see the old docs and the new docs, and test your changes. Also note that using the B5 mixins requires imports:

  @import "mixins";
  @import "functions";

• Updating imports. Look at how imports are done in other parts of HQ that use SCSS, such as hqwebapp and cloudcare. Imports may not be one-for-one replacements, since B3 and B5 are organized differently, and you may have dropped some B3 styles or added some B5 styles.
• Remember to update the <link> elements in HTML, including changing the type from text/less to text/scss.

It is important to update any variables from the old Bootstrap 3 stylesheet to their Bootstrap 5 counterparts. Please work closely with the lead developer of the Bootstrap migration, so you can receive guidance to do this part accurately.

You will have to update bootstrap5_diff_config.json manually when working with stylesheet files. To confirm which directories to add, check your app for directories with "bootstrap3" in the name:

find corehq/apps/<app_name> -type d | grep bootstrap3

You can then see existing stylesheet file configuration in the config file as an example for how to update it for your stylesheets.

Step 3: Migrating Views

The next step is to migrate the app's views one-by-one. It is recommended to make a pull request for each view or group of small related views, rather than migrating a lot of views and having a very large Pull Request.

The migration process begins by either applying @use_bootstrap5 decorator the view function or @method_decorator(use_bootstrap5, name='dispatch') to the view class. Also update the template's name in the view from bootstrap3/foo.html to bootstrap5/foo.html.

The automation should have updated the view template so that it inherits from bootstrap5 base templates. It should also have updated any bootstrap3 javascript dependencies to their bootstrap5 versions.

The automation will have added comments to the template and javascript files noting what changes need to be made. Each comment will be marked with todo B5 and the type of change that needs to be made. Each type of change has a corresponding document in the changes_guide directory explaining what needs to be done.

Once a view's template and javascript files are updated, it is now time to load that view and test for any javascript or visual errors. The view should be as similar as possible to its Bootstrap 3 version, which you can compare with on production.

A pixel-perfect migration from Bootstrap 3 to Bootstrap 5 will likely not be possible for each page, as Bootstrap 5 has removed several components (like panel, for instance) and replaced them with new components (like cards). The migration helper migrate_app_to_bootstrap5 should help with guidance for making more complex replacements of removed components.

Un-Split View Files

Once your view is functional under the Bootstrap 5 changes, check to see whether the boostrap3 versions of your template or javascript files are being referenced elsewhere. If not, it is time to "un-split" that template by removing the bootstrap3 version of that file and moving the bootstrap5 version into the file's original location before the split.

If the file you are un-splitting is a template, run:

./manage.py complete_bootstrap5_migration <app_name> --template <filename>

If the file you are un-splitting is a javascript file, run:

./manage.py complete_bootstrap5_migration <app_name> --javascript <filename>

Please follow the guidance of the complete_bootstrap5_migration management commands for suggested commit messages and when to commit changes.

Migrating Report Views

If you are migrating a Report, you may realize there is no dispatch method to apply the @use_bootstrap5 decorator to. That's because reports are rendered by a ReportDispatcher.

Migrating a Report View will be different from other Views:

1. Set the use_bootstrap5 class variable to True

For example, if we want to migrate the SubmitHistory report, we would do the following:

class SubmitHistory(...):
    ...
    use_bootstrap5 = True
    ...

2. Use the report debugging tool

class SubmitHistory(...):
    ...
    debug_bootstrap5 = True
    ...

You can then view the output in the console for tips on what templates and filters to migrate. Important: please don't commit this line at the end!

3. Migrate report templates (if needed)

Check to see that your report doesn't have custom templates that override base report templates. If it does, it should be flagged by the debug tool from step 2—so you will know!

If a custom template is set, and this is the only report using that template, you can then follow the procedure above to un-split template files. Prior to doing this, make sure to replace the bootstrap3 path of this template with the bootstrap5 path. You can then update the template's HTML as needed.

4. Migrate filters and any related javascript

The report debugging tool should provide a list of filters that haven't been migrated. You should take the following steps:

1. Make sure that all related filters are visible in the report (you might have to enable feature flags).
2. Make sure all the filters modify the report as expected, without javascript errors. Update related javascript as necessary.
3. Make sure any buttons on the page work as expected.

4. Commit your changes and save progress

Once you are done migrating the report, you must use the following migration tool to mark the report and its filters as complete:

./manage.py complete_bootstrap5_report <ReportClassName>

Deploy to Staging

Once you have your migrated view working locally, it is also helpful to deploy to staging and smoke test there. This is particularly important for migrations that involve switching a JavaScript library from a Bootstrap 3 to Bootstrap 5 version, such as switching the Bootstrap 3 date picker with Botostrap 5-compatible tempus dominus.

Most view migrations should also go through QA. Migrations can skip QA if they make relatively superficial changes and are in low-risk areas.

Rebuild Diffs

It is likely that you will have to run build_bootstrap5_diffs after the view migration. This will regenerate any diffs on split files affected by these changes.

Before you rebuild diffs, please update the bootstrap5_diff_config.json file with the command below:

./manage.py build_bootstrap5_diffs --update_app <app_name>

You can then commit those changes, if any, and then re run build_bootstrap5_diffs without the update_app option to rebuild the diffs for split files.

Step 4: Completing the App Migration

An app is considered fully migrated when all split files created in "Step 2" have been "un-split" and moved back to their original location. These files now fully inherit from boostrap5 base templates and dependencies.

When you have reached this point in the app migration, it's time to move on to the next application. Please update the app's migration status in this table.

You can run the following command to mark the application as complete:

./manage.py complete_bootstrap5_migration <app_name>

Please make sure you re-run build_bootstrap5_diffs with the update_app option as described above. If there are any changes, please commit those and then re-run build_bootstrap5_diffs without the update_app option.

Final Cleanup Process

Once all of the apps have completed the migration process, it will be time to do the final global cleanup of HQ. This involves making bootstrap5 the default behavior for templates by removing any split code changes referencing request.use_bootstrap5 or get_bootstrap_version() in templates, template tags, and other files.

Once these changes are complete, the @use_bootstrap5 decorator can be removed from all the views.

Additionally, the use_bootstrap5 class variable can be removed from all the reports. Cleanup of the bootstrap3 templates listed in COMMON_REPORT_TEMPLATES in corehq.apps.hqwebapp.utils.bootstrap.reports.debug can also be completed, as well as the filter templates marked completed by manage.py complete_bootstrap5_report.

Finally, LESS files can be removed from all parts of HQ and COMPRESS_PRECOMPILERS in settings.py can be updated to only precompile SCSS. Any LESS-related dependencies can be removed from package.json and commcare-cloud, and new developer setup guides like DEV_SETUP.md can be updated to remove references to LESS.

When you have completed the above changes, it is time to think about updating this guide to reflect on the process of this migration and suggestions for how to handle future migrations. LESS-related references will likely no longer be needed for future migrations, as Boostrap will stay on SASS/SCSS, so it will be fine to remove support for that. However, it might be good to leave pointers to commits, files, and other resources that we have used to support this migration. This page can then serve as the starting point for anyone embarking on a future (and hopefully less painful) challenge of the next Bootstrap Migration.

Deep-Dive Resources

Here are the official migration guides from Bootstrap. These guides detail every change made from the previous version to the next. The migration tool we created is meant to help automate the process of pouring over these guides for each change over and over again. However, if you are curious to read more, this is a good place to start.

• Bootstrap 4.6 Migration Guide
• Bootstrap 5.3 Migration Guide

If you are interested in reading about how we initially spec'd the process of Migrating from Bootstrap 3 to 5 on CommCare HQ, Dimagi folks are welcome to review the initial Tech Spec here.