[DEPR]: Paver

[kdmccormick]

:information_source: Context: the operator-facing Paver assets module was accepted for deprecation back in May 2023, but we haven't removed it yet. Now, in May 2024, we are ready to deprecate the entire Paver suite. For simplicity, we are deprecating all Paver commands in this single ticket, assets included.

Timeline

Communicated

Operator-facing commands: 2023-03-08 (thread) Remaining commands: 2024-05-01 (thread)

Acceptance

Operator-facing commands: 2023-05-03 Remaining commands: 2024-05-21

Replacement available

Operator-facing commands: 2024-05-07, Redwood Internal CI commands: 2024-11-09, Sumac (Target)

Earliest removal

2024-11-07, Sumac

Rationale

edx-platform historically handled its build scripts with paver:

Paver is a Python-based software project scripting tool along the lines of Make or Rake. It is not designed to handle the dependency tracking requirements of, for example, a C program. It is designed to help out with all of your other repetitive tasks (run documentation generators, moving files about, downloading things), all with the convenience of Python’s syntax and massive library of code.

Using paver has a few problems:

• Paver adds dependencies to edx-platform: Paver follows the Python packaging model, so anything that's a dependency of edx-platform's pavelib suite becomes a dependency of edx-platform itself. In addition to the paver package, this means that we install libsass-python in production, which takes 60+ seconds (!) to compile and install.
• Paver makes Python a requirement for all build actions: When your build tooling runs on Python, then you need a Python environment in order to do anything, even things that don't involve Python (including building static assets, running JS tests, etc.). This harms the cachability of edx-platform Dockerfiles, because it means that any change to the Python requirements list will invalidate any build steps used by Paver.
• Paver is indirect: Paver basically just wraps shell commands that you could have run directly, adding a layer of complexity and potential bugs. Build scripts are ideally very simple--unlike application code, repetition is preferable to abstraction; they should be easy to write, easy to read, and easy to throw away and replace if necessary. Standard tools like Makefiles, Bash scripts, and Dockerfiles all encourage and excel at this sort of simplicity; Paver does not.
• Paver is idiosyncratic: Unlike Make and Bash, Paver is not at all a standard part of a programmer's toolset, so it makes edx-platform less approachable for new contributors. Even for programmers who want to script in Python, Click is significantly more popular, well-documented, and user-friendly.

For the Paver Asset commands in particular, more depth is provided in the 'Reimplement edx-platform static asset processing' ADR.

Replacement

Module	Replacement
assets	New static assets guide
quality	Directly invoking tools (pylint, etc.) when possible, Makefile targets otherwise
test_js	Makefile target
prereqs	pip
docs	Makefile target
i18n	Makefile targets for common tasks; i18n_tool for advanced tasks
servers	Tutor (or the deprecated Docker-based Devstack, or bare-metal development

Migration

To ease migration for Paver users, in Redwood, each Paver Asset command will simply proxy to its replacement command, and will raise a deprecation warning explaining the new command that it is running.

By Nov 7th 2024, operators will need to have switched to the new commands, taking the steps indicated below by ACTION REQUIRED.

Requirements (Operators: ACTION REQUIRED)

In Redwood and earlier, Paver and its dependencies were included in requirements/edx/base.txt

Starting in Sumac, these dependencies will be removed from requirements/edx/base.txt. Instead, operators will need to install:

• requirements/edx/assets.txt to build static assets
• requirements/edx/testing.txt to run Python tests and linting
• requirements/edx/base.txt download translations and collect static assets

Commands with known users (Operators: ACTION REQUIRED)

These replacements are production-ready as of May 7th, for Redwood.

Module	Known Users	Before	After
assets	Ansible, Devstack	paver update_assets	npm run build && ./manage.py lms collecstatic --noinput && ./manage.py cms collectstatic
assets	None	paver process_xmodule_assets	No longer needed - no replacement
assets	Tutor, pavelib	paver compile_sass	npm run compile-sass
assets	pavelib	./manage.py [lms/cms] compile_sass	npm run compile-sass
assets	pavelib	paver webpack	npm run webpack
assets	Devstack	paver watch_assets	npm run watch
prereqs	Ansible, Devstack, pavelib	paver install_prereqs	pip install -r requirements/edx/base.txt -r requirements/edx/assets.txt && npm clean-install
prereqs	pavelib	paver install_node_prereqs	npm clean-install
prereqs	pavelib	paver uninstall_python_packages	Will not be replaced
prereqs	pavelib	paver install_coverage_prereqs	pip install -r requirements/edx/coverage.txt
prereqs	pavelib	paver install_python_prereqs	pip install -r requirements/edx/base.txt

Commands without known users

These commands have been replaced for a long time. We don't know of any users of them, except the old Vagrant Devstack, which was deprecated 8 years ago.

Module	Before	After
docs	paver build_docs	make docs
i18n	paver i18n_validate_gettext	which xgettext
i18n	paver i18n_extract	make extract_translations
i18n	paver i18n_dummy	i18n_tool dummy && i18n_tool generate
i18n	paver i18n_generate	i18n_tool generate
i18n	paver i18n_generate_strict	i18n_tool generate --strict
i18n	paver i18n_clean	make clean_translations
servers	paver check_settings	None
servers	paver lms	./manage.py lms runserver
servers	paver studio	./manage.py cms runserver
servers	paver run_all_servers	tutor local ...
servers	paver devstack	tutor dev ...

Commands used by internal CI

These commands are only used by the upstream openedx project for CI. They will replaced before Sumac. Operators do not need to take action.

Module	Replacement Status	Before	After
quality	This is a no-op	paver find_fixme	Remove
quality	Need to implement	paver run_eslint	eslint
quality	Need to implement	paver run_stylelint	make stylelint_js
quality	Need to implement	paver run_xsslint	make xsslint
quality	Need to implement	paver run_pii_check	make pii_check
quality	Need to implement	paver check_keywords	make check_keywords
quality	Ready to use	paver run_quality	pylint
quality	Ready to use	paver run_pylint	pylint
quality	Need to implement	paver run_pep8	pycodestyle
js_test	Need to implement	paver diff_coverage	make diff_coverage_js
js_test	Need to implement	paver test_js	make test_js
js_test	Need to implement	paver test_js_run	make test_js
js_test	Need to implement	paver test_js_dev	make test_js MODE=browser

Django settings for Asset commands (Operators: ACTION REQUIRED)

In order to reimplement the Paver Asset commands without Python/Django, we are changing how several settings are configured.

The following Django settings are becoming read-only mirrors. If you override them now, remove your overrides:

• LMS:
  • STATIC_ROOT (a string, loaded from STATIC_ROOT_LMS env var)
  • COMPREHENSIVE_THEME_DIRS (loaded from the env var, parsed into a list of strings)
• CMS:
  • STATIC_ROOT (a string, loaded from STATIC_ROOT_CMS env var)
  • COMPREHENSIVE_THEME_DIRS (loaded from the env var, parsed into a list of strings)

The following Django settings are being removed. If you override them now, remove your overrides:

• LMS:
  • STATIC_ROOT_BASE
  • WEBPACK_CONFIG_PATH
  • JS_ENV_EXTRA_CONFIG
• CMS:
  • STATIC_ROOT_BASE
  • WEBPACK_CONFIG_PATH
  • JS_ENV_EXTRA_CONFIG

The following new environment variables are available. Set these in your environment, in place of the overrides you removed above:

• STATIC_ROOT_LMS (path)
• STATIC_ROOT_CMS (path)
• COMPREHENSIVE_THEME_DIRS (colon-separated paths)
• WEBPACK_CONFIG_PATH (path)
• JS_ENV_EXTRA_CONFIG (serialized as json)

Note: If you previously set the STATIC_ROOT_BASE Django setting to /blah, then you should now set the LMS and CMS environment variables as so:

• STATIC_ROOT_LMS=/blah
• STATIC_ROOT_CMS=/blah/studio

Deprecation

In time for Redwood, deprecation warnings will be added to all edx-platform Paver Asset commands.

Removal

• In edx-platform:

  • rm -rf ./pavelib/
  • rm ./requirements/edx/paver.in
  • rm ./requirements/edx/paver.txt
  • Search docs for 'paver' and replace with new commands

• Elsewhere in the openedx GitHub org:

  • Search docs for 'paver' and replace with new commands

• In Tutor:

  • Replace paver Asset references with new commands

### Tasks/PRs
- [ ] https://github.com/openedx/edx-platform/issues/31624
- [ ] https://github.com/openedx/edx-platform/pull/31810
- [ ] https://github.com/openedx/edx-platform/pull/31934
- [ ] https://github.com/openedx/edx-platform/pull/31790
- [ ] https://github.com/openedx/edx-platform/pull/32717
- [ ] https://github.com/openedx/edx-platform/pull/32767
- [ ] https://github.com/openedx/edx-platform/pull/32804
- [ ] https://github.com/openedx/edx-platform/pull/32823
- [ ] https://github.com/openedx/edx-platform/pull/32866
- [ ] https://github.com/openedx/edx-platform/pull/34318
- [ ] https://github.com/openedx/edx-platform/pull/34468
- [ ] https://github.com/openedx/edx-platform/pull/34554
- [ ] https://github.com/overhangio/tutor/pull/1042
- [ ] https://github.com/openedx/edx-platform/pull/34829
- [ ] https://github.com/openedx/edx-platform/pull/34830
- [ ] https://github.com/openedx/edx-platform/issues/34845
- [ ] https://github.com/openedx/edx-platform/pull/34832
- [ ] TODO update all edx-platform documentation
- [ ] TODO update external documentation

[kdmccormick]

I have updated the Django settings to clarify that we will also be removing the STATIC_ROOT_BASE Django setting.

[kdmccormick]

Now that this is accepted, I'll split the removal into three steps:

• [x] https://github.com/openedx/edx-platform/pull/34829
• [ ] https://github.com/openedx/edx-platform/pull/34830
• [ ] https://github.com/openedx/edx-platform/pull/34832

Step 1 is free of downstream contingencies. It is safe to happen at any time.

For Step 2, we'll coordinate with early deployers (2U) so that they have time to update their deployment scripts to avoid the assets and prereqs commands. This step will deliver a speed-up to the production Tutor build.

For Step 3, we'll need to move the remaining quality/js checks out of Paver, uninstall Paver, and a do bulk sweep through the docs and update all instructions that still specify "paver".

[feanil]

FYI, I created https://github.com/openedx/edx-platform/issues/34834 to handle a specific github workflow around paver assets that also needs an enhancement.

[kdmccormick]

That workflow update ^ is now handled.

Here is an epic to handle the remaining upstream CI updates:

• https://github.com/openedx/edx-platform/issues/34845

[kdmccormick]

As of May 7th 2024, the operator-facing Paver commands have been fully replaced with npm & django commands. The replacement commands are being invoked in Redwood (directly) and on edx.org (via legacy Paver wrappers). Given that, we're targeting Nov 7th 2024 for removal of the Paver wrappers, or earlier if our first deployers (2U) indicate that they're ready. I've updated this ticket's description and added "Operators: ACTION REQUIRED" wherever I believe operators need to migrate; I'll make sure those land in the Sumac release notes too. Operators, let me know if there's anything I can do to clarify. Thanks!

[timmc-edx]

The ticket currently says under Replacement Available « Internal CI commands: 2024-11-09, Sumac (Target) » but that's after the targeted removal date of Nov 7. It sounds like maybe the replacements are already available, though?

[feanil]

So Kyle broke this out into a couple of different sections above (different tables). The Commands with known users (Operators: ACTION REQUIRED) and the Commands without known users sections have replacements that have already been implemented as of right now. In some cases the "replacement" is just calling the underlying command directly.

The unfinished section is the Commands used by internal CI where some have been implemented but some still remain. Let me know if that doesn't answer your questions.

[kdmccormick]

What @feanil says is right. I've tried to amend the description to be a bit clearer. In retrospect, it may have been clearer to just have two DEPR tickets, but doing that now would probably just add confusion. @timmc-edx let me know if it's still unclear.

[timmc-edx]

I see now, thanks!

[dianakhuang]

Just putting it here as a note for something that I/someone else could do: would be nice to create a make command for running asset processing, because otherwise I will forget the correct way of doing it.

[feanil]

Just putting it here as a note for something that I/someone else could do: would be nice to create a make command for running asset processing, because otherwise I will forget the correct way of doing it.

FWIW, it's just npm run build or npm run build-dev, that does all the things and aligns with our MFEs which do npm build to build out the serialized assets.

Also there is a full guide in the docs: https://github.com/kdmccormick/edx-platform/blob/master/docs/references/static-assets.rst

[dianakhuang]

Oh, awesome! I have been around too long and of course I didn't think to read the doc. 😅