[DEPR]: Asset processing in Paver

[kdmccormick]

ℹ️ This DEPR is a prerequisite to the general Paver DEPR.

Proposed

10 March 2023

Accepted

3 May 2023

First Open edX Named Release Without This Functionality

Sumac.

Details:

Release	Old (paver *_assets ...)	New (npm run build, etc)
Palm	Supported	Available
Quince	Supported	Available
Redwood	Deprecated	Supported
Sumac	Removed	Supported

(Latest timeline update)

Rationale

See Reimplement static asset processing: Context: Current Pain Points

Removal

We are replacing edx-platform's current CLI for building, collecting & watching assets, which is built on an old Python library, paver.

The affected commands are:

• edx-platform: paver update_assets
• edx-platform: paver watch_assets
• edx-platform: paver compile_sass
• edx-platform: paver process_xmodule_assets
• tutor: tutor ... [run|exec] [lms|cms] openedx-assets ...

and the affected LMS & CMS Django settings are:

• JS_ENV_EXTRA_CONFIG
• WEBPACK_CONFIG_PATH

Cleanup steps

• [ ] Search the codebase & docs, and update and dangling references to:
  • pavelib/assets.py
  • pavelib.assets
  • paver update_assets
  • paver watch_assets
  • paver webpack
  • paver process_xmodule_assets
  • paver compile_sass
• [ ] Remove pavelib/assets.py and all associated unit tests
• [ ] Remove asset-related requirements from requirements/edx/paver.in:
  • libsass (note: this should have been added to a separate asset.in file, for use by scripts/build-assets.sh)
  • watchdog
• [ ] ...anything else?

Replacement

The new CLI for building, collecting and watching assets will use npm scripts and django management commands.

Deprecation

The paver commands will raise warnings before the Redwood release.

Migration

• For Tutor users
• For direct edx-platform users

Additional Info

The eventual goal is to remove not just Paver, but all Python from the asset build process. The work to remove Python will continue once this DEPR is complete:

• https://github.com/openedx/edx-platform/issues/31800

[kdmccormick]

Announcement: https://discuss.openedx.org/t/deprecation-removal-edx-platform-asset-processing-in-paver/9552

[kdmccormick]

The new implementation won't be landing in Palm, most likely. I pushed the acceptance date out to May 3rd, and pushed the final removal out to Redwood.

[kdmccormick]

This deprecation has been Accepted. Development of the replacement asset script is ongoing (see issues linked in description). When the replacement is ready, I'll put a warning in the output of the old asset processing script and move this issue to Deprecated.

[kdmccormick]

The new asset processing suite has been merged as "experimental". I will continue to test it, and will communicate when it's time to mark the Paver assets suite as deprecated and encourage folks to switch to the new suite.

[kdmccormick]

I am hoping to mark the old asset build as deprecated in Quince so we can remove it in Redwood, although the content libraries V2 project is competing for my time. Here's my WIP PR: https://github.com/kdmccormick/tutor/pull/37

[kdmccormick]

@feanil As I alluded to earlier, carrying out this DEPR will allow us to circumvent libsass-python's Python API (which isn't compatible with Python>3.9) and instead go straight to the underlying libsass C API. I'll take this on.

[kdmccormick]

This has renewed urgency, since the Python 3.8->3.12 upgrade is blocked by libsass-python. This must be resolved well ahead of the Redwood cutoff (late April).

Here is my updated plan:

• Before the Redwood cutoff, to unblock the Python upgrade:
  • I will make paver compile_sass a thin wrapper around npm run compile-sass. Not a breaking change.
  • I will update compile_sass.py so that it does not depend on libsass-python. It will still depend on libsass and we will still install libsass-python, but we will not need to execute any Python code from libsass-python. Not a breaking change.
  • I will update Tutor to use npm scripts instead of paver. This could be breaking or non-breaking for Tutor users, depending on whether we want to keep openedx-assets around for one more release.
• Before the Sumac cutoff:
  • I will remove all of pavelib/assets.py. This is the breaking change. Users of the paver assets CLI, such as the openedx/configuration and openedx-unsupported/devstack, will need to update to the new npm CLI.

I've updated the ticket description with the latest dates and migration steps.

FYI @feanil , @awais786 , @robrap , @regisb

[robrap]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 90.94%. Comparing base (ac9f2d5) to head (22fb824). Report is 49 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #10320       +/-   ##
===========================================
+ Coverage   48.75%   90.94%   +42.18%     
===========================================
  Files          32        8       -24     
  Lines        3095      497     -2598     
===========================================
- Hits         1509      452     -1057     
+ Misses       1586       45     -1541     

Flag	Coverage Δ
autograder	?
migrator	?
python_submitty_utils	?
submitty_daemon_jobs	90.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

[kdmccormick]

UPDATE: This PR switches the old paver static asset build commands to be simple wrappers around npm run commands, with instructive deprecation warnings:

• https://github.com/openedx/edx-platform/pull/34318.

This prepares us to drop the paver asset build commands after Redwood. After this PR merges, I will move the ticket status to Deprecated.

[kdmccormick]

I am rolling this DEPR into a larger proposed deprecation of all edx-platform Paver usage: https://github.com/openedx/edx-platform/issues/34467