apache
AIP-79 Generate assets for Flask application in FAB provider (#44744)
Revert the revert #45057 of #44744.
It failed because of:
Traceback (most recent call last):
File "/opt/airflow/./scripts/ci/pre_commit/compile_www_assets.py", line 86, in <module>
if is_fab_provider_installed():
File "/opt/airflow/./scripts/ci/pre_commit/compile_www_assets.py", line 79, in is_fab_provider_installed
return importlib.util.find_spec("airflow.providers.fab") is not None
File "/usr/local/lib/python3.9/importlib/util.py", line 94, in find_spec
parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'airflow'
I updated the function is_fab_provider_installed and check if airflow is installed first. I ran breeze release-management prepare-airflow-package --package-format wheel and it succeeds (but fails without the fix).
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
I updated this PR to create a new pre-commit to generate assets from FAB provider. It is basically the solution 2 described by @potiuk in this comment. I duplicated the script scripts/ci/pre_commit/compile_www_assets.py to scripts/ci/pre_commit/compile_fab_assets.pyto make it easier when we remove the Airflow 2 Flask UI, we will only have to delete scripts/ci/pre_commit/compile_www_assets.py (and its associated pre-commit).
The artifacts from FAB provider are not built on build time but are now part of the codebase.
The static check seems to delete the generated files I checked in. I dont understand why, I am still trying to find out why
The static check seems to delete the generated files I checked in. I dont understand why, I am still trying to find out why
Without looking into details, it might be that generating the files (and pre-commit in general) that generates them has a slightly different environmen in CI than in your env - maybe the local env has different npm/yarn version (not pinned) or maybe there is another configuration setting in CI that makes the result different.
There are a number of "tricks" to make things reproducible regardless of the environment (for example when we generate image snapshot of breeze we fix the terminal width, and hard-code some variable things (like default number of processors) that can vary between the different systems.
Sometimes this is unavoidable - for example when things are run on Mac and Linux there might be some real differences - then we might revert to running stuff in docker (to ensure maximum environment similarity) or have another trick - often we can calculate hash of the "source" files (and we do it in a few places) and implement the pre-commit to only actually do something when the hash changes - we do that with breeze help snapshots. We store the hash as part of the commit and pre-commit will only actually regenerate stuff if the hash of sources changes. That might be an easy way out as well.
The static check seems to delete the generated files I checked in. I dont understand why, I am still trying to find out why
Without looking into details, it might be that generating the files (and pre-commit in general) that generates them has a slightly different environmen in CI than in your env - maybe the local env has different npm/yarn version (not pinned) or maybe there is another configuration setting in CI that makes the result different.
There are a number of "tricks" to make things reproducible regardless of the environment (for example when we generate image snapshot of breeze we fix the terminal width, and hard-code some variable things (like default number of processors) that can vary between the different systems.
Sometimes this is unavoidable - for example when things are run on Mac and Linux there might be some real differences - then we might revert to running stuff in docker (to ensure maximum environment similarity) or have another trick - often we can calculate hash of the "source" files (and we do it in a few places) and implement the pre-commit to only actually do something when the hash changes - we do that with breeze help snapshots. We store the hash as part of the commit and pre-commit will only actually regenerate stuff if the hash of sources changes. That might be an easy way out as well.
Yep, that's what we do here. We generate a hash and re-generate the assets only if the hash changes. A per the output of the tests, the root cause might be because the CI fails to install dependencies with yarn. Looking at it :)
The static check seems to delete the generated files I checked in. I dont understand why, I am still trying to find out why
Without looking into details, it might be that generating the files (and pre-commit in general) that generates them has a slightly different environmen in CI than in your env - maybe the local env has different npm/yarn version (not pinned) or maybe there is another configuration setting in CI that makes the result different. There are a number of "tricks" to make things reproducible regardless of the environment (for example when we generate image snapshot of breeze we fix the terminal width, and hard-code some variable things (like default number of processors) that can vary between the different systems. Sometimes this is unavoidable - for example when things are run on Mac and Linux there might be some real differences - then we might revert to running stuff in docker (to ensure maximum environment similarity) or have another trick - often we can calculate hash of the "source" files (and we do it in a few places) and implement the pre-commit to only actually do something when the hash changes - we do that with breeze help snapshots. We store the hash as part of the commit and pre-commit will only actually regenerate stuff if the hash of sources changes. That might be an easy way out as well.
Yep, that's what we do here. We generate a hash and re-generate the assets only if the hash changes. A per the output of the tests, the root cause might be because the CI fails to install dependencies with
yarn. Looking at it :)
Mhm, Edge with the legacy UI probably will follow soon also by adding some UIs to replace the legacy. I assume there might be more coming step by step. I'd favor a dynamic generation at packaging time but I am also OK if we make it static first and then change later.
