Ensure all released repo dependencies are in the `openedx` GitHub organization

[kdmccormick]

Context

Starting with Palm, if a repo is in the named release, then its first-party dependencies must be in the openedx GH org.

More details: https://discuss.openedx.org/t/interim-guidance-on-the-openedx-github-organization/8518

This is related to, but not the same as: https://github.com/openedx/build-test-release-wg/issues/203

Known issues

Deps to edx-platform:

• https://github.com/edx/outcome-surveys
• https://github.com/edx/edx-name-affirmation
• https://github.com/edx/braze-client
• https://github.com/edx/learner-pathway-progress
• https://github.com/open-craft/xblock-poll
• https://github.com/mitodl/edx-sga

Deps to default frontend-apps (learning, account, gradebook, profile, authn, discussions, course-authoring):

• TBD

Acceptance Criteria

Write tooling to find all offending dependencies.

Then, for each dependency, either:

• move it to the openedx org; or
• remove it from edx-platform's requirements list, and tell interested operators (notably 2U) that those packages must be added to extra/private requirements in order to be installed with the platform.

[kdmccormick]

The "Known Issues" list as of right now was based on some initial discovery by @nedbat . We have not yet checked the dependencies of the default frontend-apps.

Also: We have not checked the dep lists for course-discovery, ecommerce, and other release-tagged IDAs. Tthis is OK, because none of those IDAs are run by default in the named release.

[ghassanmas]

I think you might want to add https://github.com/edx/mongoid-tree, which is used by https://github.com/openedx/cs_comments_service

refs:

• https://github.com/openedx/cs_comments_service/blob/3420c0a86aa1f749deed08a47758909dcc1844e1/Gemfile#L30
• https://github.com/openedx/cs_comments_service/blob/3420c0a86aa1f749deed08a47758909dcc1844e1/Gemfile3#L30

Or may be not, because forum is not run by defualt...

[nedbat]

I've added a check to edx-repo-health to ensure all main repos are in the correct organization: https://github.com/openedx/edx-repo-health/pull/340

[kdmccormick]

Thanks Ned!

[mariajgrimaldi]

Hi folks! Is there something we could help you with so we can move this forward?

[kdmccormick]

@nedbat , had you been working on a script to check whether dependencies of released repos are in the openedx org? Or was that just for top-level released repos?

[nedbat]

I worked on it back in December: https://github.com/openedx/repo-tools/blob/master/edx_repo_tools/find_dependencies/README.rst

[kdmccormick]

Nice, thanks @nedbat .

@mariajgrimaldi this issue remains important, but I think everyone involved is just busy right now. I'm not sure if Feanil or Ned plan on picking up it; if not, I'd probably be able to work more on it in July-ish.

[feanil]

I think the next step is to re-run Ned's script and get a concrete list of all the repos that are outside the openedx organization. Once that's done we can ticket each one individually and start picking them off as it makes sense. I'm in the same boat as Kyle and can't spend a lot of time on it now but @mariajgrimaldi if you have time to do that and update this ticket with the findings, that would be a great help.

[mariajgrimaldi]

Do you folks still need help with this? I've now allocated the time for it -- sorry for the delay!

[kdmccormick]

@mariajgrimaldi no worries. If you have time and are interested in working on this, then certainly we could still use the help!

[nedbat]

I re-ran the dependency analyzer last week and got these results:

These are the 47 repos tagged for Palm:

openedx/blockstore openedx/configuration openedx/course-discovery openedx/credentials openedx/cs_comments_service openedx/devstack openedx/docs.openedx.org openedx/DoneXBlock openedx/ecommerce openedx/ecommerce-worker openedx/edx-analytics-configuration openedx/edx-analytics-dashboard openedx/edx-analytics-data-api openedx/edx-analytics-pipeline openedx/edx-app-android openedx/edx-app-ios openedx/edx-documentation openedx/edx-notes-api openedx/edx-platform openedx/enterprise-access openedx/enterprise-catalog openedx/frontend-app-account openedx/frontend-app-authn openedx/frontend-app-communications openedx/frontend-app-course-authoring openedx/frontend-app-discussions openedx/frontend-app-ecommerce openedx/frontend-app-gradebook openedx/frontend-app-learner-dashboard openedx/frontend-app-learner-record openedx/frontend-app-learning openedx/frontend-app-library-authoring openedx/frontend-app-ora-grading openedx/frontend-app-payment openedx/frontend-app-profile openedx/frontend-app-publisher openedx/frontend-app-support-tools openedx/frontend-template-application openedx/license-manager openedx/openedx-aspects openedx/openedx-demo-course openedx/openedx-i18n openedx/openedx-test-course openedx/repo-tools openedx/testeng-ci openedx/tubular openedx/xqueue

These are the 54 repos in the openedx organization in the transitive dependencies:

openedx/api-doc-tools openedx/auth-backends openedx/blockstore openedx/brand-openedx openedx/code-annotations openedx/codejail openedx/completion openedx/credentials-themes openedx/django-config-models openedx/django-lang-pref-middleware openedx/django-pyfs openedx/django-splash openedx/django-user-tasks openedx/ecommerce-worker openedx/edx-ace openedx/edx-analytics-data-api-client openedx/edx-bulk-grades openedx/edx-celeryutils openedx/edx-custom-a11y-rules openedx/edx-django-utils openedx/edx-drf-extensions openedx/edx-enterprise openedx/edx-enterprise-data openedx/edx-enterprise-subsidy-client openedx/edx-milestones openedx/edx-organizations openedx/edx-proctoring openedx/edx-rbac openedx/edx-rest-api-client openedx/edx-search openedx/edx-toggles openedx/edx-when openedx/eslint-config openedx/event-bus-kafka openedx/event-bus-redis openedx/frontend-build openedx/frontend-component-footer openedx/frontend-component-header openedx/frontend-enterprise openedx/frontend-platform openedx/i18n-tools openedx/olxcleaner openedx/opaque-keys openedx/openedx-events openedx/openedx-filters openedx/openedx-learning openedx/paragon openedx/react-unit-test-utils openedx/reactifex openedx/stylelint-config-edx openedx/XBlock openedx/xblock-lti-consumer openedx/xblock-utils openedx/xss-utils

These are the 9 repos in adjacent organizations that are in the transitive dependencies:

edx/brand-edx.org edx/braze-client edx/edx-name-affirmation edx/frontend-component-footer-edx edx/getsmarter-api-clients edx/new-relic-source-map-webpack-plugin edx/token-utils edx/ux-pattern-library mitodl/edx-sga

BUT: Ed noticed that ora2 isn't mentioned. We aren't yet analyzing dependencies that get installed via other mechanisms. We still have some work to do on the analyzer.

[kdmccormick]

Thanks @nedbat.

By "other mechanisms" I assumed you mean things like EDXAPP_PRIVATE_REQUIREMENTS, but ora2 is installed in base.txt: https://github.com/openedx/edx-platform/blob/ddb092c07c6e2b663dc3d52e7197b42f7f017988/requirements/edx/base.txt#L776

[nedbat]

Yeah, I'm digging into this now. I don't understand why ora2 doesn't end up in the results yet.

[nedbat]

Updated results (from this repo-tools fix):

These are the 80 repos in the openedx organization in the transitive dependencies:

https://github.com/openedx/RecommenderXBlock https://github.com/openedx/XBlock https://github.com/openedx/acid-block https://github.com/openedx/api-doc-tools https://github.com/openedx/auth-backends https://github.com/openedx/blockstore https://github.com/openedx/brand-openedx https://github.com/openedx/browserslist-config https://github.com/openedx/ccx-keys https://github.com/openedx/code-annotations https://github.com/openedx/codejail https://github.com/openedx/completion https://github.com/openedx/credentials-themes https://github.com/openedx/crowdsourcehinter https://github.com/openedx/django-config-models https://github.com/openedx/django-lang-pref-middleware https://github.com/openedx/django-pyfs https://github.com/openedx/django-splash https://github.com/openedx/django-user-tasks https://github.com/openedx/ecommerce-worker https://github.com/openedx/edx-ace https://github.com/openedx/edx-analytics-data-api-client https://github.com/openedx/edx-bootstrap https://github.com/openedx/edx-bulk-grades https://github.com/openedx/edx-celeryutils https://github.com/openedx/edx-custom-a11y-rules https://github.com/openedx/edx-django-release-util https://github.com/openedx/edx-django-sites-extensions https://github.com/openedx/edx-django-utils https://github.com/openedx/edx-drf-extensions https://github.com/openedx/edx-enterprise https://github.com/openedx/edx-enterprise-data https://github.com/openedx/edx-enterprise-subsidy-client https://github.com/openedx/edx-milestones https://github.com/openedx/edx-ora2 https://github.com/openedx/edx-organizations https://github.com/openedx/edx-proctoring https://github.com/openedx/edx-rbac https://github.com/openedx/edx-rest-api-client https://github.com/openedx/edx-search https://github.com/openedx/edx-submissions https://github.com/openedx/edx-toggles https://github.com/openedx/edx-ui-toolkit https://github.com/openedx/edx-val https://github.com/openedx/edx-when https://github.com/openedx/enmerkar-underscore https://github.com/openedx/eslint-config https://github.com/openedx/event-bus-kafka https://github.com/openedx/event-bus-redis https://github.com/openedx/event-tracking https://github.com/openedx/frontend-build https://github.com/openedx/frontend-component-cookie-policy-banner https://github.com/openedx/frontend-component-footer https://github.com/openedx/frontend-component-header https://github.com/openedx/frontend-enterprise https://github.com/openedx/frontend-lib-content-components https://github.com/openedx/frontend-lib-special-exams https://github.com/openedx/frontend-platform https://github.com/openedx/help-tokens https://github.com/openedx/i18n-tools https://github.com/openedx/mockprock https://github.com/openedx/olxcleaner https://github.com/openedx/opaque-keys https://github.com/openedx/openedx-calc https://github.com/openedx/openedx-chem https://github.com/openedx/openedx-events https://github.com/openedx/openedx-filters https://github.com/openedx/openedx-learning https://github.com/openedx/paragon https://github.com/openedx/react-unit-test-utils https://github.com/openedx/reactifex https://github.com/openedx/studio-frontend https://github.com/openedx/stylelint-config-edx https://github.com/openedx/super-csv https://github.com/openedx/tinymce-language-selector https://github.com/openedx/user-util https://github.com/openedx/web-fragments https://github.com/openedx/xblock-lti-consumer https://github.com/openedx/xblock-utils https://github.com/openedx/xss-utils

These are the 10 repos in adjacent organizations that are in the transitive dependencies:

https://github.com/edx-unsupported/eslint-config-edx https://github.com/edx/brand-edx.org https://github.com/edx/braze-client https://github.com/edx/edx-name-affirmation https://github.com/edx/frontend-component-footer-edx https://github.com/edx/getsmarter-api-clients https://github.com/edx/new-relic-source-map-webpack-plugin https://github.com/edx/token-utils https://github.com/edx/ux-pattern-library https://github.com/mitodl/edx-sga

[nedbat]

Hmm, there are still some missing repos because some don't declare their URL in their metadata:

Repo URL is UNKNOWN in files/openedx_django_wiki-2.0.1-py3-none-any.whl
No repo URL in files/staff_graded_xblock-2.1.1-py3-none-any.whl
Repo URL is UNKNOWN in files/pynliner-0.8.0-py2.py3-none-any.whl
Repo URL is UNKNOWN in files/xblock_poll-1.13.0-py3-none-any.whl
No repo URL in files/django_multi_email_field-0.7.0-py3-none-any.whl
No repo URL in files/shapely-2.0.1-cp38-cp38-macosx_10_9_x86_64.whl
Repo URL is UNKNOWN in files/done_xblock-2.1.0-py3-none-any.whl
No repo URL in files/xblock_drag_and_drop_v2-3.2.0-py3-none-any.whl
No repo URL in files/click_didyoumean-0.3.0-py3-none-any.whl
Repo URL is UNKNOWN in files/xblock_google_drive-0.4.0-py2.py3-none-any.whl
Repo URL is UNKNOWN in files/edx_user_state_client-1.3.2-py3-none-any.whl

(I think the difference between "No repo URL" and "Repo URL is UNKNOWN" is due to how the packages are built?)

I've made some pull requests to add the URL to our packages. Thumbs appreciated:

• [ ] https://github.com/openedx/django-wiki/pull/198
• [ ] https://github.com/openedx/staff_graded-xblock/pull/193
• [ ] https://github.com/open-craft/xblock-poll/pull/101
• [ ] https://github.com/openedx/DoneXBlock/pull/164
• [ ] https://github.com/openedx/xblock-drag-and-drop-v2/pull/345
• [ ] https://github.com/openedx/xblock-google-drive/pull/61
• [ ] https://github.com/openedx/edx-user-state-client/pull/93

[ghassanmas]

I don't if this also releavnt while working on something I found a dependency in frontend-app-learning called frontend-lib-learning-assistant ref in pacakge.json whichs exits in edx https://github.com/edx/frontend-lib-learning-assistant