gpdb icon indicating copy to clipboard operation
gpdb copied to clipboard

Collapse the pythonsrc-ext submodule.

Open higuoxing opened this issue 1 year ago • 2 comments

We're going to build gpdb on runway/buildweb in the future, on which we are unable to fetch submodules from outside repos. This patch helps collapse pythonsrc-ext submodules. This patch also helps clean up unused/outdated packages.

Previously, if the 'gpMgmt/bin/pythonSrc/ext' directory is empty, we assume that users install these packages from PyPI otherwise these packages are compiled while building gpdb.

After collapsing the pythonsrc-ext submodule, we introduced a new argument --with-pythonsrc-ext (default=no) for ./configure. If user specifies --with-pythonsrc-ext, these packages are compiled while building gpdb, otherwise they will not be compiled.

Required packages are moved to gpMgmt/bin/pythonSrc/ext:

  • PyGreSQL-5.2 used by gpMgmt tools
  • PyYAML-5.3.1 used by gpload
  • behave-1.2.6 used for gpMgmt tools testing
  • mock-1.0.1 used for gpMgmt tools testing
  • parse-1.8.2 used by behave-1.2.6
  • psutil-5.7.0 used by gpMgmt tools
  • pylint-0.21.0 used for linting python codes
  • setuptools-36.6.0 used for installing dependencies

Outdated/Unused packages are not included:

  • argparse-1.2.1: it's a builtin package since python3.2
  • behave-1.2.4: it's outdated.
  • ecdsa-0.13: it's removed by commit d4df61d532496ec0fbd0653f0068584a391e2f00
  • epydoc-3.0.1: it's removed by commit 766ecfdc0c73b50818731e68adef4d2071ca15ed
  • lockfile-0.9.1: it's removed by commit 8190ed4059b7be6c8c0edde146afd1ba3157f14b
  • logilab-astng-0.20.1 and logilab-common-0.50.1: they are not used in gpdb7.
  • paramiko-1.18.4 and paramiko-1.7.6-9: they are removed by commit 3b8ea185312c3940fc15b3600b430ee185aa3e9c and d4df61d532496ec0fbd0653f0068584a391e2f00
  • parse-1.5.3: it's outdated.
  • psutil-4.0.0: it's outdated.
  • pycrypto-2.0.1 and pycrypto-2.6.1: they are removed by commit d4df61d532496ec0fbd0653f0068584a391e2f00
  • unittest2-0.5.1: it's used for running unit tests for Python2.

Co-authored-by: Hao Zhang [email protected]

higuoxing avatar Aug 29 '22 07:08 higuoxing

This PR is for discussion. Seems that we can do some clean-up jobs to python packages.

higuoxing avatar Aug 29 '22 08:08 higuoxing

Pipeline: https://dev.ci.gpdb.pivotal.io/teams/main/pipelines/gpdb-dev-gpr-972-collapse-python-modules

higuoxing avatar Sep 21 '22 06:09 higuoxing

Can we remove pylint as well? Seems that we don't have pipelines that running pylint to check our python code. Besides, the version of pylint in our repo is 0.21.0 which released in 2010. Maybe we can remove it first and consider how to integerate a newer version pylint to our pipeline later.

higuoxing avatar Sep 27 '22 03:09 higuoxing