Automatic pull requests and testing for updates

[frafra]

Description

In order not to get stuck with old versions, I suggest to use one of the following services, to automatically check for new updates:

• https://dependabot.com/
• http://pyup.io/
• https://requires.io/

[tomkralidis]

Note that we recently unpinned dependencies in master to try to keep current more efficiently. This will catch errors early, and we can pin when releasing. Does this approach work?

[frafra]

Those tools are useless with unpinned dependencies. The idea is to have pinned dependencies (or at least a lock file) to update via a nice pull requests with tests.

If you have unpinned dependencies the risk is that your code could break silently because you have not run the tests yet; if you pin them and evaluate these auto-pull request you do not break master and have a better insight and control about what is happening and why (especially if you have multiple updates at the same time).

A mixed approach would be to have a unpinned/partially-pinned dependency file and a lock file with the latest validated versions (like it is usually done with pipenv, poetry etc.).

[tomkralidis]

cc @ocefpaf @kalxas @ricardogsilva

Thanks @frafra. If we have unpinned dependencies doesn't Travis help in identifying breakage, at least foreach commit?

For releases, we recently discussed the following approach:

• keep requirements.txt unpinned
• when creating a stable release (minor or major), create the release branch first, pin dependencies and tag the release from the stable release branch

Thoughts?

[ricardogsilva]

folks I see some divergence here, maybe due to the way some people might be using pycsw, along with using different install procedures.

• if you use a <download code> + <install from requirements> method, you are using pycsw as an application. In this case your dependencies should be fixed and known to work. For this reason, our requirements file should have pinned dependencies.

  The approach suggested by @frafra fits in with that. we'd use a service like dependabot and it would make a PR whenever some new version of a lib is out, we'd run the tests and accept the PR, which would update our requirements file with the new version.

  This means that our requirements file will always perscribe a fixed config that we have tested and vouch that works

  So I'm +1 on @frafra's suggestion at this issue's description

• if you do a pip install pycsw you are installing pycsw for use as a library

  In this case, the install_requires attribute on setup.py is used to fetch other dependencies that pycsw needs. These dependencies should not be pinned, or rather should be more softly pinned, as stated in this python packaging page.

  That is why in #562 I am advocating that we should not build the value for setup.install_requires from the contents of requirements.txt, as these two files serve different purposes.

[kalxas]

I agree to change the strategy to pinning the requirements.txt and not pinning the setup.py (the downstream projects can do their own testing with unpinned versions)