Pull from upstream for non-git upstreams

[lbarcziova]

Context: last point from https://github.com/packit/packit-service/issues/1879#issuecomment-1429223256

Currently, in pull_from_upstream we support any Git repository as upstream. The next step, which is the goal of this epic is to support any upstream, not requiring to be a git repository.

Tasks:

• [x] packit/packit#2099
• [x] #2203
• [x] dashboard changes if needed
• [x] packit/packit.dev#927

[jpopelka]

Could you share examples of such (non-git) upstreams? Thanks

[lbarcziova]

I remember there should be e.g. multiple ruby packages like this, but @lachmanfrantisek who I discussed this with should know some concrete examples.

[lachmanfrantisek]

E.g. Pavel Valena wanted this for the Ruby packages, but there are more upstream projects that don't work with GIT. The thing is, that we don't need to do anything with the git -- the source is downloaded from the URL specified in specfile, specfile is taken from the dist-git and the only info taken from upstream might be changelog if they use that way of changelog generation (what is not always wanted/needed).

[pvalena]

Hello, thanks for looking into this!

I have several examples:

rubygem-globalid (most likely there's up to several hundred similar packages): https://src.fedoraproject.org/rpms/rubygem-globalid//blob/rawhide/f/rubygem-globalid.spec

Update PR would look like (this one is created by my automation): https://src.fedoraproject.org/rpms/rubygem-globalid/pull-request/4

Source is hosted on rubygems.org (downloaded directly or via gem fetch ...): https://rubygems.org/downloads/globalid-1.2.1.gem

New versions can be determined via anytia (release-monitoring.org), or here: https://rubygems.org/gems/globalid

In this case, git is needed for additional Source1, which also needs to be created/uploaded (but there are various/numerous approaches to this).

Apart that, the spec file doesn't need anything from upstream git; simple changelog message is enough. (Sometimes, tagged git release even differs from the gem file; plus, the gem files are checksumed & accepted as the source of truth for Ruby community).

---

Example without the additional source: https://src.fedoraproject.org/rpms/rubygem-fog-libvirt/pull-request/4

Another example with sample file change: https://src.fedoraproject.org/rpms/rubygem-addressable/pull-request/4

Additional to sources file, .gitignore is usually updated by fedpkg, but that can be solved by glob (it's not yet everywhere): https://src.fedoraproject.org/rpms/rubygem-ethon/pull-request/2#request_diff

[sergiomb2]

packit just need to know the new version , as the https://github.com/fedora-infra/the-new-hotness do . After that is just run :

package=$(basename $(pwd))
bugs_str="Update $package to ${version} (#$bugid)"
rpmdev-bumpspec -n $version -c "${bug_str}" $package.spec
spectool -g $package.spec (to download the source) 

[pvalena]

packit just need to know the new version , as the https://github.com/fedora-infra/the-new-hotness do . After that is just run :

package=$(basename $(pwd))
bugs_str="Update $package to ${version} (#$bugid)"
rpmdev-bumpspec -n $version -c "${bug_str}" $package.spec
spectool -g $package.spec (to download the source) 

It might be helpful to add the bug on another line like so:

https://src.fedoraproject.org/rpms/rubygem-rails/c/3e4297b98bfbc40fdca58f97fb1d529d28270ae1?branch=rawhide

Bodhi then should auto-close the bug.

[pvalena]

FYI the main reason for fetching the source in another way is that the gem files are already packaged and distributed via other means than git (and git website). Git can be used as a backup, but the tag is not the official release package or "source" (the packaged gem file can have different file listings).

---

This is not ideal, as we sometimes need to fetch the test suite from git (and add to lookaside cache for koji), but we don't want to diverge from the upstream (and other distros).

[michel-slm]

For python-selenium, we currently pull from PyPI (not ideal since it does not ship test cases, but the upstream git repo is gargantuan and also contains non-Python code...). The repo takes a few minutes to clone, and worse, for the last 4.20 release upstream monitoring noticed the PyPI release half an hour before the git repo is actually tagged (guessing the pypi release was done by hand).

So being able to track the actual upstream used by the package would be great

[lachmanfrantisek]

@michel-slm thanks for writing down your usecase!