Contribution script to generate config?

[aschleifer]

Hey @tony ,

how willing would you be to include some contribution scripts in the repository to help generate an initial config file?

I use vcspull to manage ~70 repos in my work place all under one gitlab group. I wrote a small shell script to scan this group and generate an initial config file out of it.

Initially I thought to just share this internally in my company, but if you are willing we could add this to the repository (after i made it a bit more abstract).

Please let me know your thoughts about this.

Best regards Segaja

[tony]

@aschleifer Yes, happily

In the long run I am considering a vcspull scan type feature (#25). If such a thing were to exist, it may deprecate such a script.

Assuming there was a vcspull scan, is there any behavior / functionality you'd find fit your / all needs?

[aschleifer]

I think vcspull scan would be a nice addition, but it only helps if you already have a bunch of repos checked out in a given folder structure. My script would talk to a gitlab instance and give you a list of all the repos under one group and write a config out of that.

I think both cases are valid but not eliminating each other.

On the current topic: I will make my script more generic and create a pull request.

[tony]

I think vcspull scan would be a nice addition, but it only helps if you already have a bunch of repos checked out in a given folder structure. My script would talk to a gitlab instance and give you a list of all the repos under one group and write a config out of that.

This seems clever.

It may make sense for future command/plugin for a bitbucket/github/gitlab api wrapper to pull a list of repos.

On the current topic: I will make my script more generic and create a pull request.

Sounds good.

If it can work with pure standard library (without requiring an additional dependency), even better. But if that makes it prohibitively difficult, feel free to use whatever packages work.

[aschleifer]

Right now my script is a basic shell script using curl and jq. If you want I could paste it here and maybe it can be converted to python and integrated into vcspull directly. For a first step I think it would already help to provide it as bash script

[tony]

@aschleifer I think pasting it here would be a good next step, good ahead!

[aschleifer]

#!/bin/bash

if [ -z "${GITLAB_TOKEN}" ]; then
    echo 'Please provide the environment variable $GITLAB_TOKEN'
    exit 1
fi

# these variables have to be set for each run
prefix="" # path to put the checkouts into (could be used as from cwd)
namespace="" # namespace/group to "scan" on gitlab instance
gitlab_host="" # gitlab hostname without protocol


current_namespace_path=""

curl --silent --show-error --header "Authorization: Bearer ${GITLAB_TOKEN}" "https://${gitlab_host}/api/v4/groups/${namespace}/projects?include_subgroups=true&per_page=100" \
    | jq -r '.[]|.namespace.full_path + " " + .path' \
    | LC_ALL=C sort \
    | while read namespace_path reponame; do
        if [ "${current_namespace_path}" != "${namespace_path}" ]; then
            current_namespace_path="${namespace_path}"

            echo "${prefix}/${current_namespace_path}:"
        fi

        # simplified config not working - https://github.com/vcs-python/vcspull/issues/332
        #echo "  ${reponame}: 'git+ssh://git@${gitlab_host}/${current_namespace_path}/${reponame}.git'"

        echo "  ${reponame}:"
        echo "    url: 'git+ssh://git@${gitlab_host}/${current_namespace_path}/${reponame}.git'"
        echo "    remotes:"
        echo "      origin: 'ssh://git@${gitlab_host}/${current_namespace_path}/${reponame}.git'"
    done \
   | tee vcspull.yaml

That is the version I have right now. The first 3 variables have to be given at runtime together with the GITLAB_TOKEN.

Once #332 gets resolved the 4 echo lines in the while loop could be replaced by the commented out one liner.

[tony]

@aschleifer How about you create a PR with this script in the scripts/ directory? Then we / I can include it in the docs/. We can also uncomment the simplified config when #332 is fixed.

I was going to suggest contrib/ but think that may fit python modules better (e.g. vcspull.contrib)

Another possibility, if you had time, it to write the above script in vanilla python 3 w/ standard library. That would actually the best

[aschleifer]

@tony I can easily provide this in a scripts/ folder via PR. I will also go ahead and add make the needed variables required on execution.

I haven't written python code in a while so it might take some time but could be fun to do it, but I think the first version is fine to be a shell script.

[tony]

The shell script would be fine to PR. 

If you want to, optionally, you can make a second PR with python, see if you can stick to standard lib (so the user wouldn't need to install extra packages) On 2/23/2022 10:13:22 AM, Andreas Schleifer @.> wrote: @tony [https://github.com/tony] I can easily provide this in a scripts/ folder via PR. I will also go ahead and add make the needed variables required on execution. I haven't written python code in a while so it might take some time but could be fun to do it, but I think the first version is fine to be a shell script. — Reply to this email directly, view it on GitHub [https://github.com/vcs-python/vcspull/issues/334#issuecomment-1048950502], or unsubscribe [https://github.com/notifications/unsubscribe-auth/AAAGNYHAIEBKXFVO4OIHZUTU4UBSDANCNFSM5PBL3J3Q]. You are receiving this because you were mentioned.Message ID: @.> [cbdce5b9-c7bf-4b10-84ae-949d1e211b5e]

[Segaja]

Thanks for adding the scripts to the documentation ( https://vcspull.git-pull.com/config-generation.html ).

If desired I can try to update the shell script to remove some of the differences between the two scripts from a usage point of view.

Also, once the new command-file structure ( discussed in https://github.com/vcs-python/vcspull/issues/333 ) is in place I might spend the time and reworking all this into a "native" vcspull command and then the two scripts get obsolete.

[tony]

Thanks for adding the scripts to the documentation ( https://vcspull.git-pull.com/config-generation.html ).

If desired I can try to update the shell script to remove some of the differences between the two scripts from a usage point of view.

Also, once the new command-file structure ( discussed in #333 ) is in place I might spend the time and reworking all this into a "native" vcspull command and then the two scripts get obsolete.

Yes you are welcome to

Also, once the new command-file structure ( discussed in https://github.com/vcs-python/vcspull/issues/333 ) is in place I might spend the time and reworking all this into a "native" vcspull command and then the two scripts get obsolete.

That is coming next - i am guessing the next 2 weeks or so. The reason why I don't have a concrete date is I'm reading the codebase from scratch and would approach things differently now - some areas are so opaque it'd turn off people from contributed