Ability to filter repos

[dorner]

Right now it seems like the repositories used are all or nothing. Our use case for this tool would be more like "add this PR for all repositories that use Ruby as its main language", or possibly 'all repositories that have the text "circleci" in the file "ci.yml"'. GitHub supports these filters/searches, so it would be great if we could use that in this tool.

[lindell]

Hello! I assume you would like to simply use the same search as GitHub provides on the website?

multi-gitter run -X "user:lindell language:go" ...

I think this is a good idea! But I have some concerns that needs to be though of as to not allow users to shoot themselves in the foot without knowing it.

A user could simply just run.

multi-gitter run -X "language:javascript" ...

Which would result in the searching of all Javascript repos of github.

But this might be mitigated but either always log a warning, or when the result set is over 100 if this feature is used.

[lindell]

Language filtering would also be an option where this pitfall would not exist.

[lindell]

I know it would be better to have this done before any script is run, and I intend to implement something for it.

But in the meantime, adding a check in the top of your script to check for the text "circleci" in the file "ci.yml" and exit with a non-zero exit code if that is not the case would be an option :)

https://stackoverflow.com/questions/4749330/how-to-test-if-string-exists-in-file-with-bash

[dorner]

Yep, we could do filtering in the script itself, but it would mean cloning hundreds of repos that we don't need. :)

I was thinking of this filter to be an add-on to the existing owner / user flags rather than allowing it to be used on its own for all repos in GitHub. :)

[lindell]

Then I think just a language filter is the best option 👍

The API does not provide filtering in the querying itself. But the data returned contains the language, so the right repos might be filtered out based on it. So no cloning would be needed.

[dorner]

Hmm... language filter is a good first step but I'm afraid it wouldn't be sufficient because GitHub isn't always great about deciding what the "primary" language of a repo is. For example, one of our repos has a lot of generated documentation, so GitHub thinks the language is HTML instead of Ruby. :)

The repositories query doesn't allow for filtering by code, but we could use an additional check via the search API and filter out the repositories we use by the ones returned from that request.

[lindell]

The repositories query doesn't allow for filtering by code, but we could use an additional check via the search API and filter out the repositories we use by the ones returned from that request.

The problem is that the GitHub code searching is very rough. From what I've understood out of experience with it, both the query and the code is split by any seperator, and the parts in the query should exist in the code.

Meaning that a query like this: firstKey: firstValue will match this code:

firstKey: secondValue
secondKey: firstValue

While hello will not match

helloThere

---

This type of search might be something to add in the future. But I'm unfortunately not comfortable with adding it before seeing more cases where this might be useful.

I will leave the issue open for anyone else to comment on in the future 🙂

[dorner]

Hmm... my use case is nearly always a single string in a single file, and I haven't had any problems with my searches in the past 🤔 .

Still, I can see that this might not be the main use case for this tool. One other thing I might need to do is actually fork the repo before pushing, because in our company everyone has read access to everything but only write access to specific repos.

I'm wondering if there's some way to implement a general "pre-process" step? Or a "wrapper" around the command so that I can write my own repo that just calls this one but does the pre-processing before calling the command? So I could have my own code do the search and fork and then pass the set of repositories it found to multi-gitter?

[lindell]

One other thing I might need to do is actually fork the repo before pushing, because in our company everyone has read access to everything but only write access to specific repos.

I think this is a good addition. I don't personally use this kind of workflow where multi-gitter is useful to me, but it should still support it 🙂 But I think it's a separate issue and should be discussed separately.

I'm wondering if there's some way to implement a general "pre-process" step? Or a "wrapper" around the command so that I can write my own repo that just calls this one but does the pre-processing before calling the command? So I could have my own code do the search and fork and then pass the set of repositories it found to multi-gitter?

You could easily archive this with some bash magic 😉

echo my-cool-org/repo1,my-cool-org/repo2 | xargs multi-gitter run ./script -m "message" -R

But just replace echo my-cool-org/repo1,my-cool-org/repo2 with a script that returns a list of the repos you want to run on.

[dorner]

Ah, so it takes a comma-delineated list - I wasn't sure about that based on the documentation. Maybe that's something that can be made a bit clearer?

This helps a lot because I could even provide a UI for searching, and pipe the output of that UI into the command. 👍

[lindell]

Since it is not clear exactly what the filtering of this issue would be, and there have been different types of filters added since, I will close this issue, in favor of potential more specific filter issues in the future.