[API] No reliable way to query for remote status of multiple packages

[samhocevar]

I have a list of packages, e.g. "git vim vlc", and I would like to get their respective version information from the online repositories. Ideally this would be through the API, but I realise choco doesn’t seem to provide this either.

Summary of issues

• choco search without arguments takes a very long time and downloads a lot of data
• choco search without arguments does not actually return all packages
• --only-id only works with a single string or substring
• --page does not appear to reduce the server load or transferred data amount
• --id-starts-with omits a lot of results
• --page does not appear to work with --id-starts-with

Description of attempts

Using the command line I can do something like this in bash:

for pkg in git vim vlc; do choco search -r -e --by-id-only $pkg; done

With the following output:

git|2.28.0
vim|8.2.1651
vlc|3.0.11

The drawback is that this performs one query per package. If I have 50 packages this becomes cumbersome. With more packages it becomes unrealistic (my IP has been banned from chocolatey.org) and there doesn’t seem to be a middle ground between exact search and “list all packages” that suits my simple need.

I can search for all packages and filter manually using grep:

choco search -r | grep '^\(git\|vim\|vlc\)|'

But this query takes forever (70 seconds) and downloads about 90 MiB of data from the server. Also, plenty of packages are missing (as reported in #2001) and this appears to be non-deterministic.

Chunking the search results in pages (choco search -r --page-size=100 --page=0) to reduce the load on the server actually takes exactly as long; it’s as if paging was performed on the client side, making it useless for that purpose.

I noticed that vlc and vim both start with v so I thought I’d search for packages that start with v and filter the results, to reduce the work on the server:

choco search -r --id-starts-with v | grep '\<\(vim\|vlc\)\>'

But while choco indeed returns 21 packages starting with v, they neither include vim nor vlc. This may be related to #1836 and the fact that --id-starts-with causes choco to return significantly fewer packages than usual (in contrast, choco search -r | grep '^v' returns 365 packages). So I thought, maybe these packages are on the second result page:

choco search -r --id-starts-with v --page 1

But this returns zero results. So maybe I can increase the page size:

choco search -r --id-starts-with v --page-size 2000

But this still returns the same 21 packages as before.

Feature requests

The help for choco search says the argument is a “filter” but does not specify what a filter is. If I understand correctly, it is just a verbatim string (or substring). I think such a syntax extension could be helpful:

choco search -r -e --by-id-only git vim vlc      # multiple arguments for search
choco search -r -e --by-id-only git,vim,vlc      # comma-separated filter
choco search -r -e --by-id-only 'git|vim|vlc'    # regular expression
choco search -r -e --by-id-only '(git|vim|vlc)'  # alternate regex syntax

Note that the Debian package manager allow the following convenient invocation: apt search --names-only '^(git|vim|vlc)$'

This may also be what #1345 is about, but the request is a bit unclear to me. There is also mention of --order-by-recent but I believe this hasn’t been implemented yet.

[ferventcoder]

I think there is some discussion to be had here for sure on a better way to keep queries down and also ensure results coming back are more appropriate.

Also, yes, there is some paging performed on the client side with the nuget.core library.

[samhocevar]

FYI, I believe I have found an elegant OData query that does exactly what I want:

$filter=(Id ne null and IsLatestVersion and(tolower(Id)eq'vlc'or tolower(Id)eq'git'or tolower(Id)eq'vim'))

See the result here:

https://chocolatey.org/api/v2/Packages()?$filter=(Id%20ne%20null%20and%20IsLatestVersion%20and(tolower(Id)eq%27vlc%27or%20tolower(Id)eq%27git%27or%20tolower(Id)eq%27vim%27))