debug version parsing?

[tannevaled]

Hi,

i try to package portable-openssh i thought the version parsing would be easy but it seems it does not work as expected.

versions:
  url: https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/
  match: /.*\.tar\.gz/
  strip:
    - /^openssh-/
    - /.tar.gz$/
distributable:
  url: https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-{{version}}.tar.gz
  strip-components: 1
...

$ VERBOSE=1 DEBUG=1 pkg build openssh.com
...
 building openssh.com
++ resolve.ts openssh.com
{ versions: [] }
error: Uncaught Error: not-found: version: openssh.com
    throw new Error(`not-found: version: ${utils.pkg.str(spec)}`)
          ^
    at Object.resolve (file:///home/dd.linux/.pkgx/pkgx.sh/brewkit/v0.46.4/lib/usePantry.ts:43:11)
    at eventLoopTick (ext:core/01_core.js:183:11)
    at async file:///home/dd.linux/.pkgx/pkgx.sh/brewkit/v0.46.4/libexec/resolve.ts:15:15
+ PKG=

Is there a way to really debug the version parsing?

[jhheider]

I think what you're seeing is what's implemented at the moment. But the bug in your version key is visible to me:

  match: /.*\.tar\.gz/

is going to greedy-match way too much stuff. you want:

  match: /openssh-\d+\.\d+p1\.tar\.gz/
    strip:
    - /^openssh-/
    - /p1.tar.gz$/

[tannevaled]

in fact you can have something like openssh-9.3p2.tar.gz

 match: /openssh-\d+\.\d+p\d+\.tar\.gz/
 strip:
    - /^openssh-/
    - /.tar.gz$/

does not work.

[mxcl]

I've struggled here before too, it would be nice if the matching code would provide more debug info.

[jhheider]

in fact you can have something like openssh-9.3p2.tar.gz does not work.

agreed. we have no way to support this at present, other than the above, or hard-coding versions. what we need is a syntax for version transformation, but we don't have one, though it's been discussed a few times.

i chose p1 as the most prevalent.

[mxcl]

wouldn't be too hard to allow embedding a function:

versions:
  transformer: str => str.match(/\d+/)

As an example. The function should be more robust.

[tannevaled]

i hacked a little bit the brewkit code to have a better understanding of how it works (and some more debug output)

$ cat ~/.pkgx/pkgx.sh/brewkit/v0.46.4/lib/usePantry.getVersions.ts
...
async function handleAPIResponse({ fetch, ignore, strip }: APIResponseParams): Promise<SemVer[]> {
  const version_match: RegExp = /^V_\d+.*/
...
      const v = semver.parse(name)
      console.debug({pre_strip_name:pre_strip_name, name:name, v:v})
      if (!v) {
        if (version_match.test(pre_strip_name)) {
            #rv.push(pre_strip_name)
            rv.push(name)
        } else {
            console.debug({ignoring: pre_strip_name, reason: 'unparsable'})
        }
      } else if (v.prerelease.length <= 0) {
...
  console.debug({rv:rv})
  return rv

$ DEBUG=1 VERBOSE=1 pkg build openssh.com
...
+ for PKG in $PKGS
+ gum '# building openssh.com'
+ pkgx gum format -- '# building openssh.com'
  
  building openssh.com
++ resolve.ts openssh.com
error: Uncaught TypeError: a.compare is not a function
    return versions.filter(x => this.satisfies(x)).sort((a,b) => a.compare(b)).pop()
                                                                   ^
    at https://deno.land/x/[email protected]/src/utils/semver.ts:308:68
    at Array.sort (<anonymous>)
    at Range.max (https://deno.land/x/[email protected]/src/utils/semver.ts:308:52)
    at Object.resolve (file:///home/dd.linux/.pkgx/pkgx.sh/brewkit/v0.46.4/lib/usePantry.ts:40:30)
    at eventLoopTick (ext:core/01_core.js:183:11)
    at async file:///home/dd.linux/.pkgx/pkgx.sh/brewkit/v0.46.4/libexec/resolve.ts:15:15
+ PKG='{ pre_strip_name: "V_2_1_1_P3", name: "V.2.1.1.P3", v: undefined }
{ pre_strip_name: "V_2_1_1_P4", name: "V.2.1.1.P4", v: undefined }
{ pre_strip_name: "V_2_2_0_P1", name: "V.2.2.0.P1", v: undefined }

{
  rv: [
    "V_2_1_1_P3", "V_2_1_1_P4", "V_2_2_0_P1", "V_2_3_0_P1",  "V_2_5_0_P1",
    "V_2_5_1_P1", "V_2_5_1_P2", "V_2_5_2_P1", "V_2_5_2_P2",  "V_2_9_P2",
    "V_2_9_9_P1", "V_2_9_9_P2", "V_3_0_P1",   "V_3_0_1_P1",  "V_3_0_2_P1",
    "V_3_1_P1",   "V_3_2_2_P1", "V_3_2_3_P1", "V_3_3_P1",    "V_3_4_P1",
    "V_3_5_P1",   "V_3_6_P1",   "V_3_6_1_P1", "V_3_6_1_P2",  "V_3_7_P1",
    "V_3_7_1_P1", "V_3_7_1_P2", "V_3_8_P1",   "V_3_8_1_P1",  "V_3_9_P1",
    "V_4_0_P1",   "V_4_1_P1",   "V_4_2_P1",   "V_4_3_P1",    "V_4_3_P2",
    "V_4_4_P1",   "V_4_5_P1",   "V_4_6_P1",   "V_4_7_P1",    "V_4_9_P1",
    "V_5_0_P1",   "V_5_1_P1",   "V_5_2_P1",   "V_5_3_P1",    "V_5_4_P1",
    "V_5_5_P1",   "V_5_6_P1",   "V_5_7_P1",   "V_5_8_P1",    "V_5_8_P2",
    "V_5_9_P1",   "V_6_0_P1",   "V_6_1_P1",   "V_6_2_P1",    "V_6_2_P2",
    "V_6_3_P1",   "V_6_4_P1",   "V_6_5_P1",   "V_6_6_P1",    "V_6_7_P1",
    "V_6_8_P1",   "V_6_9_P1",   "V_7_0_P1",   "V_7_1_P1",    "V_7_1_P2",
    "V_7_2_P1",   "V_7_2_P2",   "V_7_3_P1",   "V_7_4_P1",    "V_7_5_P1",
    "V_7_6_P1",   "V_7_7_P1",   "V_7_8_P1",   "V_7_9_P1",    "V_8_0_P1",
    "V_8_1_P1",   "V_8_2_P1",   "V_8_3_P1",   "V_8_4_P1",    "V_8_5_P1",
    "V_8_6_P1",   "V_8_7_P1",   "V_8_8_P1",   "V_8_9_P1",    "V_9_0_P1",
    "V_9_1_P1",   "V_9_2_P1",   "V_9_3_P1",   "V_9_3_P2",    "V_9_4_P1",
    "V_9_5_P1",   "V_1_2_PRE3", "V_1_2_PRE4", "V_1_2_PRE5",  "V_1_2_PRE6",
    "V_1_2_PRE7", "V_1_2_PRE8", "V_1_2_PRE9", "V_1_2_PRE10", "V_1_2_PRE11",
    ... 38 more items
  ]
}'

To allow an arbitrary version case don't we need to provide a function for the version sorting (lexicographic sort for example)? The transformer seems not enough.

[mxcl]

All versions must be converted to semantic versions for pkgx, which is what we propose the transformer does.

We acknowledge some open source projects don't map great onto semver, but it is a good standard that we think we should force.

[tannevaled]

ok. cf https://semver.org/ considering pX should be the patch number, in this case something like V_9_5_P1 should be translated to 9.5.1 or 9.5.0-p1 to compare with 3.6.1-p2 (V_3_6_1_P2)?

[mxcl]

yes

• 9_5_p1 is 9.5.1
• 3.6.1-p2 is 3.6.1.2 which is not valid semver but our systems support it since it's common enough for projects to keep adding numbers.