Support multiple pkg repositories

[Freaky]

pkg supports an arbitrary number of active package repositories, but SameDrucker seems to assume There Can Be Only One🌩.

[dlangille]

What problems does this create?

I think you refer to this line:

https://github.com/dlangille/SamDrucker/blob/master/clients/samdrucker.sh#L53

Also, I think repo is getting the wrong values in that field.

Examples from my database. pkg should not be there at the start of each line.

samdrucker=# select distinct(repo) from host;
                                    repo                                     
-----------------------------------------------------------------------------
 pkg http://fedex.int.example.org/packages/120amd64-default-mysql56/
 pkg https://fedex.example.org/packages/121amd64-default-master-list/
 pkg http://fedex.example.org/packages/121amd64-default-master-list/
 pkg https://fedex.example.org/packages/121i386-default-master-list-i386/
 pkg http://fedex.int.example.org/packages/120amd64-default-pg96/
 pkg http://fedex.example.org/packages/120amd64-default-pg12/
 http://pkg.freebsd.org/FreeBSD:12:amd64/latest/
 pkg http://fedex.int.example.org/packages/120amd64-default-pg95/
 pkg http://fedex.int.example.org/packages/120amd64-default-mysql80/
 pkg http://fedex.int.example.org/packages/120amd64-default-unifi/
 pkg http://fedex.int.example.org/packages/120amd64-default-pg94/
 pkg http://fedex.int.example.org/packages/120amd64-default-pg11/
 pkg http://fedex.int.example.org/packages/120amd64-default-master-list/
 pkg http://fedex.int.example.org/packages/121amd64-default-master-list/
 pkg http://fedex.int.example.org/packages/120amd64-default-pg10/
 pkg http://fedex.int.example.org/packages/121amd64-default-pg12-121/
 pkg http://fedex.int.example.org/packages/120amd64-default-mysql57/
 pkg http://fedex.example.org/packages/120amd64-default-master-list/
(18 rows)

samdrucker=# 

[Freaky]

I think it mostly works by accident - jo happens to silently ignore arguments without = but that contain :, so feeding it bare URLs does nothing.

This looks to be an implementation detail, it's not documented behaviour from what I can see.

[dlangille]

In the past, I used multiple repos, but stopped. That has contributed to this.

[dlangille]

@jpmens see above where jo is mentioned.

[jpmens]

I noticed; thanks :)

[dlangille]

You are everywhere.

[jpmens]

Actually I thought you were referring to the README. Now I see @Freaky might have uncovered a small issue and/or something which isn’t documented, so if you think it’s a bug I’d appreciate a bug report and more so a fix. :)

[dlangille]

Also, I think repo is getting the wrong values in that field.

Examples from my database. pkg should not be there at the start of each line.

samdrucker=# select distinct(repo) from host;
                                    repo                                     
-----------------------------------------------------------------------------
 pkg http://fedex.int.example.org/packages/120amd64-default-mysql56/
...

samdrucker=# 

I think I see why. That value is actually:

[dan@pg02:~] $ repo=`/usr/sbin/pkg -vv | grep  url | cut -f2 -d \"`
[dan@pg02:~] $ echo $repo
pkg+http://fedex.int.unixathome.org/packages/121amd64-default-pg12-121/
[dan@pg02:~] $ 

It makes me think of url encoding issues.

[Freaky]

@jpmens This is my first time using jo, and I only skimmed the docs, but this does look unintentional considering other invalid argument forms produce a warning diagnostic on stderr:

-% jo foo:bar
{}
-% jo foobar
Argument `foobar' is neither k=v nor k@v
{}

[Freaky]

@dlangille Maybe you want --data-urlencode instead of just -d?

[dlangille]

@Freaky I was just trying that, but:

$ $CURL -d --data-urlencode "$SAMDRUCKER_ARG=$payload" -H "Content-Type: application/x-www-form-urlencoded" -X POST $SAMDRUCKER_URL
curl: (3) nested brace in URL position 204:
packages={
   "name": "pg02.int.example.org",
   "os": "FreeBSD",
   "version": "12.1-RELEASE-p1",
   "repo": "pkg+http://fedex.int.example.org/packages/121amd64-default-pg12-121/",
   "packages": [
      "SamDruckerClientShell-0.0.1",
      "anvil-0.0.17_2",
      "apr-1.7.0.1.6.1",
      "bash-5.0.11",
      "bind-tools-9.14.9",
      "ca_root_nss-3.49",
      "curl-7.68.0",
      "cyrus-sasl-2.1.27",
      "db5-5.3.28_7",
      "expat-2.2.8",
      "gdbm-1.18.1_1"
$

[dlangille]

If I manually urlencode payload before invoking curl, the database gets the correct results.

[dlangille]

I thought using -d (POST, content-type:application/x-www-form-urlencoded) would remove the need for urlencoding the data. I am seeing that was incorrect. Unless there is more curl can do here.

[Freaky]

Don't mix -d with --data-urlencode:

-% curl -d "foo={bar+[foo]}" https://voi.aagh.net/post.php -X POST
array(1) {
  ["foo"]=>
  string(11) "{bar [foo]}"
}

-% curl -d --data-urlencode "foo={bar+[foo]}" https://voi.aagh.net/post.php -X POST
curl: (3) nested brace in URL position 10:
foo={bar+[foo]}
         ^
-% curl --data-urlencode "foo={bar+[foo]}" https://voi.aagh.net/post.php -X POST
array(1) {
  ["foo"]=>
  string(11) "{bar+[foo]}"
}

[dlangille]

This works:

$ $CURL --data-urlencode "$SAMDRUCKER_ARG=$payload" -X POST $SAMDRUCKER_URL

[dlangille]

I think you've found it.

[jpmens]

I'm only half paying attention, but IMO URL encoding the data is not really needed, and passing all that data on the command ine might even blow some limit (MAX ARGS?) some time / some day.

If this were mine, I would run jo's output into a file and then curl up the raw data:

t=$(mkstemp /tmp/xxxxxx)
jo .... > $t
curl --data @${t} ...

The @ as first char in curl's --data or -d has it read the data from a file and uses that data uninterpreted.

[jpmens]

@Freaky there are likely a number of undocumented / unintentional cases in jo. :)

[dlangille]

This works:

$ $CURL --data-urlencode "$SAMDRUCKER_ARG=$payload" -X POST $SAMDRUCKER_URL

This works in release 0.0.2

[dlangille]

I'm only half paying attention, but IMO URL encoding the data is not really needed, and passing all that data on the command ine might even blow some limit (MAX ARGS?) some time / some day.

If this were mine, I would run jo's output into a file and then curl up the raw data:

t=$(mkstemp /tmp/xxxxxx)
jo .... > $t
curl --data @${t} ...

The @ as first char in curl's --data or -d has it read the data from a file and uses that data uninterpreted.

This works:

payload=$(mktemp /tmp/SamDrucker.payload.XXXXXX)
$JO -p name=$hostname os=$uname version=$version repo=$repo $pkg_args > $payload
$CURL --data-urlencode ${SAMDRUCKER_ARG}@${payload} -H "Content-Type: application/x-www-form-urlencoded" -X POST $SAMDRUCKER_URL

More importantly, so does this:

$CURL --data-urlencode ${SAMDRUCKER_ARG}@${payload} ${SAMDRUCKER_URL}

[jpmens]

I'm sure you have a reason for --data-urlencode even if I don't see it, but I haven't looked at the server part of all this.

FWIW, I typically use --data and save on encoding/decoding, but if you're using a form on the backend then this is fine.

[jpmens]

Ah, I see it here:

json_decode($_REQUEST['packages']);

yes, fine. Just FYI, if you prefer to avoid the encoding you can pick up the raw POST (sent with curl --data) with

$data = file_get_contents('php://input');
json_decode(...);

[dlangille]

I encoded because we're sending data in a URL. I thought urlencoding was the thing to do when sending data in a URL.

At present, we are just sending one parameter over, packages. I was leaving room for sending more items.

[dlangille]

Sorry, this got dropped. Not sure what is next.