OSGeo4W workflow: Archive artifacts

[HuidaeCho]

This is a follow-up PR from #692. In this PR, I'm testing artifact uploading to archive packages built by the workflow.

[HuidaeCho]

Hmm... Is it a critical error?

Total file count: 11200 ---- Processed file #496 (4.4%)
An error has been caught http-client index 1, retrying the upload
Error: read ECONNRESET
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:201:27) {
  errno: 'ECONNRESET',
  code: 'ECONNRESET',
  syscall: 'read'
}
Exponential backoff for retry #1. Waiting for 6490.789634257077 milliseconds before continuing the upload at offset 0
An error has been caught http-client index 0, retrying the upload
Error: Client has already been disposed.
    at HttpClient.request (D:\a\_actions\actions\upload-artifact\v2\dist\index.js:5600:19)
    at HttpClient.sendStream (D:\a\_actions\actions\upload-artifact\v2\dist\index.js:5561:21)
    at UploadHttpClient.<anonymous> (D:\a\_actions\actions\upload-artifact\v2\dist\index.js:7008:37)
    at Generator.next (<anonymous>)
    at D:\a\_actions\actions\upload-artifact\v2\dist\index.js:6742:71
    at new Promise (<anonymous>)
    at module.exports.608.__awaiter (D:\a\_actions\actions\upload-artifact\v2\dist\index.js:6738:12)
    at uploadChunkRequest (D:\a\_actions\actions\upload-artifact\v2\dist\index.js:7006:46)
    at UploadHttpClient.<anonymous> (D:\a\_actions\actions\upload-artifact\v2\dist\index.js:7043:38)
    at Generator.next (<anonymous>)
Exponential backoff for retry #1. Waiting for 5002.353774690806 milliseconds before continuing the upload at offset 0
Total file count: 11200 ---- Processed file #683 (6.0%)
Finished backoff for retry #1, continuing with upload

Looks like it's fine...

Total size of all the files uploaded is 301303239 bytes
Finished uploading artifact grass-osgeo4w64. Reported size is 301303239 bytes. There were 0 items that failed to upload
Artifact grass-osgeo4w64 has been successfully uploaded!

[HuidaeCho]

The artifacts download page shows the uncompressed size, not the actual download size. It can be confusing.

Artifact upload strips away the root directory when compressing. For example, uploading C:\OSGeo4W64 compresses directories under this root folder, creating the following directory structure:

OSGeo4W.bat
OSGeo4W.ico
apps/
bin/
cmake/
etc/
include/
lib/
opt/
share/
var/

instead of

OSGeo4W64/OSGeo4W.bat
OSGeo4W64/OSGeo4W.ico
OSGeo4W64/apps/
OSGeo4W64/bin/
OSGeo4W64/cmake/
OSGeo4W64/etc/
OSGeo4W64/include/
OSGeo4W64/lib/
OSGeo4W64/opt/
OSGeo4W64/share/
OSGeo4W64/var/

Also, is this compressed OSGeo4W64 folder recognized by the OSGeo4W64 installer when later one tries to upgrade this installation?

[HuidaeCho]

What is the format of artifact URLs? https://github.com/OSGeo/grass/suites/1725687151/artifacts/32660968 How can we download them programmatically?

[ninsbl]

What is the format of artifact URLs? https://github.com/OSGeo/grass/suites/1725687151/artifacts/32660968 How can we download them programmatically?

See https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#list-artifacts-for-a-repository

[HuidaeCho]

What is the format of artifact URLs? https://github.com/OSGeo/grass/suites/1725687151/artifacts/32660968 How can we download them programmatically?

See https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#list-artifacts-for-a-repository

That's cool. Thanks!

https://api.github.com/repos/OSGeo/grass/actions/artifacts

However, clicking url or archive_download_url doesn't start downloading.

• https://api.github.com/repos/OSGeo/grass/actions/artifacts/32660968
• https://api.github.com/repos/OSGeo/grass/actions/artifacts/32660968/zip

Clicking the latter URL says "You must have the actions scope to download artifacts." In https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#download-an-artifact, "GitHub Apps must have the actions:read permission to use this endpoint." Whatever that means...

[hellik]

Also, is this compressed OSGeo4W64 folder recognized by the OSGeo4W64 installer when later one tries to upgrade this installation?

see PackagingInstructions how OSGeo4W packages should be packaged.

Especially:

Note: Please use tar & bzip2 from osgeo4w's msys (%OSGEO4W_ROOT%\apps\msys\bin). Don't use 7zip as setup.exe can't understand the archives it creates, and there is mixed success with gnuwin32 bzip2 and cygwin bzip2. Linux tar/bzip2 has no reported problems.

and the naming convention:

Each package directory should have a single setup.hint file, and one or more versions of that package as a .tar.bz2 file. These should normally have names like gdal-1.5.0-7.tar.bz2 where "1.5.0" is the version of GDAL that was packaged, and "-1" indicates the version of the packaging attempt. The directory can also contain corresponding source snapshots named like gdal-1.5.0-7-src.tar.bz2. Note that the basename should exactly match the directory in which the package is found. So the "gdal" directory has gdal-<version>-<revision>.tar.bz2 package files. 

see http://download.osgeo.org/osgeo4w/x86_64/release/grass/grass/ and http://download.osgeo.org/osgeo4w/x86_64/release/grass/grass-daily/ for OSGeo4W winGRASS packages.

and see add "Windows batchfiles for use with R" to OSGeo4W where I were successfull to build a correct and working OSGeo4W package ;-)

[hellik]

and see https://github.com/landam/wingrass-maintenance-scripts for Martin's script to automatically build the OSGeo4W packages.

[HuidaeCho]

Also, is this compressed OSGeo4W64 folder recognized by the OSGeo4W64 installer when later one tries to upgrade this installation?

see PackagingInstructions how OSGeo4W packages should be packaged.

Especially:

Note: Please use tar & bzip2 from osgeo4w's msys (%OSGEO4W_ROOT%\apps\msys\bin). Don't use 7zip as setup.exe can't understand the archives it creates, and there is mixed success with gnuwin32 bzip2 and cygwin bzip2. Linux tar/bzip2 has no reported problems.

and the naming convention:

Each package directory should have a single setup.hint file, and one or more versions of that package as a .tar.bz2 file. These should normally have names like gdal-1.5.0-7.tar.bz2 where "1.5.0" is the version of GDAL that was packaged, and "-1" indicates the version of the packaging attempt. The directory can also contain corresponding source snapshots named like gdal-1.5.0-7-src.tar.bz2. Note that the basename should exactly match the directory in which the package is found. So the "gdal" directory has gdal-<version>-<revision>.tar.bz2 package files. 

see http://download.osgeo.org/osgeo4w/x86_64/release/grass/grass/ and http://download.osgeo.org/osgeo4w/x86_64/release/grass/grass-daily/ for OSGeo4W winGRASS packages.

and see add "Windows batchfiles for use with R" to OSGeo4W where I were successfull to build a correct and working OSGeo4W package ;-)

Thanks for the info. Currently, it's just a zipped OSGeo4W folder with GRASS under C:\OSGeo4W64\opt\grass, not a package. It's directly unzippable into C:\ assuming no pre-existing installation of OSGeo4W. We could create two different downloads.

1. Simply unzippable package including all required OSGeo4W files: No admin rights are required. My original question was about this package because this uncompressed OSGeo4W is not installed by its installer.
2. As an OSGeo4W package without other OSGeo4W files: Admin rights are required to install OSGeo4W.

[ninsbl]

However, clicking url or archive_download_url doesn't start downloading.

• https://api.github.com/repos/OSGeo/grass/actions/artifacts/32660968
• https://api.github.com/repos/OSGeo/grass/actions/artifacts/32660968/zip

Clicking the latter URL says "You must have the actions scope to download artifacts." In https://docs.github.com/en/free-pro-team@latest/rest/reference/actions#download-an-artifact, "GitHub Apps must have the actions:read permission to use this endpoint." Whatever that means...

Yes, in the API response you get a list of available artifacts with a download URL. In order to use that download URL, you need to create a Personal Access Token, I guess.

Here: https://github.community/t/access-to-github-actions-npm-registry-for-hostedtoolcache/17546/6 is a post that may cover the topic...

But others, like @wenzeslaus , know much more about Github Actions than I do...

[wenzeslaus]

For "daily" (or per-commit) builds, isn't the issue this: https://github.com/actions/upload-artifact/issues/21? However, for releases, there seems to be a way: https://github.com/softprops/action-gh-release. In any case, you can also upload from a job, e.g., to an FTP server.

[ninsbl]

Bumping up milestone as 8.0.1 is due in two days, while this has not been part of RC1 and there has not been activity for some time.

[HuidaeCho]

Maybe related: https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/making-authenticated-api-requests-with-a-github-app-in-a-github-actions-workflow

[echoix]

Do we now know what are the files that are needed to get out of the runner at the end of the job?

Does it need to be an osgeo4w feedable archive?

[echoix]

Once we know that info, it would be quite fast to refresh this PR (or a new one) with the current patterns

[jef-n]

Do we now know what are the files that are needed to get out of the runner at the end of the job?

Does it need to be an osgeo4w feedable archive?

No, the osgeo4w nightlies are run outside of github. package.sh is based on my original osgeo4w script and is now also used for building the OSGeo4W nightlies (and release packages).

https://github.com/jef-n/OSGeo4W/blob/5d2486c04fa6e12fa070de723b088f702c8a75ce/src/grass8/osgeo4w/package.sh#L110