#2503 push SNAPSHOT Maven artifacts to github Packages

[icrc-fdeniger]

trafficstars

Fixes #2503

Description Publish Maven artifacts in GitHub Packages

Type Feature

Checklist

• Tests done https://github.com/icrc-fdeniger/android-fhir
• Maven packages are deployed here: https://github.com/icrc-fdeniger?tab=packages&repo_name=android-fhir ( will be done on google/android-fhir)
• maven-repository.zip artifact is deployed as usual ( not sure about the usage of this artifact but not changes).
• See https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment

Versions

• When the maven artifacts are deployed in GitHub Packages the SNAPSHOT prefix is used to follow Maven coventions.
• When the maven artifacts are deployed in official maven repositories, the SNAPSHOT prefix is not used ( as it's done actually as far as I can understand)
• In the PR, the artifact versions has been updated to next expected release version:
  • for instance, commonmodule will be automatically deployed with the version 0.1.0-alpha06-SNAPSHOT in GitHub Packages and with 0.1.0-alpha06 in the next release.

Suggestion/question why not removeing beta/alpha as versions like 0.X.Y are considered as development version ?

Current Version	Used in this PR	Proposition
common:0.1.0-alpha05	common:0.1.0-alpha06	common:0.1.1
engine:1.0.0	engine:1.0.1	engine:1.0.1
data-capture:1.1.0	data-capture:1.1.1	data-capture:1.1.1
workflow:0.1.0-alpha04	workflow:0.1.0-alpha05	workflow:0.1.1
ontrib-barcode:0.1.0-beta3	ontrib-barcode:0.1.0-beta4	ontrib-barcode:0.1.1
contrib-locationwidget:0.1.0-alpha01	contrib-locationwidget:0.1.0-alpha02	contrib-locationwidget:0.1.1
knowledge:0.1.0-alpha03	knowledge:0.1.0-alpha04	knowledge:0.1.1

if a dev Team wants to use an alpha/beta version it can use a dependency to a SNAPSHOT version ( the "unstable version created from the last build"). it's also possible to specify the artifact of a build with by example: engine-1.0.1-20240423.083712-3

[icrc-fdeniger]

Hello @santosh-pingle, @ktarasenko please tell me if this PR need to be modified. I understand that the version updates are the "tricky" part and will be happy to modify this. My point was to update them to the next version as GitHub Actions should produce SNAPSHOT of next versions.

Thanks

[vorburger]

@icrc-fdeniger FYI the artifact version number changes introduced here will cause build failures once you rebase this after #2533 is merged. The required fix (after you have rebased it, not right now) is as simple as changing those version numbers also in the (new, TBD) docs/use/api.md file; hopefully that's trivial. Please do shout if unclear!

@MJ1998 @santosh-pingle @ktarasenko merge this only after #2533.

[icrc-fdeniger]

@vorburger @santosh-pingle @ktarasenko should be ok now

[vorburger]

@vorburger @santosh-pingle @ktarasenko should be ok now

LGTM! I say let's merge this... @santosh-pingle @ktarasenko @MJ1998 no objects, right?

[MJ1998]

Thanks for this PR @icrc-fdeniger. LGTM.

About the version proposal we would wanna keep the alpha/beta naming conventions. Reason - Using alpha and beta suffixes aligns with established conventions in semantic versioning and open-source software development. So we would like to stick with it.

[icrc-fdeniger]

@santosh-pingle any news on that PR ?

[MJ1998]

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?

Yes it will overwrite.

Another thing we might need to be careful about is just any incurred cost - https://github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

I doubt if there will be a problem but we will beinformed by Google in case the quota reaches limit..

[icrc-fdeniger]

Thansk @icrc-fdeniger for this contribution! 🙏

For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only) Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:

repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages?

Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here: https://github.com/icrc-fdeniger/android-fhir/packages/2129825 It's a kind of feature as we can use a specific SNAHPSHOT Version.

So yes it could be an issue with the storage. It seems there are no perfect solution for this point: https://github.com/orgs/community/discussions/48971

[jingtang10]

Thansk @icrc-fdeniger for this contribution! 🙏 For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only) Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:

repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here: https://github.com/icrc-fdeniger/android-fhir/packages/2129825 It's a kind of feature as we can use a specific SNAHPSHOT Version.

So yes it could be an issue with the storage. It seems there are no perfect solution for this point: https://github.com/orgs/community/discussions/48971

is there a way to publish a github package on whatever is on HEAD? i feel like that's much more useful?

@icrc-fdeniger have you tested that the current local packages (what i mentioned above) are generated in your forked repo? I'm not seeing any successful github build actions?

See https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment

this run seems to have failied?

also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

[vorburger]

also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

How about, instead of capturing this with comments here, contributing (in this PR) clarifications for any questions related to all this in a snapshot.md (or whatever) doc in https://github.com/google/android-fhir/tree/master/docs/contrib? (And link it in mkdocs.yaml.

[icrc-fdeniger]

Thansk @icrc-fdeniger for this contribution! 🙏 For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only) Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:

repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here: icrc-fdeniger/android-fhir/packages/2129825 It's a kind of feature as we can use a specific SNAHPSHOT Version. So yes it could be an issue with the storage. It seems there are no perfect solution for this point: github.com/orgs/community/discussions/48971

is there a way to publish a github package on whatever is on HEAD? i feel like that's much more useful?

@icrc-fdeniger have you tested that the current local packages (what i mentioned above) are generated in your forked repo? I'm not seeing any successful github build actions?

See icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment

this run seems to have failied?

also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

it's failing because of the feature "git pages" that I didn't activate on my repo. I tested it on April on my branch master-deploy-repo. To do the PR I had to revert my "test setup" : https://github.com/icrc-fdeniger/android-fhir/commit/82934301f1b8301f5fa6b21a3ce15bffefaa3a54

You can have a look here: https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 and here:

The publication is working. If you want I can modify the github actions on my branch to show that the action will be launched for every push on master-deploy-repo and it's ok.

If the PR is accepted as is on the official repo, The publish task is launched automatically for all push to master only ( so when a PR is merged on master branch). For PR, the task will be skipped: https://github.com/icrc-fdeniger/android-fhir/blob/6cc76eec4476188bce55a455755968fa97a38cfb/.github/workflows/build.yml#L79

[icrc-fdeniger]

also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

How about, instead of capturing this with comments here, contributing (in this PR) clarifications for any questions related to all this in a snapshot.md (or whatever) doc in master/docs/contrib? (And link it in mkdocs.yaml.

@vorburger I should create a new file shapshot.md in docs/contrib and add a line related to this in mkdocs.yml and in the sections "Contributors". Am I correct ?

[vorburger]

@vorburger I should create a new file shapshot.md in docs/contrib and add a line related to this in mkdocs.yml and in the sections "Contributors". Am I correct ?

Yup, exactly; that's how you can contribute documentation! (If you want to, this is a suggestion.) You can name the MD file with any filename you like, obviously.

[icrc-fdeniger]

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

[vorburger]

another solution could be to use jfrog.io or https://oss.sonatype.org/ as a maven repository

Does the build just produce a "standard Maven repo layout" on the filesystem - with a directory structure à la https://repo1.maven.org/maven2/? Or could it be made to? If so, then why not simply copy that into the site/ directory during the build, and make it show up and available on https://google.github.io/android-fhir/ ? Just a thought.

[icrc-fdeniger]

another solution could be to use jfrog.io or oss.sonatype.org as a maven repository

Does the build just produce a "standard Maven repo layout" on the filesystem - with a directory structure à la repo1.maven.org/maven2? Or could it be made to? If so, then why not simply copy that into the site/ directory during the build, and make it show up and available on google.github.io/android-fhir ? Just a thought.

But will the artifcats be available via a Maven Repository like: https://maven.google.com/web/index.html?#com.google.android.fhir:engine ?

my objective is just to be able to use latest android-fhir artifacts in our Android Application in a usual way and via dependencies declaration :)

[jingtang10]

Thansk @icrc-fdeniger for this contribution! 🙏 For some background on the maven-repository.zip artifact, see this PR from @yigit: #249. We also have some documentation and I just realised it's a bit messed up after our recent documentation migration to this page: google.github.io/android-fhir/contrib but for your information the original text is this:

Use unreleased GitHub build (for development/testing only) Each GitHub build of the project also contains a maven repository. You can access unreleased features by downloading it and including it in your project. To acquire the maven repository, go the the actions page, and click on the build you want. In the build artifacts dropdown, you will find a maven-repository.zip file. After downloading and extracting it to a local folder, you can add it to your gradle setup via:

repositories {
  maven {
    url "file:///<path to the unzipped folder>"
  }
}

These artifacts are versioned as a combination of the current version + buildid (e.g. 0.1.0-alpha01-build_123). You can find the version in the zip file by checking the contents of it and update your build file to match that version.

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

the old version will be kept as you can see here: icrc-fdeniger/android-fhir/packages/2129825 It's a kind of feature as we can use a specific SNAHPSHOT Version. So yes it could be an issue with the storage. It seems there are no perfect solution for this point: github.com/orgs/community/discussions/48971

is there a way to publish a github package on whatever is on HEAD? i feel like that's much more useful? @icrc-fdeniger have you tested that the current local packages (what i mentioned above) are generated in your forked repo? I'm not seeing any successful github build actions?

See icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 for a full deployment this run seems to have failied? also can you update the description to clarify when the packages are generated? is it on every github action run? will different branches and prs overwrite each other's packages?

it's failing because of the feature "git pages" that I didn't activate on my repo. I tested it on April on my branch master-deploy-repo. To do the PR I had to revert my "test setup" : icrc-fdeniger@8293430

You can have a look here: https://github.com/icrc-fdeniger/android-fhir/actions/runs/8798380904/job/24145256498 and here:

The publication is working. If you want I can modify the github actions on my branch to show that the action will be launched for every push on master-deploy-repo and it's ok.

If the PR is accepted as is on the official repo, The publish task is launched automatically for all push to master only ( so when a PR is merged on master branch). For PR, the task will be skipped: https://github.com/icrc-fdeniger/android-fhir/blob/6cc76eec4476188bce55a455755968fa97a38cfb/.github/workflows/build.yml#L79

thanks for the detailed explanation - this sounds all good to me.

[jingtang10]

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

@bashir2 do you push any project to maven central? i can't remember.

[jingtang10]

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

i have been considering this actualy - perhaps we can have a discussion at the next android fhir sdk developers call. let me konw if you can attend @icrc-fdeniger (also @vorburger).

[jingtang10]

@icrc-fdeniger as a follow up - will you be able to add a little to the doc site on how to use the github packages? thanks!

[jingtang10]

actually one last thing before merging - i feel we're really publishing a snapshot of the HEAD, instead of any particular versions... I couldn't find a way to do this but when we're publishing github packages can we just use the build id or something that is not tied to our version number for public releases?

@icrc-fdeniger @vorburger

[MJ1998]

actually one last thing before merging - i feel we're really publishing a snapshot of the HEAD, instead of any particular versions... I couldn't find a way to do this but when we're publishing github packages can we just use the build id or something that is not tied to our version number for public releases?

@icrc-fdeniger @vorburger

I guess this is better because someone might be using a particular snapshot in their dependency and could upgrade it to the latest build snapshot which is available. On the other hand, using version will overwrite the snapshot and the the developer may get confused that even though they are using the latest snapshot available they are not getting all the features added. Only problem I see is that each PR will push a snapshot and this can grow huge... So we might need another GitHub action (if possible) to delete old snapshots?

[icrc-fdeniger]

@jingtang10 @vorburger documentation added.

[bashir2]

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or https://oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see https://github.com/icrc/openmrs-android-fhir/blob/main/README.md#setup).

@bashir2 do you push any project to maven central? i can't remember.

Sorry forgot to reply this earlier; yes we do: https://central.sonatype.com/search?q=com.google.fhir.gateway

@icrc-fdeniger I have not read all the discussions on this PR but IIUC you want to use SNAPSHOT versions of the android-fhir libraries in your dependent code. One approach that I have seen people use is to check-out/build locally if SNAPSHOT version of a dependency is needed. IIRC, OpenMRS is also set up that way. So for example, if I needed to change something in openmrs-core and then use that in development of one of OpenMRS modules, I would checkout core and do a mvn install locally (can be built as part of CI process too). Would that solve this problem?

[icrc-fdeniger]

OK, now onto github packages - with this change, do we always overwrite the SNAPSHOT? so if I have a PR in development, and I push a new commit it will overwrite the github packages? Another thing we might need to be careful about is just any incurred cost - github.com/features/packages#pricing. With our repo under google organisation I'm hoping that we'll be covered by the google organisation quota/allowance. But I will investigate.

@jingtang10: another solution could be to use jfrog.io or oss.sonatype.org as a maven repository as it's free for opensource project and it will be easier for other projects to integrate android-fhir as a SNAPSHOT dependency : we don't need to provide a Git Token to download artifacts ( see icrc/openmrs-android-fhir@main/README.md#setup).

@bashir2 do you push any project to maven central? i can't remember.

Sorry forgot to reply this earlier; yes we do: central.sonatype.com/search?q=com.google.fhir.gateway

@icrc-fdeniger I have not read all the discussions on this PR but IIUC you want to use SNAPSHOT versions of the android-fhir libraries in your dependent code. One approach that I have seen people use is to check-out/build locally if SNAPSHOT version of a dependency is needed. IIRC, OpenMRS is also set up that way. So for example, if I needed to change something in openmrs-core and then use that in development of one of OpenMRS modules, I would checkout core and do a mvn install locally (can be built as part of CI process too). Would that solve this problem?

Thanks @bashir2 for your answer. You are right: we want to use SNAPSHOT versions of android-fhir in our dependent code.

For OpenMRS ( given as an example but true for all projects :)) there are 2 main use cases:

1. the situation you describe: you want to modify openmrs-core and so you have to check-out/build locally. This use case is clear.
2. you want to test your application with the latest version of openmrs-core. In this case, the usual way is to use SNAPSHOT version to avoid redoing the full check-out/build process.

For the second point, for sure developpers can check-out/build the source locally but:

• It's a long process: I must be sure that all developpers in our team performed a pull/build locally of android-fhir. Maybe some of them will forget or a commit in android-fhir can break some of our features ( and we will have to checkout an earlier commit) , etc...We will have to deal with all these situations.
• With a SNAPSHOT dependency, the update will be automatically done and we can force it with the option --refresh-dependencies to be sure we have the latest. If a commit in android-fhir breaks our application, we can modify our dependencies to use a pinned snapshot version temporarely ( until the commit is fixed in android-fhir or our application is modified) and so all developpers can go on.
• Impossible or very complex to integrate this approach "check-out/build" in a CI Tool to launch integration tests or to generate demo "apk". This point is surely questionable but no choice for now if we want to use the latest improvements in FHIR engine concerning OpenMRS. As soon as a FHIR engine version is released and contains all features we need we will use this released version and then we will stop using SNAPHSOT for sometimes to release our application. And then we will prepare a v2 and use SNAPSHOT versions again to be sure we are up-to-date with latest dev ...

IMHO, deploying SNAPSHOT versions in Maven Central will be better than using GitHub packages ( no token needed and no extra cost). For now this PR is using GitHub Packages but will be happy the modify it to use Maven Central.

[vorburger]

to use Maven Central

Maven Central is... erm, "evolving" ("cough"), and I think they won't support SNAPSHOT repos in the future.

Just FYI, https://github.com/vorburger/ch.vorburger.exec/issues/212 (unrelated, just FYI).

[jingtang10]

to use Maven Central

Maven Central is... erm, "evolving" ("cough"), and I think they won't support SNAPSHOT repos in the future.

Just FYI, vorburger/ch.vorburger.exec#212 (unrelated, just FYI).

hi @vorburger - can you pls add more details on maven central's support for snapshot?

[vorburger]

hi @vorburger - can you pls add more details on maven central's support for snapshot?

My advice is to stay clear of Maven Central, for SNAPSHOT.

I advise to either use GitHub packages (with some auto clean), or simply a SNAPSHOT repo on the docs site.

[icrc-fdeniger]

tweaked a few things in the doc.

thanks so much @icrc-fdeniger for this! excited for this change!

thanks this change will help us :) Do I have something to do or just wait for the workflow to go on ?

[jingtang10]

tweaked a few things in the doc.

thanks so much @icrc-fdeniger for this! excited for this change!

thanks this change will help us :) Do I have something to do or just wait for the workflow to go on ?

No I've enabled auto merge so hopefully it'll go in once tests pass