gimp-appimage
gimp-appimage copied to clipboard
aferrero2707
Build on official GIMP infrastructure
It would be great if this could eventually be made official by the GIMP team. I asked on Twitter, and they said for this to become official, it would need to be built on the official GIMP infrastructure. @schumaml suggested to join #gimp on IRC to kick off the process.
https://twitter.com/probonopd/status/988170965224960000
Relevant discussion also here: https://twitter.com/Mister_Dolus/status/988615705514987520
@probonopd, ask @prokoudine
P.S.: GIMP 2.10 released!
- https://github.com/aferrero2707/gimp-appimage/issues/10
@probonopd I already proposed a couple of times on the gimp_developer mailing list to make my AppImage packages more "official", but never really got feedback on how to proceed in practice.
I need to understand how much work would be to port my Travis CI scripts to the GIMP build environment. If they support Docker containers than it should not be too difficult, as I am using a custom container in the Travis case.
Unfortunately I can rarely spend time on IRC, for me emails work much better because my available time is very scattered.
This is unfortunate, because IRC works best for us. I hope schumaml and you can work something out.
Ok, let's say we could maybe arrange an appointment? I do not want to look like I am trying to block things out... what are the "usual" time slots for GIMP discussions on IRC?
We are usually around between 6pm and 1am in +2:00GMT (think Berlin). This week is different, because part of the team is at LGM in Spain.
Hi!
I already proposed a couple of times on the gimp_developer mailing list to make my AppImage packages more "official", but never really got feedback on how to proceed in practice.
I must have missed this, mostly because I don't see all ML emails (as many, I'm subscribed to too many lists). The best is to make bug reports, then it won't disappear slowly as time passes. And I read all bug reports (at least the newly created ones).
Basically the requirement to make your AppImage official are:
- Made with scripts which have to be publicly committed to our repository, and therefore can be reviewed by anyone.
- Documented (just a "howto" or "README" text file next to the script which explains how to build, or main specific points of the build).
- Supported! This is maybe the single most important point. When there will be bug reports on our bug tracker, we expect you to fix what needs to be. Of course, everyone has one's own schedule, we are not asking for "busy work schedule", just "don't drop a binary on us then run away", that's all. :-) We are also not asking for you to promise to stay with us for dozens of years (life happens! We never know of tomorrow! This is also why the second point "documentation" is important so that someone may continue your work if you could not stay with us). But at least don't plan to abandon us a few weeks after you drop an AppImage on us please.
In other words: don't make our life difficult by giving us more work. Contributing is not about giving code, it is actually about giving time. Many people can do the first, few do the second.
Last point is that ideally builds are made on a "neutral" place, which means done by the automated scripts (which you would have committed in the repo) on a server controlled by the team. This may be a bit difficult since time is a bit missing on our side. Actually we have been accepting binaries built on contributors machine for years, but the reason was also that we have known these contributors for years too (so that's a trust thing).
Anyway as you say:
If they support Docker containers than it should not be too difficult, as I am using a custom container in the Travis case.
I know we can do this (because I have been told so).
So I propose such course of action:
(1) Make a patch against our source tree to add your scripts and howto texts in there (build/appimage/ is where it makes sense) and make a bug report where you'd upload your patch.
(2) Prepare your docker container meant to run your script and we'll see how we can have this run.
Thanks for the hints!
Regarding the support, at the moment I do not expect to step away, and I have been supporting the packages for several months.
One issue I am aware of and I am trying to fix is the fact that python plugins don't work on some systems.
Just to understand, when you say: " Last point is that ideally builds are made on a "neutral" place, which means done by the automated scripts (which you would have committed in the repo) on a server controlled by the team. This may be a bit difficult since time is a bit missing on our side."
I wonder if the current setup would be acceptable, as it is based on Travis CI and packages are automatically uploaded. No private computer is involved in the process, except for updating the Docker container when needed. Everything is documented in public github repositories, including the Docker configuration, so anybody can in principle take over in case I would not be available for any reason.
and I have been supporting the packages for several months.
I know and that's a good point for you, because (1) it means you are dedicated (2) your build has been used quite a bit, and that adds to the trust of publishing binaries. :-)
One issue I am aware of and I am trying to fix is the fact that python plugins don't work on some systems.
Ok. Well I don't mind non-perfect builds. As long as we are transparent about the issues (the flatpak also has a few issues; I need to make a post about this) and that we try to improve this with time (continuous support!).
I wonder if the current setup would be acceptable, as it is based on Travis CI and packages are automatically uploaded. No private computer is involved in the process, except for updating the Docker container when needed. Everything is documented in public github repositories, including the Docker configuration, so anybody can in principle take over in case I would not be available for any reason.
I am not an admin, but from what you say, yes it looks great. Don't take my word for a promise because I am not in charge of server administration, but I could see 2 ways to make the situation viable: (1) either you move your setup to one of our server (2) or you give some other GIMP people access to your server. We would probably not do much. That's just about "being able to check if something is wrong". Obviously (2) would be the faster solution because it would not rely on some other contributors (who already really lack time) to step in very actively.
I am not an admin, but from what you say, yes it looks great. Don't take my word for a promise because I am not in charge of server administration, but I could see 2 ways to make the situation viable: (1) either you move your setup to one of our server (2) or you give some other GIMP people access to your server. We would probably not do much. That's just about "being able to check if something is wrong". Obviously (2) would be the faster solution because it would not rely on some other contributors (who already really lack time) to step in very actively.
I also would prefer solution (2), as it would minimize the amount of work (and I think we are all short in free time). Travis CI is AFAIK a trusted and reliable build infrastructure, and I have no problem at all to give full access to this repository to any interested/relevant GIMP developer.
What is probably missing is some automated mechanism to upload the packages to the GIMP file servers, so that they become easy to access.
Note that I am using a similar setup to maintain the AppImage packages of few other projects, namely:
- PhotoFlow, my own RAW image editor
- RawTherapee
- HDRMerge
- LuminanceHDR
The first two, together with GIMP, are the more advanced ones (I would say production-ready), the last two still need a bit of polishing (only in the build infrastructure, the AppImage packages are working well in all cases AFAIK).
Meanwhile, I will put some short documentation of the build process in the README file of the repository.
Glad to see this coming together.
Supported! This is maybe the single most important point. When there will be bug reports on our bug tracker, we expect you to fix what needs to be. Of course, everyone has one's own schedule, we are not asking for "busy work schedule", just "don't drop a binary on us then run away", that's all. :-)
@aferrero2707 has a long-standing, proven track record over at pixls.us where he as been working on GIMP and other AppImages and has been interacting with users very interactively for quite some time. :+1:
https://www.google.com/search?q=pixls.us+appimage
So I'm really confident that the opposite of "drop a binary on us then run away" will happen :-)
Having a docker file that is set up to install all the required packages, clone and build GIMP and anything else and then the packages would be nice.
There is one in git master already: that one gets Debian testing and then just installs all packages for a GIMP build. Treat it as a proof of concept and not finished in any way, it's just a test to see whether we can build on our virtual machine. It's also not related to build.gimp.org in any way. This was done in about two hours and unless we want to base images on it, its usefulness might be over.
Do not make this overly complicated - on one end it is just someone without any package system experience, but access to a docker instance which can easily put things to download.gimp.org. On the other it would be people like you who could answrr my stupid questions about why a build of a packaging format does what it is doing.
On April 28, 2018 4:49:58 PM GMT+02:00, probonopd [email protected] wrote:
Glad to see this coming together.
Supported! This is maybe the single most important point. When there will be bug reports on our bug tracker, we expect you to fix what needs to be. Of course, everyone has one's own schedule, we are not asking for "busy work schedule", just "don't drop a binary on us then run away", that's all. :-)
@aferrero2707 has a long-standing, proven track record over at pixls.us where he as been working on GIMP and other AppImages and has been interacting with users very interactively for quite some time. :+1:
https://www.google.com/search?q=pixls.us+appimage
So I'm really confident that the opposite of "drop a binary on us then run away" will happen :-)
-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/aferrero2707/gimp-appimage/issues/9#issuecomment-385181517
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Travis CI is AFAIK a trusted and reliable build infrastructure, and I have no problem at all to give full access to this repository to any interested/relevant GIMP developer.
Actually after this sentence, I just realized that Travis CI was a third-party service. Somehow I thought (mistakenly) it was just a software for continuous integration (and you installed it on a server of your own). That makes it even more straightforward as long as the service keeps a trustworthy track.
We'll have to discuss but somehow I consider it may be better to continue delegate all the building part to such third-party service (same as our flatpak is built by flathub, which makes it a lot easier because a lot less to care about). Well… at least in our case where we are lacking a bit of contributor time.
@schumaml
Having a docker file that is set up to install all the required packages, clone and build GIMP and anything else and then the packages would be nice.
There is one in git master already: that one gets Debian testing and then just installs all packages for a GIMP build. Treat it as a proof of concept and not finished in any way, it's just a test to see whether we can build on our virtual machine. It's also not related to build.gimp.org in any way. This was done in about two hours and unless we want to base images on it, its usefulness might be over.
I think that some technical explanation can be useful to explain how things are done. One of the advantages of the AppImage packages is that they allow to run bleeding-edge software also on relatively old systems, for which up-to-date packages are not available. This is achieved by compiling the software on the oldest supported system, in the GIMP case Ubuntu 14.04. Due to the lack of up-to-date dependencies, this requires compiling many packages from sources, hence the use of an ad-hoc Docker container. In this case, I am using jhbuild to handle the source dependencies when preparing the container.
The Travis CI configuration fetches the Docker container and the build scripts, and uses the container to run (almost) all the build steps. Hence, each Travis job fetches the BABL/GEGL/GIMP sources as well as several popular GIMP plugins, compiles the code and bundles all the required binaries, libraries and configuration files into the AppImage structure. Once ready, the AppImage is uploaded back to the dedicated GitHub release page.
The Travis jobs are configured to automatically run once per week. If the build fails I get an email notification, otherwise the new packages automagically appear in the release page without any need of human intervention.
@Jehan my feeling is that relying on a third-party service like Travis is better than using GIMP's own build servers, even just for releasing some unneeded workload from the team...
@Jehan @schumaml by the way, if you are interested we could push the mechanism even further, and do the same for Windows and OSX packages.
Notice that I am routinely building my PhotoFlow Win and OSX packages on Travis, and we have been testing the same for RT Win builds as well. The Win packages are cross-compiled with ming-w64 using crossroad, which you should know very well ;-). The OSX ones are based on homebrew, and shipped as a standard OSX application bundle.
I must confess that I am a big fan of such automated package creation, because it releases a lot of repetitive workload from the developers. Do you think it would be worth to open a bug report to discuss about this ideas in a proper way?
@aferrero2707 We are interested by Windows/macOS continuous integration. This is something we have often be hoping to have one day, but nobody ever had the time to do this (and I fear nobody from current team will ever have if we don't get some new blood).
The Win packages are cross-compiled with ming-w64 using crossroad, which you should know very well ;-).
Ahahah. Yeah, I do. :P Nice to know this is used. ;-)
As for Win/macOS packages (assuming you mean "stable packages" for releases on our download page, not just CI), we'd be happy to automatize them a bit too though you'd have to discuss with our current Windows installer maintainers (Jernej and Ell) and macOS maintainer (Kris) because a lot of work have been done over the year to make these packages as good as possible (it is not just about building the software). We don't want to waste these efforts. It would be really better if you could come discuss this on IRC.
Let's do one thing at a time: would it be possible to give me keys to the build system for AppImage, just enough to review things and check that everything is sane? I am pretty sure everything is fine, but with just written discussion, we won't go anywhere. I guess if I can see things, we can go forward.
would it be possible to give me keys to the build system for AppImage
What do you mean by "keys to the build system"? The builds are running in public on Travis CI https://travis-ci.org/aferrero2707/gimp-appimage/ using https://github.com/aferrero2707/gimp-appimage/blob/master/.travis.yml
Well I don't really know how things work here. I was thinking that seeing from the inside how it is set up would be helpful. But from what you are saying, I guess we can do this without account, which is even better (the less account the easier).
For the record: I am not an admin and don't know much about CI (apart from what it does, but not how it is set up) and am only doing a review because someone has to step up if we want to get this done. Otherwise, I can tell you: I would prefer to not do this! Coding is 100 times much funnier! :P
Anyway I'll have a look when I can have some time. Hopefully some time soon.
@Jehan I have updated the README.md file of the repository to describe the current building procedure, and added you as a collaborator.
Please do not hesitate to ask questions if something is not clear...
Hi @aferrero2707 ! I really hoped I could make some time to review and understand how this all works, but history shows us I have not been able in the last month. :P
I am unsure when I will, but I would really suggest you to come on IRC and discuss with other contributors. In particular maybe with Sam Gleske (samrocketman on IRC) who is our CI guy and is very active lately.
Or maybe you could make a patch/merge request with all the files which need to go in our repository (put them under build/appimage/, you'll see there are already various build directories under build/: for flatpak, Windows, etc.) and a good overview on how this would work for the AppImage.
Though really we would also appreciate if you could come to IRC sometimes too. I am not a big IRC person as well, and therefore am not there all the time. But I still make appearances a few times a week because I just know it makes things easier. If we include your AppImage officially, we'll work as a team, which means you'll have to support it in our gitlab, and if sometimes we need to discuss things, being available on IRC is much better, etc.
I will try to show up on IRC during next weekend, if time allows... meanwhile, there is one basic aspect that needs to be sorted out: do we want the AppImage CI script to reside in the GIMP trunk, or in a separate repository? The second option has the advantage that the same set of scripts can be used to build in parallel several branches, while the first option has the advantage of keeping everything in a single place... For most of the projects I am sticking to external scripts.
Unless the AppImage is a very complicated setup with hundreds of files or whatever (I hope it's not the case! Many files is what would make difficult to review and understand; the flatpak for instance is all in a single, and quite short/understandable file), I'd prefer that the scripts are inside the main GIMP repo. Yet it could be in a separate repository. If so, that should be one hosted by us, alongside the GIMP repo (for instance, we have the gimp-web repo for the website or the gimp-help-2 repo for the documentation, etc.).
The second option has the advantage that the same set of scripts can be used to build in parallel several branches
I don't understand this one. Why wouldn't it be possible to build several branches if the scripts were all in the main GIMP repo. I used to have several flatpak manifests (building different branches, up to 3!) in the GIMP repo, so that I could indeed build various branches.
@aferrero2707 has cleaned up the repository root during our discussion on IRC yesterday evening - many thanks for that.
A similar cleanup to the modulesets and patches would make it easier to spot what is happening - most of the contents there are copied straight from the jhbuild and shouldn't be relevant for building GIMP at all.
For an official AppImage, I'd like to see all third-party plug-ins in an optional build step which is skippable.
@aferrero2707 @probonopd I have documented how to provision Jenkins locally. One of the benefits of the new Jenkins infrastructure is that one is able to have a complete copy of https://build.gimp.org/ running locally on their own development machine. Here are some tips:
- I have updated the overall project docs: https://gitlab.gnome.org/World/gimp-ci/documentation
- I have documented how to provision Jenkins locally: https://gitlab.gnome.org/World/gimp-ci/jenkins
TL;DR instructions
Assuming you have Vagrant and VirtualBox installed.
git clone --recursive https://gitlab.gnome.org/World/gimp-ci/jenkins.git
cd jenkins/
export VAGRANT_JENKINS=1
vagrant up
./jenkins_bootstrap.sh
^^ will automatically provision a VM and inside of the VM: install docker, pull down the GIMP docker image, install and start Jenkins. jenkins_bootstrap.sh will configure Jenkins and automatically start the build pipelines to create a GIMP artifact.
Developing locally inside of Vagrant
I still need to document how to develop in Vagrant. But basically you can SSH to the Vagrant VM the following ways.
# SSH into the VM with vagrant command
vagrant ssh
# alternately SSH into vagrant VM using SSH command
vagrant ssh-config > ssh_config
ssh -F ./ssh_config default
If you add ssh_config to ~/.ssh/config you can push and pull git source code to the default host. For example, if I wanted to develop jenkins-dsl I would apply the following patch to https://gitlab.gnome.org/World/gimp-ci/jenkins
diff --git a/configs/job_generator.xml b/configs/job_generator.xml
index 206ae71..cfcb778 100644
--- a/configs/job_generator.xml
+++ b/configs/job_generator.xml
@@ -9,7 +9,7 @@
<configVersion>2</configVersion>
<userRemoteConfigs>
<hudson.plugins.git.UserRemoteConfig>
- <url>https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git</url>
+ <url>/tmp/jenkins-dsl</url>
</hudson.plugins.git.UserRemoteConfig>
</userRemoteConfigs>
<branches>
diff --git a/configs/shared-pipelines.groovy b/configs/shared-pipelines.groovy
index 0e02bdd..e57153d 100644
--- a/configs/shared-pipelines.groovy
+++ b/configs/shared-pipelines.groovy
@@ -5,7 +5,7 @@ pipeline_shared_libraries = [
'allowVersionOverride': false,
'includeInChangesets': true,
'scm': [
- 'remote': 'https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git'
+ 'remote': '/tmp/jenkins-dsl'
]
]
]
Don't forget to run ./jenkins_bootstrap.sh to update the configuration of Jenkins.
Inside of the Vagrant VM I would create a bare repository at /tmp/jenkins-dsl.
vagrant ssh
mkdir /tmp/jenkins-dsl
cd /tmp/jenkins-dsl
git init --bare
Then in my locally cloned copy of jenkins-dsl I would add it as a development remote to push code for testing out new pipelines.
git clone https://gitlab.gnome.org/World/gimp-ci/jenkins-dsl.git
cd jenkins-dsl/
git remote add vagrant vagrant@default:/tmp/jenkins-dsl
# push your code to the Vagrant VM so Jenkins uses your development copy.
git push vagrant master
I went ahead and wrote up a local development guide. https://gitlab.gnome.org/World/gimp-ci/documentation/blob/master/develop-locally.md
@samrocketman from my side, I will continue cleaning up this repository as well as the docker-trusty-gimp one, and introduce the required changes such that the AppImage creation can run entirely inside the docker container.
Once this is ready, it will be possible to run the docker container on any CI system supporting Docker, and this will provide the final AppImage artefact, ready for uploading in some official place.
Hi!
As said on the mailing list, I'd be interested to resume discussing you going upstream. To be clear, one of the main issue is that your build has so many files and scripts that it's hard to review, unlike flatpak or snap where there is basically a single easy manifest file to review (which can be done in a few minutes).
Anyway… can you confirm me the start point is mkappimage.sh? (that's what I understood from .travis.yml)
Also you won't be able to package the additional (non-core) plug-ins when you go upstream. Releasing third-party plug-ins basically makes us liable and we'd have to support these. This is not possible (it is already hard to support our current core plug-ins with the few we are). You are welcome to continue providing a separate AppImage on the side with all the plug-ins you want, but the official AppImage has to be bare.
Also is the built GIMP as complete as possible? Does it have all the core plug-ins? Looking at build-appimage-centos7.sh, I see very few dependencies installed (but maybe they are installed otherwise).
We want a bare GIMP, yet obviously as full-featured as possible (the flatpak for instance is full-featured, apart from whatever the sandbox prevents or what I could not resolve yet).
About package-appimage.sh, what is this move_blacklisted() function for?
I see you clone the appimage-helper-scripts.git, but for security reason, it is preferred to not depend on non-upstream files (this makes harder to review too, makes non-reproducible builds if the contents change, may break more easily, etc.).
I actually noticed the same issue when calling upload_rotate.sh downloaded remotely as well.
Basically apart from well known projects, it is preferred to not download remote files in build scripts.
Also as a general rule, even when downloading from known upstream location, you should check the downloaded files. In particular it means to check a tarball against an expected checksum (SHA256 or similar preferably) and to check commit hashes too when cloning git repositories. Currently your script are completely subject to man-in-the-middle attacks and this won't do.
Moreover why are you saying our "desktop file is a bit strange"? Our desktop file is validated in unit tests and is perfectly fine AFAIK. And even if it is not (bugs are always possible!), then we must fix whatever needs to be. Don't just override our desktop file. Even more as it took a lot of work and dedication from our translators to translate all fields in 81 languages and you are basically just throwing away all this work on a whim. So please use the official desktop file (and let's figure out any issue together if needed) for official builds. Also don't rename the application. In the desktop file, it is called "GNU Image Manipulation Program" not "GIMP AppImage".
I will have probably a lot more to say. As I said, there are so many files, and I have not really looked in details to how there are all linked. This comment is only the result of a really quick review based on a 5-minute check, following the execution flow starting from .travis.yml (as you say this is the starting point of Travis build in your README). Some of the files, I haven't even seen yet when you run them.
Anyway that's already quite a lot to improve/clean/clarify.
@Jehan Let's go through your questions and comments, one-by-one...
can you confirm me the start point is
mkappimage.sh?
Yes, that's correct.
You are welcome to continue providing a separate AppImage on the side with all the plug-ins you want, but the official AppImage has to be bare.
This is already the case for the latest packages. For example, GIMP_AppImage-git-2.10.7-20180913-x86_64.AppImage is the bare package, while GIMP_AppImage-git-2.10.7-withplugins-20180913-x86_64.AppImage ships additional plug-ins.
About package-appimage.sh, what is this move_blacklisted() function for?
Obsolete, I will remove it
I see you clone the appimage-helper-scripts.git, but for security reason, it is preferred to not depend on non-upstream files
I will incorporate the scripts directly into the gimp-appimage repository, so that changes can be traced more easily.
I actually noticed the same issue when calling upload_rotate.sh downloaded remotely as well.
This helper script is only needed to upload the packages from Travis and back to GitHub. If we go for a different CI/hosting system, this will certainly change. But I can add this script to the gimp-appimage repository as well, if preferred.
to check commit hashes too when cloning git repositories.
Is this also needed when cloning the git HEAD? In this case the commit hash is not known/predictable... For example, jhbuild allows to check the hash of downloaded tar archives, but AFAIK considers git clones as trustworthy.
why are you saying our "desktop file is a bit strange"?
Again, I suspect this is some old relic. I will test with the official desktop file and see if I get problems. Probably not...
Let me do the clean-up, I will post a message here when finished.
Anyway, it will be good if we can progress on this!