Debian packaging

[cinemast]

This is a discussion thread for bringing sigal into Debian. I'll add comments and links during the review.

[cinemast]

We currently have troubles mostly with the themes folder. Colorbox is mostly fine, because we can replace it with existing dependencies already in the Debian archive.

Galleria seems to be packageable as a separate dependency, but this might take additional time. The same goes for photoswipe.

Once we have taken care of the JavaScript dependencies, there shouldn't be much trouble left. All the python dependencies are already packaged.

[siccegge]

How are things? need some help?

[cinemast]

I currently lost focus on that. Situation is still the same. If you'd like to help, any time.

[saimn]

Thanks for your input @cinemast @siccegge - Help is always welcome, so if you want to tackle this it's great!

[anarcat]

@cinemast can you share your work so far? i'd like to explore that packaging too and it would be great not to waste efforts. ;)

[anarcat]

i have file a Request For Packaging (RFP) on Debian's side to make sure no further duplication of effort occurs, see 879239 in the Debian BTS.

[anarcat]

We currently have troubles mostly with the themes folder. Colorbox is mostly fine, because we can replace it with existing dependencies already in the Debian archive.

Galleria seems to be packageable as a separate dependency, but this might take additional time. The same goes for photoswipe.

Once we have taken care of the JavaScript dependencies, there shouldn't be much trouble left. All the python dependencies are already packaged.

I've also encountered similar issues. Debian frowns upon duplicate copies of code, so normally, the jquery in sigal should be symlinked against the libjs-jquery package shipped in Debian. Except that version is 3.3, so probably not compatible with Sigal. Even worse, sigal ships two different copies of jquery.

More importantly, the unminified source code of those libraries need to be available somewhere in the tarball. This seems to be a rather broader problem: all the .js files are minified, not only on install, but also in the source. It would be best to have the non-minified versions in the source and have them minified at build time. Would you welcome a patch to perform such an operation?

Otherwise the Debian package will have to ship unminified copies of all those libraries, which will make the debian package needlessly diverge...

[anarcat]

also, it looks like all those javascript libraries are out of date, except colorbox...

[cinemast]

@anarcat I have never written any actual packaging code. The problem is as you said outdated dependencies in Debian. First thing would be to update them first by filing bugs at Debian.

[anarcat]

On 2017-10-26 02:18:41, Peter Spiess-Knafl wrote:

@anarcat I have never written any actual packaging code. The problem is as you said outdated dependencies in Debian. First thing would be to update them first by filing bugs at Debian.

er. sorry, which dependencies are out of date?

in my investigation, i found that the javascript code packaged with sigal was out of date, not the python dependencies (all) already packaged in debian.

in fact, when i installed with pip on stretch here, all depdendencies were satisfied with system packages.

so no problem on debian's side.

[saimn]

Great to see interest on this! I'm aware of the Debian requirements, which I understand though it's also cumbersome, but let's try to move forward on this (I would be very happy to see sigal included in Debian):

Even worse, sigal ships two different copies of jquery.

Just because the themes are independent, so not updated at the same time, but jquery's version should not be an issue.

More importantly, the unminified source code of those libraries need to be available somewhere in the tarball.

Yep, because (at least until now) including the source seems useless for these widely know librairies, and it increases the tarball size etc. But it can changes if needed (but Debian needs the sources but will not use it since you will use the jquery/etc. package).

This seems to be a rather broader problem: all the .js files are minified, not only on install, but also in the source. It would be best to have the non-minified versions in the source and have them minified at build time. Would you welcome a patch to perform such an operation?

Yes, seems a good solution. The current way is just because it was easy to do, and these files do not need to be minified at each build, and good practises recommend (recommended?) to use only one minified file to minimize the number of requests. I will try to have a look soon to see what's the best solution (maybe just concatenating the files is enough, the amount of js/css code specific to sigal is not that big).

also, it looks like all those javascript libraries are out of date, except colorbox...

Yeah, I try to update the librairies before a release, and have to ship a new version when I find the time. And last time I tried to update galleria there were changes needed in the css which was broken (that's always my fear when I update that it breaks something as it's not easily testable).

[anarcat]

I completely understand the desire to keep the source tarball small. I would make the distinction, however, between the source repository (this git repo and automatically generated tarballs) and the installed binary (the pip wheel). The repository can contain the full source but the wheel can just contain compressed copies.

(In my Python projects, in fact, I only publish the wheel on PyPI, and rely on the source control (e.g. github, gitlab, etc) to generate the tarball. Otherwise there are two tarballs which are usually different because they are built with a different process, which makes it hard to make builds reproducible.)

In fact, thanks to git submodules, you could not include the actual sources of the javascript libraries at all, and just have submodules pointing to those various repositories. That way your source repo is even smaller than it currently is - you just track metadata, as it should be.

I would also consider looking at NPM or whatever Javascript is doing these days to manage this stuff. I am blissfully ignorant of that as I generally try to stay away from JS in general, but in this case there's a large enough variety that it may be a good idea to use a little more heavy-handed approach.

Now as to specific comments:

Just because the themes are independent, so not updated at the same time, but jquery's version should not be an issue.

What do you mean here? Shouldn't plugins/themes use a single version of jquery?

But it can changes if needed (but Debian needs the sources but will not use it since you will use the jquery/etc. package).

So Debian does not absolutely need the source - if the compressed version is not actually in the source! This is why having submodules could be interesting here: Debian could use the actual, unmodified Sigal tarball, but then use existing Debian packages for the compressed versions of those Javascript libraries.

Now the problem, as I mentioned, is that there isn't necessarily a packaged version of all those libs in Debian. And even if there is, it's usually not the same version as in Sigal... There are ways to manage those changes in Debian (we call that a "transition", e.g. when GCC goes from version 4 to 5, things need to happen across the board), but we'd first need to get "synchronised" with the latest upstream, both in Sigal and in Debian, for this to start on the right foot. Most of Debian is up to date, as far as I can tell, for the libraries that are packaged.

So the work on Debian side would be to package the missing libraries, while the work in Sigal side would be to update the libraries to the latest upstream versions across the board and see how to remove them from the tarball. The latter part is optional: Debian packaging can remove them by hand or just add the source in the debian/ subdirectory for compliance.

For what it's worth, I made an inventory of the libraries used in Sigal here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879239#17

Of the JS dependencies, only jquery and jquery-colorbox are packaged. The other libraries are not packaged at all, so presumably we could get a free pass here and just ship the source with the Debian package for those without having to update them to synchronize them with upstream or Debian. It's really Jquery that's the biggest challenge here, I believe.

And last time I tried to update galleria there were changes needed in the css which was broken (that's always my fear when I update that it breaks something as it's not easily testable).

I hear you. :) It must be quite a challenge to keep up!

Have you considered splitting themes out of the distribution somehow? They could be individual projects with their own release schedules. You could have a single one shipped by default with Sigal and the others would be optional (installable through pip of course!)... The Sopel IRC bot uses this approach for plugins and I find it is an elegant approach to that complex issue...

That way you'd have to mostly worry about a single theme for the core of the project, and "contrib" themes would be more "loose" and wouldn't have as hard a requirement to be kept up to date...

Anyways, thanks for your feedback and openness regarding those issues. I know it can be annoying to have to deal with each distro's peculiar requirements, and I truly appreciate the time you are taking to accommodate Debian's policies.

[saimn]

Sorry for lagging behind (holidays, work etc.).

About submodules, it could be a solution but in this case I prefer not, because it adds the need to clone 3 (currently) more git repos, with their history that can be big, and moreover because currently sigal just copy what's in the theme directory into the output directory. Same for NPM, it's not worth the hassle to add a dependency to this. The simplest solution is to include the js source file, which is fine.

I was also wondering how you'll manage to use js files from other deb packages (jquery etc.) ? What would be a practical solution for this, do you how it is done for other packages like for instance pelican ?

What do you mean here? Shouldn't plugins/themes use a single version of jquery?

Not necessarily, different js libraries can have different requirement on the jquery version. And for the same reason as above (theme files copied to the output) it's easier to have a theme directory with everything inside. So on sigal it's not an issue to handle multiple version. But anyway I don't think the version of jquery can be an issue, so using the one provided by debian should be fine.

Have you considered splitting themes out of the distribution somehow? They could be individual projects with their own release schedules.

With only 3 themes I think it's not worth the effort. And it's more convenient to have everything installed at once. It could be a solution if there was more contributed or external themes, but not sure it will happen. Also having everything at the same place is more convenient to maintain the themes.

Anyways, thanks for your feedback and openness regarding those issues. I know it can be annoying to have to deal with each distro's peculiar requirements, and I truly appreciate the time you are taking to accommodate Debian's policies.

Thank you as well for taking time for this. As you probably noticed, on my side the free time I can dedicate to sigal is small, so I will always prefer straightforward solutions and try to keep the maintenance easier :wink: . But I hope we can find good compromises and make this happen. I will try to work on updating the themes and make a new release. I'm summarizing below the tasks needed, please tell me is you see something else.

• [ ] Debian: package the missing libraries if possible (or get a free pass here and just ship the source with the Debian package ?)
• [ ] Sigal: update the libraries to the latest upstream versions. And by the same occasion add the js/css source files.
• [ ] Sigal: see how to remove/override libraries for the Debian tarball, ideas welcome. (optional: Debian packaging can remove them by hand or just add the source in the debian/ subdirectory for compliance).

[anarcat]

On 2017-11-14 20:57:48, Simon Conseil wrote:

Sorry for lagging behind (holidays, work etc.).

No problem, of course. :)

About submodules, it could be a solution but in this case I prefer not, because it adds the need to clone 3 (currently) more git repos, with their history that can be big, and moreover because currently sigal just copy what's in the theme directory into the output directory. Same for NPM, it's not worth the hassle to add a dependency to this. The simplest solution is to include the js source file, which is fine.

Okay. As long as the source files are there, there should be no issue with Debian.

I was also wondering how you'll manage to use js files from other deb packages (jquery etc.) ? What would be a practical solution for this, do you how it is done for other packages like for instance pelican ?

The way this is usually handled is that, in the binary package, the files are replaced by symlinks from the other packages.

I don't exactly know how the Pelican package handles this, but from what I can tell, it removed one of the themes because the files were non-free, for example:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868047

I don't know about the duplicate copies if code. Maybe that was just missed in the reviews.

What do you mean here? Shouldn't plugins/themes use a single version of jquery?

Not necessarily, different js libraries can have different requirement on the jquery version. And for the same reason as above (theme files copied to the output) it's easier to have a theme directory with everything inside. So on sigal it's not an issue to handle multiple version. But anyway I don't think the version of jquery can be an issue, so using the one provided by debian should be fine.

I see. Well, I'm not very familiar with Javascript stuff, to be honest. I would have assumed that different versions may not be compatible with each other, but we can probably just try and bump it up.

[...]

Anyways, thanks for your feedback and openness regarding those issues. I know it can be annoying to have to deal with each distro's peculiar requirements, and I truly appreciate the time you are taking to accommodate Debian's policies.

Thank you as well for taking time for this. As you probably noticed, on my side the free time I can dedicate to sigal is small, so I will always prefer straightforward solutions and try to keep the maintenance easier :wink: . But I hope we can find good compromises and make this happen. I will try to work on updating the themes and make a new release. I'm summarizing below the tasks needed, please tell me is you see something else.

• [ ] Debian: package the missing libraries if possible (or get a free pass here and just ship the source with the Debian package ?)
• [ ] Sigal: update the libraries to the latest upstream versions. And by the same occasion add the js/css source files.
• [ ] Sigal: see how to remove/override libraries for the Debian tarball, ideas welcome. (optional: Debian packaging can remove them by hand or just add the source in the debian/ subdirectory for compliance).

That looks good. I think you can probably get a free pass for libs that are not yet in Debian, as long as source is shipped in the source distro.

As for the third step, push that back to the Debian folks, don't worry about it. It's typical Debian packaging work and we should deal with the hassle, not you...

a.

-- Celui qui ne connaît pas l'histoire est condamné à la revivre. - Karl Marx

[anarcat]

i think #291 and #281 are work in progress for step two here, which is to update to the latest libraries and add source files (possibly removing minified ones as well).

[anarcat]

so, progress report:

• [x] Sigal: ship the Javascript libraries' source code (#281)
• [x] Sigal: update Javascript libraries (#281, #296, 23a44e1552e5a72cc1231a6a4572368f71f2b097, etc)
• [ ] Sigal: minify at the gallery build stage (#291, optional?)
• [ ] Debian: package the missing libraries (see BTS#879239 for progress on that)
• [x] Sigal: remove the Javascript libraries from the tarball not necessary?
• [ ] Debian: create actual Debian package for Sigal, which means replacing the JS libraries on install

As I previously mentioned, I'm not sure step 3 is necessary here: we are free to ship duplicate copies of code in source tarballs, I believe, as long as binary packages don't actually use the duplicate copies but instead the already packaged libraries. This is essentially section 4.13: Convenience copies of code in the Debian policy manual, which states:

Debian packages should not make use of these convenience copies [...] If the included code is already in the Debian archive in the form of a library, the Debian packaging should ensure that binary packages reference the libraries already in Debian and the convenience copy is not used. If the included code is not already in Debian, it should be packaged separately as a prerequisite if possible.

By the way, the rationale for this is in a footnote in a policy:

Having multiple copies of the same code in Debian is inefficient, often creates either static linking or shared library conflicts, and, most importantly, increases the difficulty of handling security vulnerabilities in the duplicated code.

So I'll need to package Galleria, Photoswipe and so on in Debian. That shouldn't be too much trouble, but it's a hurdle that needs to be done. It's also possible that those libraries have dependencies on their own, unfortunately, which is where things get a little more complicated in Debian.

But that's not your concern, and I'll move that part of the conversation to the Debian bugtracker. The only nice thing that could happen here would be to have minification happen during the gallery build (#291) so that we can use whatever library is actually installed instead of assuming we build from the Sigal source. This could complicate things, so don't do it unless you feel it's necessary. I just thought it made sense considering that minification was skipped for Photoswipe in #281...

[anarcat]

Actually, there's still an issue with the js libraries... there are still two different versions of jQuery shipped because of inconsistencies between colorbox and galleria... the former is looking into the problem of updating jquery in https://github.com/jackmoore/colorbox/issues/834 so maybe that could get solved on its own.

there's also the problem that galleria ships an out of date copy of leaflet, which is behind the version in debian, so there are still javascript library updates to perform.

one solution, for debian's side of things, would be to simply not ship the galleria and colorbox themes for now while those things get fixed.. that would allow Sigal to enter Debian properly and then those issues can be fixed later to include more themes...

Those are the bugs to track packaging of the three themes:

• 891978 (photoswipe)
• 891977 (colorbox)
• 891975 (galleria)

There's also a discussion on Debian's side about packaging multiple jquery versions that is relevant to us as well (680282), although that's been open for a long time without any progress so I wouldn't hold my breath on that...

[saimn]

Sigal: minify at the gallery build stage (#291, optional?)

I think this can wait, it will be good to have but nothing critical.

there are still two different versions of jQuery shipped because of inconsistencies between colorbox and galleria...

yes, last time I tried colorbox was not working with Jquery 3 (I don't remember the details).

there's also the problem that galleria ships an out of date copy of leaflet,

looks like I forgot this one :open_mouth:

one solution, for debian's side of things, would be to simply not ship the galleria and colorbox themes for now while those things get fixed.. that would allow Sigal to enter Debian properly and then those issues can be fixed later to include more themes...

Hmm, having sigal with only one theme would be quite restrictive for users. I also understand how packaging these js libraries can be a difficult thing (I'm also having trouble to track all these dependencies to update them!) but it would be good to have all the themes. If needed it is also possible to do a small release of sigal with updated deps (leaaflet and maybe jquery 3 for colorbox if it works ?)

[anarcat]

On 2018-03-04 15:06:51, Simon Conseil wrote:

one solution, for debian's side of things, would be to simply not ship the galleria and colorbox themes for now while those things get fixed.. that would allow Sigal to enter Debian properly and then those issues can be fixed later to include more themes...

Hmm, having sigal with only one theme would be quite restrictive for users. I also understand how packaging these js libraries can be a difficult thing (I'm also having trouble to track all these dependencies to update them!) but it would be good to have all the themes. If needed it is also possible to do a small release of sigal with updated deps (leaaflet and maybe jquery 3 for colorbox if it works ?)

I understand it can be restrictive, but unfortunately until those libraries are fixed, I'm afraid it will be the only solution in the short term, or at least to get sigal passed the first vetting process. ;)

Now if we can figure out how to update colorbox to jquery3 and fix the depends on galleria, that's great too and we can move on... but i'm thinking of just getting one step done: it won't be a definitive limitation!

a.

[anarcat]

note: leaflet update in #310 is relevant here.

[anarcat]

furthermore, #300 added two python dependencies we should keep track of.