OpenBSD installation improvement suggestions

[0x1eef]

Bug

gem install nokogiri fails on OpenBSD when trying to extract a libxml2 archive (.xz). NOKOGIRI_USE_SYSTEM_LIBRARIES=yes gem install nokogiri works. It would be nice if a plain gem install nokogiri didn't fail, either by defaulting to using system libraries on OpenBSD or by extracting the archive properly (xzcat libxml2-2.9.14.tar.xz | tar xf - should work).

$ cat ext/nokogiri/tmp/x86_64-unknown-openbsd7.2/ports/libxml2/2.9.14/extract.log
tar: unknown option J
usage: tar {crtux}[014578befHhjLmNOoPpqsvwXZz] [blocking-factor | archive | replstr] [-C directory] [-I file]
[file ...] tar {-crtux} [-014578eHhjLmNOoPpqvwXZz] [-b blocking-factor]
[-C directory] [-f archive] [-I file] [-s replstr] [file ...]

Thanks.

[0x1eef]

@flavorjones

I saw this issue has been brought up a few times. Sorry about that. It would be nice to have a solution in place that "just worked", or that made the path to fixing it more obvious.

Maybe the build process could detect this error, and offer a handful of solutions at the console. There seems to be at least two that are practical, xzcat libxml2-2.9.14.tar.xz | tar xf - is probably too much of a special case based on what I saw in mini_portile.

• pkg_add gtar, alias tar=gtar, gem install nokogiri
• NOKOGIRI_USE_SYSTEM_LIBRARIES=yes gem install nokogiri

[0x1eef]

My preferred option is NOKOGIRI_USE_SYSTEM_LIBRARIES=yes gem install nokogiri, the gtar solution is a bit invasive and personally I'd prefer to let the system package manager take care of dependencies rather than nokogiri.

[0x1eef]

These are my thoughts on a possible improvement. The error could be caught during the build process, and the following suggestions given:

Suggestion 1

• Install "gtar"
• ~~Do not ask the user to alias tar to gtar, but edit mini_portile to check for the presence of "gtar" instead.~~ (mini_portile has this check already)

Suggestion 2

• Depend on system libraries instead of letting nokogiri build them.
• Suggest each lib that will be needed (pkg_add libiconv, etc)

Let me know what you think, and if it is a practical solution. Thanks.

[flavorjones]

@0x1eef I'm sorry you are having problems. I'd love, ABSOLUTELY love, if someone who uses OpenBSD would be willing to submit documentation improvements to https://github.com/sparklemotion/nokogiri.org or submit patches to extconf.rb.

Note for future readers: OpenBSD installation is documented at https://nokogiri.org/tutorials/installing_nokogiri.html#openbsd-7

I'd be open to extconf.rb checking for gtar and notifying users. Based on my experience dealing with support issues for the last 14 years, defaulting to system libraries is unlikely to be better for most users and will increase my personal workload and stress.

[0x1eef]

Based on my experience dealing with support issues for the last 14 years, defaulting to system libraries is unlikely to be better for most users and will increase my personal workload and stress.

No worries. I trust your judgement on that.

Note for future readers: OpenBSD installation is documented at https://nokogiri.org/tutorials/installing_nokogiri.html#openbsd-7

I think the issue is that it's not apparent where to find the documentation, or know-how on the remedy for this situation. I found the documentation through another GitHub issue, and I only became aware of $NOKOGIRI_USE_SYSTEM_LIBRARIES by reading the source code.

I'd be open to extconf.rb checking for gtar and notifying users.

What do you think about recommending gtar, but also making people aware of the option to use system libraries instead ? I think there's a visibility issue, and solving it in extconf.rb might save people some time.

Cheers!

[0x1eef]

pkg_add libiconv gtar alias tar=gtar gem install nokogiri

Are these instructions up to date ? Based on what I saw in mini_portile, it would seem that the alias is unnecessary. I also found that nvm suggested a similar approach, but the alias wasn't picked up by the shell it spawned. I'm not sure if that applies here.

[flavorjones]

Are these instructions up to date?

Honestly I don't know, I can't easily run a FreeBSD system in development. What you see in the docs (and in extconf.rb, and in mini_portile) is almost entirely a maintainer trying to interpolate bug reports from FreeBSD users who aren't fully engaged in helping solve the problems well.

So now is probably a good time for me to say: Thank you for being engaged on this! I appreciate your time and attention! ♥ ♥ ♥

Could you, as a FreeBSD user, help by contributing either documentation improvements or runtime improvements? We do run CI on FreeBSD (see here for an example) but those feedback loops are a bit slow.

[0x1eef]

Honestly I don't know, I can't easily run a FreeBSD system in development. What you see in the docs (and in extconf.rb, and in mini_portile) is almost entirely a maintainer trying to interpolate bug reports from FreeBSD users who aren't fully engaged in helping solve the problems well.

Alright, I would like to help. I can test locally and see what I find. I will confirm if the alias is needed or not first. Keep in mind I am running OpenBSD, but I guess there could be some similarity with FreeBSD as well.

Please bare with me as I am toggling a few different projects, and a new job, so my available time is a bit restricted. This issue is on my radar.

[flavorjones]

Ah, apologies, see, I'm not even familiar enough to know the differences between the distros. We definitely could use your help!

[0x1eef]

Hi !

I plan to look into this issue before the end of this weekend. Apologies for the delay.

[flavorjones]

@0x1eef did you ever get a chance to follow up on this?

[0x1eef]

I'm really sorry. I lost track of it. I have access to both Free|OpenBSD now. I plan to look into it within the hour, and get back to you.

[montdidier]

I'm really sorry. I lost track of it. I have access to both Free|OpenBSD now. I plan to look into it within the hour, and get back to you.

Has it been an hour yet?

[0x1eef]

It certainly has, and then some. I have too much on my plate to work on this in the foreseeable future. I set:

export NOKOGIRI_USE_SYSTEM_LIBRARIES=true

in ~/.kshrc instead of downloading & building libxml on the fly.

[stevecheckoway]

I'm reopening this because I think we can simplify the installation instructions.

FreeBSD

I just checked a fresh install of FreeBSD and

$ pkg install ruby
$ gem install nokogiri

successfully installed Nokogiri with the packaged libraries. Subsequently,

$ pkg install pkgconf libxml2 libxslt
$ gem install nokogiri -- --use-system-libraries

installed Nokogiri with the system libraries with no problems.

OpenBSD

I also checked with OpenBSD. The instructions for installing with system libraries works.

The instructions for using the packaged libraries as well as the section "tar and xz files" mentions making a shell alias of alias tar=gtar. This alias will never be used because it's not the shell calling tar, it's mini_portile.

The actual instructions for installing on OpenBSD 7 (I didn't check any earlier versions) should be

$ pkg_add libiconv gtar
$ gem install nokogiri

This correctly installs Nokogiri (although it called the binary nokogiri31 for some reason).

Detecting tar support for .tar.xz files

mini_portile looks for gtar before tar so as long as the user has installed gtar on OpenBSD, installation should work correctly.

However, it might be nicer to detect support for xz files rather than it failing with an inscrutable error message about the J option not being supported in some text that has scrolled off screen. Here's one approach that mini_portile could take to doing that.

EMPTY_TAR_XZ = [
  0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00, 0x00, 0x04, 0xe6, 0xd6, 0xb4, 0x46,
  0x02, 0x00, 0x21, 0x01, 0x16, 0x00, 0x00, 0x00, 0x74, 0x2f, 0xe5, 0xa3,
  0xe0, 0x05, 0xff, 0x00, 0x50, 0x5d, 0x00, 0x3c, 0x00, 0x33, 0x50, 0x18,
  0x68, 0x0d, 0x1a, 0x62, 0xa5, 0x78, 0x72, 0xb4, 0x50, 0xe0, 0x51, 0xf4,
  0xe0, 0xd5, 0xc9, 0x1c, 0x09, 0x56, 0x1a, 0x02, 0x8b, 0x6e, 0x43, 0xf6,
  0x31, 0xd5, 0x4e, 0xa7, 0xae, 0xf3, 0x87, 0x3a, 0xe4, 0x11, 0xc0, 0x12,
  0x68, 0x77, 0xd6, 0xa6, 0xca, 0x68, 0x11, 0xe6, 0xfc, 0x85, 0x4e, 0x1b,
  0x3b, 0xe0, 0x38, 0xd2, 0x41, 0xb7, 0xb5, 0xa0, 0xaf, 0xaf, 0x24, 0x5b,
  0xf5, 0x71, 0xee, 0x7c, 0x3e, 0xcc, 0xa4, 0x4a, 0x39, 0x97, 0x34, 0xb7,
  0x57, 0xa9, 0x00, 0x00, 0x32, 0xef, 0x0d, 0x00, 0x7a, 0x25, 0x51, 0xd1,
  0x00, 0x01, 0x6c, 0x80, 0x0c, 0x00, 0x00, 0x00, 0xd6, 0x14, 0x53, 0x47,
  0xb1, 0xc4, 0x67, 0xfb, 0x02, 0x00, 0x00, 0x00, 0x00, 0x04, 0x59, 0x5a
].pack('c*')

IO.popen(['tar', 'Jtf', '-'], 'w', out: '/dev/null', err: '/dev/null') do |io|
  io.write(EMPTY_TAR_XZ)
end

if $? == 0
  puts(".tar.xz supported")
else
  puts(".tar.xz not supported")
end

(Edit: Those redirects should probably be to File::NULL.)

The easiest thing to do would be for mini_portile to error out early with the message that the version of tar detected does not support xz and to install gtar. A better approach may be to suggest installing xz instead and then run

xz --decompress --stdout archive.tar.xz | tar xf -

but this means detecting tar support for xz, detecting the presence of xz, and then changing mini_portile to execute multiple commands with stdout from one piped to stdin in the other. But ruby has like 5 different ways to spawn processes and I'm not sure which one to use for this use case off hand.

[0x1eef]

Nice work @stevecheckoway ... OpenBSD can also use system libraries.

# pkg_add libxml libxslt
$ gem install nokogiri -- --use-system-libraries

And that avoids the gtar issue altogether.

[flavorjones]

Can someone please make a PR to https://github.com/sparklemotion/nokogiri.org/blob/main/docs/tutorials/installing_nokogiri.md to reflect what works best for BSD systems?

I'm also open to changes upstream in mini_portile if someone wants to send a PR there to better handle xz files.