Can not build gettext-sys on windows

[vsmawoex]

error: failed to run custom build command for gettext-sys v0.19.8 (N:\gettext-rs\gettext-sys) process didn't exit successfully: N:\gettext-rs\target\debug\build\gettext-sys-70cac4af73bac902\build-script-build (exit code: 101) --- stdout cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_SYSTEM cargo:rerun-if-env-changed=GETTEXT_SYSTEM cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_DIR cargo:rerun-if-env-changed=GETTEXT_DIR cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_BIN_DIR cargo:rerun-if-env-changed=GETTEXT_BIN_DIR cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_LIB_DIR cargo:rerun-if-env-changed=GETTEXT_LIB_DIR cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_INCLUDE_DIR cargo:rerun-if-env-changed=GETTEXT_INCLUDE_DIR OPT_LEVEL = Some("0") TARGET = Some("x86_64-pc-windows-msvc") HOST = Some("x86_64-pc-windows-msvc") CC_x86_64-pc-windows-msvc = None CC_x86_64_pc_windows_msvc = None HOST_CC = None CC = None CFLAGS_x86_64-pc-windows-msvc = None CFLAGS_x86_64_pc_windows_msvc = None HOST_CFLAGS = None CFLAGS = None CRATE_CC_NO_DEFAULTS = None CARGO_CFG_TARGET_FEATURE = Some("fxsr,mmx,sse,sse2") DEBUG = Some("true") running: "tar" "xJf" "N:\gettext-rs\gettext-sys\gettext-0.19.8.1.tar.xz" "--strip-components" "1" "--force-local"

--- stderr tar: Option --force-local is not supported Usage: List: tar -tf Extract: tar -xf Create: tar -cf [filenames...] Help: tar --help thread 'main' panicked at ' command did not execute successfully, got: exit code: 1

build script failed, must exit now', gettext-sys\build.rs:251:5 note: Run with RUST_BACKTRACE=1 environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish... error: build failed

[vsmawoex]

Any parent path does not have space in it.

environment: rustup 1.16.0 (beab5ac2b 2018-12-06) cargo 1.34.0-nightly (5c6aa46e6 2019-02-22

[cecton]

@vsmawoex did you manage to build it?

[Koka]

Looks like the problem is here https://github.com/Koka/gettext-rs/blob/master/gettext-sys/build.rs#L162 - seems like not all windows tar versions have '--force-local' option, not sure if we can remove it safely. It's better to ensure that you have GNU compatible tar - I believe MSYS2 allows you to install one.

[cecton]

On my Windows 10 I have the same error. I was already surprised that Windows has tar at all. Also I didn't have XZ so I installed it manually.

[cecton]

@Koka I will investigate the build as I have plenty of time now but I don't know much about compiling on Windows.

[Koka]

@cecton Thank you, actually I don't have either Windows machine, nor time to fix the bulid on it, and I'll greatly appreciate any help on this subject.

[Minoru]

I got a Windows 10 virtual machine from Microsoft and poked around a bit. Turns out Windows ships bsdtar, which is part of libarchive package. bsdtar indeed doesn't support --force-local, which is specific to GNU tar. bsdtar works fine without this option, but GNU tar doesn't: without the option, it tries to interpret C: in C:\path\to\build\directory as a hostname or something, and fails with a "couldn't resolve" error. MSYS ships GNU tar; so we either omit the option and break MSYS, or include the option and break Windows.

I think we can avoid all of this if we stop passing an absolute path into tar. The idea is: copy the tarball into OUT_DIR, cd into it and run tar xvf gettext-0.21.tar.xz.

I won't implement this myself, because apparently my environment is wrong somehow. If I set GETTEXT_SYSTEM env var, I can cargo build --target x86_64-pc-windows-gnu both gettext-sys and gettext-rs, but cargo test --target x86_64-pc-windows-gnu fails because of missing x86_64-w64-mingw32-gcc linker. At this point, I ran out of interest, and I think someone who's interested in Windows support should take over. It'll also be nice to have some docs and a CI job that can verify all of this, otherwise we'll just break it the next time we change something.

[shijunti19]

Can't compile on windows

[cecton]

Can't compile on windows

Can you provide the error message?

[shijunti19]

@cecton

PS F:\web\xiaolu\rust\gtk\a031> cargo run
   Compiling gettext-sys v0.21.0
   Compiling gobject-sys v0.15.0 (https://github.com/gtk-rs/gtk-rs-core#9769f7c3)
   Compiling cairo-sys-rs v0.15.0 (https://github.com/gtk-rs/gtk-rs-core#9769f7c3)
   Compiling graphene-sys v0.15.0 (https://github.com/gtk-rs/gtk-rs-core#9769f7c3)
error: failed to run custom build command for `gettext-sys v0.21.0`

Caused by:
  process didn't exit successfully: `F:\web\xiaolu\rust\gtk\a031\target\debug\build\gettext-sys-a5ce50a319ba3248\build-script-bu
ild` (exit code: 101)
  --- stdout
  /usr/bin/cmp
  /usr/bin/diff
  /c/Windows/system32/find
  /e/ProgramFiles/msys64/mingw64/bin/xz
  cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_DIR
  cargo:rerun-if-env-changed=GETTEXT_DIR
  cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_BIN_DIR
  cargo:rerun-if-env-changed=GETTEXT_BIN_DIR
  cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_LIB_DIR
  cargo:rerun-if-env-changed=GETTEXT_LIB_DIR
  cargo:rerun-if-env-changed=X86_64_PC_WINDOWS_MSVC_GETTEXT_INCLUDE_DIR
  cargo:rerun-if-env-changed=GETTEXT_INCLUDE_DIR
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-pc-windows-msvc")
  HOST = Some("x86_64-pc-windows-msvc")
  CC_x86_64-pc-windows-msvc = None
  CC_x86_64_pc_windows_msvc = None
  HOST_CC = None
  CC = None
  CFLAGS_x86_64-pc-windows-msvc = None
  CFLAGS_x86_64_pc_windows_msvc = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  DEBUG = Some("true")
  running: "tar" "xJf" "D:\\rust\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\gettext-sys-0.21.0\\gettext-0.21.tar.xz" "
--strip-components" "1" "--force-local"

  --- stderr
  tar: Option --force-local is not supported
  Usage:
    List:    tar -tf <archive-filename>
    Extract: tar -xf <archive-filename>
    Create:  tar -cf <archive-filename> [filenames...]
    Help:    tar --help
  thread 'main' panicked at '
  command did not execute successfully, got: exit code: 1

  build script failed, must exit now', D:\rust\.cargo\registry\src\github.com-1ecc6299db9ec823\gettext-sys-0.21.0\build.rs:299:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed

[shijunti19]

[Minoru]

It just occurred to me that we can try running tar both with and without --force-local, and fail the build if both of them fail. If one of them succeeds, we just carry on as normal.

I got rid of my Windows VM already, but if someone has Windows without MSYS, please try editing build.rs to achieve this. When it works on your machine, submit a PR and see if it passes on Linux. If it works both for you and for CI, it'd mean it works both for GNU tar (Windows + MSYS) and bsdtar (stock Windows), and we can declare this issue resolved.

(Ideally you'd also add Windows to CI, but perhaps that's too huge an ask :)

@shijunti19, wanna try implementing that idea?

[lifegpc]

It just occurred to me that we can try running tar both with and without --force-local, and fail the build if both of them fail. If one of them succeeds, we just carry on as normal.

I agree. tar (I can install it with pacman -S msys/tar) have --force-local, but bsdtar (which included in libarchive) don't have --force-local.

[hboetes]

Although my problem is on OpenBSD I think it is caused by the same problem, portability is not being tested.

  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  running: "tar" "xJf" "/home/han/.cargo/registry/src/github.com-1ecc6299db9ec823/gettext-sys-0.19.9/gettext-0.19.8.1.tar.xz" "--strip-components" "1"

  --- stderr
  tar: unknown option J
  usage: tar {crtux}[014578befHhjLmNOoPpqsvwXZz]
             [blocking-factor | archive | replstr] [-C directory] [-I file]
             [file ...]
         tar {-crtux} [-014578eHhjLmNOoPpqvwXZz] [-b blocking-factor]
             [-C directory] [-f archive] [-I file] [-s replstr] [file ...]
  thread 'main' panicked at '
  command did not execute successfully, got: exit code: 1

  build script failed, must exit now', /home/han/.cargo/registry/src/github.com-1ecc6299db9ec823/gettext-sys-0.19.9/build.rs:251:5
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

That's right, the default tar on OpenBSD does not support the J option.

[Minoru]

We're going to drop tar altogether, fixing builds for both systems.

@hboetes, incidentally: are you aware of any CI providers that support OpenBSD natively (i.e. such that we don't need to install Qemu and bring our own image)? It'd be nice to have it in the build matrix.

[hboetes]

@hboetes, incidentally: are you aware of any CI providers that support OpenBSD natively (i.e. such that we don't need to install Qemu and bring our own image)? It'd be nice to have it in the build matrix.

@Minoru after some googling I found:

• https://man.sr.ht/builds.sr.ht/compatibility.md
• https://www.youtube.com/watch?v=6HlwhbNXe68 Hope that helps. :-)

[Minoru]

Hmm, interesting! I'm not ready to become a paying customer of theirs, and it doesn't look like there's an integration that would let us fail a PR if a builds fails. Thanks for the links though!

[hboetes]

Hmm, interesting! I'm not ready to become a paying customer of theirs, and it doesn't look like there's an integration that would let us fail a PR if a builds fails. Thanks for the links though!

Is there a way to set up a build bot on my own KVM?

[Minoru]

Apparently not, because only Linux, Windows and macOS are supported: https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#supported-architectures-and-operating-systems-for-self-hosted-runners

[hboetes]

With a bit of googling I found this: https://github.com/vmactions/freebsd-vm That looks like it can be ported to OpenBSD as well.

I'm not volunteering though. ;-)

[Minoru]

In #81, we discussed some options and ended up choosing "include GNU gettext as a submodule rather than a tarball". We also asked for the crates.io limit to be raised for gettext-sys, and it was.

What's left to be done is:

1. replace the tarball with a submodule, and rip out all the tar-related stuff we have in build.rs;
2. add a Windows config for CI (#81 is a good start).

PRs welcome!

[theCapypara]

Great! I'll look into this later today or tomorrow!

[emperor-limitless]

Anything new about this?

[cecton]

https://github.com/Koka/gettext-rs/pull/81#issuecomment-1031461015

[theCapypara]

I tried to add the sources as a submodule and build it, and I'm running into build errors. I'll be looking into this later when I'm a bit more motivated for troubleshooting, but maybe some of you could also have a look at the errors and maybe share some ideas:

• Linux: https://github.com/theCapypara/gettext-rs/actions/runs/1826894441

• Mac: https://github.com/theCapypara/gettext-rs/actions/runs/1809715980

• Windows: https://github.com/theCapypara/gettext-rs/actions/runs/1826898448

• Windows GNU: https://github.com/theCapypara/gettext-rs/actions/runs/1826885855

• Compare view: https://github.com/Koka/gettext-rs/compare/master...theCapypara:master

[theCapypara]

If I were to guess it's probably something automake / autoconf related for generating the ./configure file that needs to be changed (some other flags or something)?

[Minoru]

@theCapypara I only looked at Linux error, where gettext-runtime/intl/plural.c is missing — apparently it's autogenerated, and we need to install bison.

[theCapypara]

Okay I added Bison and a few other tools that popped up that were missing but needed (gperf, texinfo). It's still not building. I'm kinda confused, what is different to the tarball than the build form the repo (other than ./configure not being pre-generated)? Why are all these extra tools needed still and why is the build still failing? I kinda think just fixing "tar" for Windows would be way easier at this point. Otherwise even if we get it to build, a Windows system would need bison, gperf, texinfo, automake and autoconf anyway.

[Minoru]

what is different to the tarball than the build form the repo (other than ./configure not being pre-generated)? Why are all these extra tools needed still and why is the build still failing?

I wish I knew :) I mean, it's obvious that they're needed to pre-generate some files that are bundled into the tarball, but I don't know why it's done this way.

I kinda think just fixing "tar" for Windows would be way easier at this point. Otherwise even if we get it to build, a Windows system would need bison, gperf, texinfo, automake and autoconf anyway.

There were some good arguments against tar: https://github.com/Koka/gettext-rs/pull/81#issuecomment-895834856 and https://github.com/Koka/gettext-rs/pull/81#issuecomment-900661625. But I agree that requiring bison, gperf etc. is too big of an ask.

Luckily though, we now have an extra option: since the crates.io limit was increased for us, we can unpack the tarball and put its contents into the crate. That way, end users don't need tar at all (fixing the original problem), nor do they need all the tools that the submodule requires. What do you think about that approach?

[theCapypara]

We could also diff the upstream repo with the tarball and see what the precompiled files are. Then we can add the submodule to the repo and a directory with those files, copy it all over and voila. Would have the advantage of still referencing the original source and not having to just add everything to the repo.