Query string in download URL breaks wizard

[helgee]

trafficstars

Julia 1.5

julia> versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, icelake-client)
Environment:
  JULIA_PKG_DEVDIR = /Users/helge/projects/julia

Docker Desktop

BinaryBuilder

[12aac903] BinaryBuilder v0.2.5

Error Log

			# Step 3: Build for Linux(:x86_64, libc=:glibc)

You will now be dropped into the cross-compilation environment.
Please compile the package. Your initial compilation target is x86_64-linux-gnu
The $prefix environment variable contains the target directory.
Once you are done, exit by typing `exit` or `^D`

ArgumentError: '/var/folders/vp/qb910zys4w3951w9ntmb3zw40000gn/T/jl_2cTTVD/1HHxGCt5/srcdir' exists. `force=true` is required to remove '/var/folders/vp/qb910zys4w3951w9ntmb3zw40000gn/T/jl_2cTTVD/1HHxGCt5/srcdir' before copying.
Stacktrace:
 [1] checkfor_mv_cp_cptree(::String, ::String, ::String; force::Bool) at ./file.jl:308
 [2] cp(::String, ::String; force::Bool, follow_symlinks::Bool) at ./file.jl:345
 [3] cp at ./file.jl:345 [inlined]
 [4] setup(::BinaryBuilder.SetupSource{FileSource}, ::String, ::Bool) at /Users/helge/.julia/packages/BinaryBuilder/9oqch/src/Prefix.jl:311
 [5] setup_workspace(::String, ::Array{Any,1}; verbose::Bool) at /Users/helge/.julia/packages/BinaryBuilder/9oqch/src/Prefix.jl:368
 [6] step34(::BinaryBuilder.WizardState) at /Users/helge/.julia/packages/BinaryBuilder/9oqch/src/wizard/interactive_build.jl:510
 [7] run_wizard(::Nothing) at /Users/helge/.julia/packages/BinaryBuilder/9oqch/src/Wizard.jl:97
 [8] run_wizard() at /Users/helge/.julia/packages/BinaryBuilder/9oqch/src/Wizard.jl:80
 [9] top-level scope at REPL[1]:1
 [10] eval(::Module, ::Any) at ./boot.jl:331
 [11] eval_user_input(::Any, ::REPL.REPLBackend) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:134
 [12] repl_backend_loop(::REPL.REPLBackend) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:195
 [13] start_repl_backend(::REPL.REPLBackend, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:180
 [14] run_repl(::REPL.AbstractREPL, ::Any; backend_on_current_task::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:292
 [15] run_repl(::REPL.AbstractREPL, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
 [16] (::Base.var"#806#808"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:399
 [17] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [18] invokelatest at ./essentials.jl:709 [inlined]
 [19] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:383
 [20] exec_options(::Base.JLOptions) at ./client.jl:313
 [21] _start() at ./client.jl:506

WizardState [step3]

I tried starting from scratch multiple times, rebooted the machine, and deleted the offending directories. Same result...

[fingolfin]

Perhaps you could add in some information, like: which versions of macOS, Julia, BinaryBuilder are you using? Is Docker installed (that's required for doing this on macOS)?

[helgee]

Perhaps you could add in some information, like: which versions of macOS, Julia, BinaryBuilder are you using? Is Docker installed (that's required for doing this on macOS)?

Indeed. I was just exhausted after a day of failure and needed to collect myself first. See above...

The issue only occurs when I try to use these sources from SourceForge:

https://downloads.sourceforge.net/project/libtirpc/libtirpc/1.2.6/libtirpc-1.2.6.tar.bz2

[helgee]

The issue is macOS-specific. No problems on Linux...

[staticfloat]

Hmmm, I have been unable to reproduce this on my MacOS machine, using the exact same versions of everything. Are you still facing this issue?

[helgee]

I just tried it again and it's gone 🤷 Thanks for having a look!

[helgee]

I have the same problem on a different machine now 😱 (using a different SourceForge link)

IIRC the only thing that the two setups have in common is that I installed Docker immediately before using BinaryBuilder. Could this be some kind of initialization thing?

[fingolfin]

These sourceforge links all use http redirects. That might be part of the problem.

[helgee]

The issue is back on the other Mac as well.

[helgee]

Okay, I did some digging.

The problem seems to be related to the query parameters that SourceForge attaches to the direct link which need to be manually removed.

The initial direct link from SF is, e.g.

https://downloads.sourceforge.net/project/gmat/GMAT/GMAT-R2020a/GMAT-src-R2020a.zip?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fgmat%2Ffiles%2FGMAT%2FGMAT-R2020a%2FGMAT-src-R2020a.zip%2Fdownload&ts=1597900593&use_mirror=autoselect

If I do not remove the query string garbage, the source becomes

source::BinaryBuilder.SetupSource{FileSource} = BinaryBuilder.SetupSource{FileSource}("/var/folders/vp/qb910zys4w3951w9ntmb3zw40000gn/T/jl_7axuQ3/GMAT-src-R2020a.zip?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fgmat%2Ffiles%2FGMAT%2FGMAT-R2020a%2FGMAT-src-R2020a.zip%2Fdownload&ts=1597900593&use_mirror=autoselect", "943f403ac04d958b313b1d99d64fd09e3fa8e4c65363809d5bb88dd8c66e43e4", "")

The file var/folders/vp/qb910zys4w3951w9ntmb3zw40000gn/T/jl_7axuQ3/GMAT-src-R2020a.zip?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fgmat%2Ffiles%2FGMAT%2FGMAT-R2020a%2FGMAT-src-R2020a.zip%2Fdownload&ts=1597900593&use_mirror=autoselect exists on disk and if I try to cp it, e.g. cp(garbage_name_above, "Desktop"), I get the same error message:

julia> blob = "/var/folders/vp/qb910zys4w3951w9ntmb3zw40000gn/T/jl_7axuQ3/GMAT-src-R2020a.zip?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fgmat%2Ffiles%2FGMAT%2FGMAT-R2020a%2FGMAT-src-R2020a.zip%2Fdownload&ts=1597900593&use_mirror=autoselect"
"/var/folders/vp/qb910zys4w3951w9ntmb3zw40000gn/T/jl_7axuQ3/GMAT-src-R2020a.zip?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fgmat%2Ffiles%2FGMAT%2FGMAT-R2020a%2FGMAT-src-R2020a.zip%2Fdownload&ts=1597900593&use_mirror=autoselect"

shell> pwd
/Users/helge

julia> cp(blob, "Desktop")
ERROR: ArgumentError: 'Desktop' exists. `force=true` is required to remove 'Desktop' before copying.
Stacktrace:
 [1] checkfor_mv_cp_cptree(::String, ::String, ::String; force::Bool) at ./file.jl:308
 [2] cp(::String, ::String; force::Bool, follow_symlinks::Bool) at ./file.jl:345
 [3] cp(::String, ::String) at ./file.jl:345
 [4] top-level scope at REPL[3]:1

EDIT: The error is correct because Desktop does in fact exist. My bad...

TL;DR: There seem to be two issues here: ~~- A bug in Base.cp which gives an incorrect error message for a garbage source argument~~

• BinaryBuilder not scrubbing the query string when determining the output location

What is still puzzling me is that I ran into a situation were the downloaded archive became GMAT-src-R2020a.zip? but if we apply Occam's Razor the answer is probably that I had a trailing ? because I edited the URL improperly.

[bramtayl]

I ran into this and the problem was the url had ".ZIP" instead of ".zip" at the end.