JuliaPackaging
WinRPM.update() failing to download XML on Windows
Any ideas?
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.4.6 (2016-06-19 17:16 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release
|__/ | x86_64-w64-mingw32
julia> using WinRPM
WARNING: skipping repodata/repomd.xml, not in cache -- call WinRPM.update() to download
WARNING: skipping repodata/repomd.xml, not in cache -- call WinRPM.update() to download
julia> WinRPM.update()
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 1/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 2/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 3/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 4/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 5/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.2/repodata/repomd.xml
WARNING: received error 0 while downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_13.2/repodata/repomd.xml
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 1/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 2/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 3/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 4/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270094
WARNING: Retry 5/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml
WARNING: received error 0 while downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml
julia> versioninfo()
Julia Version 0.4.6
Commit 2e358ce (2016-06-19 17:16 UTC)
Platform Info:
System: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.3
julia> UInt32(2148270094)
0x800c000e
julia> reinterpret(Int32,UInt32(2148270094))
-2146697202
https://msdn.microsoft.com/en-us/library/ms695782(v=vs.85).aspx
-2146697202 0x800C000E Server security access failure.*
There was a security problem when accessing the server.
might be an https certificate issue or something, can you read the url from a browser?
Hmm. Yeah. If I paste the URL in the browser, I can see the XML response.
I'm new to this, but I suppose I should mention this is running on an Azure VM...
If you do import BinDeps: download_cmd can you do run(download_cmd("https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml", "repomd.xml")) ?
julia> download_cmd("https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml", "repomd.xml")
PowerShell[.exe] [-PSConsoleFile <file> | -Version <version>]
[-NoLogo] [-NoExit] [-Sta] [-Mta] [-NoProfile] [-NonInteractive]
[-InputFormat {Text | XML}] [-OutputFormat {Text | XML}]
[-WindowStyle <style>] [-EncodedCommand <Base64EncodedCommand>]
[-File <filePath> <args>] [-ExecutionPolicy <ExecutionPolicy>]
[-Command { - | <script-block> [-args <arg-array>]
| <string> [<CommandParameters>] } ]
PowerShell[.exe] -Help | -? | /?
-PSConsoleFile
Loads the specified Windows PowerShell console file. To create a console
file, use Export-Console in Windows PowerShell.
-Version
Starts the specified version of Windows PowerShell.
Enter a version number with the parameter, such as "-version 2.0".
-NoLogo
Hides the copyright banner at startup.
-NoExit
Does not exit after running startup commands.
-Sta
Starts the shell using a single-threaded apartment.
Single-threaded apartment (STA) is the default.
-Mta
Start the shell using a multithreaded apartment.
-NoProfile
Does not load the Windows PowerShell profile.
-NonInteractive
Does not present an interactive prompt to the user.
-InputFormat
Describes the format of data sent to Windows PowerShell. Valid values are
"Text" (text strings) or "XML" (serialized CLIXML format).
-OutputFormat
Determines how output from Windows PowerShell is formatted. Valid values
are "Text" (text strings) or "XML" (serialized CLIXML format).
-WindowStyle
Sets the window style to Normal, Minimized, Maximized or Hidden.
-EncodedCommand
Accepts a base-64-encoded string version of a command. Use this parameter
to submit commands to Windows PowerShell that require complex quotation
marks or curly braces.
-File
Runs the specified script in the local scope ("dot-sourced"), so that the
functions and variables that the script creates are available in the
current session. Enter the script file path and any parameters.
File must be the last parameter in the command, because all characters
typed after the File parameter name are interpreted
as the script file path followed by the script parameters.
-ExecutionPolicy
Sets the default execution policy for the current session and saves it
in the $env:PSExecutionPolicyPreference environment variable.
This parameter does not change the Windows PowerShell execution policy
that is set in the registry.
-Command
Executes the specified commands (and any parameters) as though they were
typed at the Windows PowerShell command prompt, and then exits, unless
NoExit is specified. The value of Command can be "-", a string. or a
script block.
If the value of Command is "-", the command text is read from standard
input.
If the value of Command is a script block, the script block must be enclosed
in braces ({}). You can specify a script block only when running PowerShell.exe
in Windows PowerShell. The results of the script block are returned to the
parent shell as deserialized XML objects, not live objects.
If the value of Command is a string, Command must be the last parameter
in the command , because any characters typed after the command are
interpreted as the command arguments.
To write a string that runs a Windows PowerShell command, use the format:
"& {<command>}"
where the quotation marks indicate a string and the invoke operator (&)
causes the command to be executed.
-Help, -?, /?
Shows this message. If you are typing a PowerShell.exe command in Windows
PowerShell, prepend the command parameters with a hyphen (-), not a forward
slash (/). You can use either a hyphen or forward slash in Cmd.exe.
EXAMPLES
PowerShell -PSConsoleFile SqlSnapIn.Psc1
PowerShell -version 2.0 -NoLogo -InputFormat text -OutputFormat XML
PowerShell -Command {Get-EventLog -LogName security}
PowerShell -Command "& {Get-EventLog -LogName security}"
# To use the -EncodedCommand parameter:
$command = 'dir "c:\program files" '
$bytes = [System.Text.Encoding]::Unicode.GetBytes($command)
$encodedCommand = [Convert]::ToBase64String($bytes)
powershell.exe -encodedCommand $encodedCommand
`powershell -Command '(new-object net.webclient).DownloadFile("https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml", "repomd.xml")'`
julia> download_cmd
download_cmd (generic function with 1 method)
sorry, run the command -
run(download_cmd("https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_13.2/repodata/repomd.xml", "repomd.xml"))
then you should end up with a repomd.xml in your current directory if that downloading mechanism works. WinRPM uses a different downloader, we'd need to narrow down what the difference is if one works and the other doesn't.
Yeah. That worked. It downloaded the XML. If I do that in .julia/WinRPM, where should I put it so it finds it?
I'll also try to figure it out myself...
the code here is pretty short, if you can find where the downloading happens you should be able to add in some @show statements to determine where it's trying to download the files to
Hmm... I ended up just copying over WinRPM from my local machine to the remote VM which got using WinRPM to work, but I need Nettle.jl and Nettle seems to run WinRPM.update() so I'd need to change Nettle, which starts feeling like too much of a hack.
If I modify WinRPM.update() so it uses download_cmd, should I submit a PR here?
I suppose we could try that. Not sure how it would fit into the current code which passes around status codes and a few other things. URLDownloadToCacheFileW has been problematic though.
Ok. I am in a bit of a bind (and time is not on my side), so I'll have a go at modifying the download function
@unix_only download(source::AbstractString) = (x=HTTPC.get(source); (bytestring(x.body),x.http_code))
@windows_only function download(source::AbstractString; retry = 5)
dest = Array(UInt16,261)
for i in 1:retry
res = ccall((:URLDownloadToCacheFileW,:urlmon),stdcall,Cuint,
(Ptr{Void},Ptr{UInt16},Ptr{UInt16},Clong,Cint,Ptr{Void}),
C_NULL,utf16(source),dest,sizeof(dest)>>1,0,C_NULL)
if res == 0
resize!(dest, findfirst(dest, 0))
filename = utf8(UTF16String(dest))
if isfile(filename)
return readall(filename),200
end
else
warn("Unknown download failure, error code: $res")
end
warn("Retry $i/$retry downloading: $source")
end
return "",0
end
Instead of BinDeps, I could do it pretty easily with Requests, but then Requests becomes a dependency of WinRPM. Are you ok with that?
No, Requests should in an ideal world be getting its binaries from here. You don't need Requests to download a file, Requests has its own binary dependencies.
Ok. So I'll try to modify the download function above to use download_cmd. Thanks :+1:
Hmm... it appears the download_cmd does not return the status. Bummer. I guess that is what you meant above :sweat_smile:
to be fair, the function you quoted above is kind of faking the status anyway.
Hi @tkelman
Here is what I have and it seems to be working :+1:
@windows_only function download(source::AbstractString)
filename = joinpath(dirname(@__FILE__),"..","cache",basename(source))
run(BinDeps.download_cmd(source,filename))
if isfile(filename)
return readall(filename),200
end
return "",0
end
Downsides:
1 It leaves the .gz files in the cache folder (in Windows) 2. It is spewing PowerShell stuff (like above) to the REPL for some reason
Are the downsides, ok? Any idea how to silence the PowerShell stuff?
I think master of BinDeps might be quieter w.r.t. powershell, it may have been a while since the last tag? What version of BinDeps are you using?
I was using the latest tagged version (a fresh install of Julia on the remote VM). I'll try master, but when I did your experience before to test download_cmd, I thought it worked without the noise (but maybe I was on master at that time - I've been checking out and freeing in a trial and error fashion).
@tkelman , if I were to submit a PR for WinRPM, should I start from the latest tagged version or from master?
Cross-ref: https://github.com/JuliaInterop/ZMQ.jl/issues/90#issuecomment-134421062
I'm having a similar problem with repomod.xml. I can download manually or visit it in a browser, but it won't download from WinRPM in Julia 0.5. The error is 2148270088 (-2146697208). I couldn't find it on Microsoft's site. I tried to checkout master. Same problem. It's preventing gtk and cairo from building. I'm not sure what changed as everything had been working fine; it may have something to do with upgrading anaconda, which changed from Python 3.5 to 3.6?
I wiped out WinRPM and rebooted. Then I ran something like Pkg.update() which re-installed but still failed on repmod.xml. But when I ran WinRPM.update() this time, it seems to have fixed the problem and rebuilt Gtk. Unfortunately, ProfileView is still broken, so something is still wrong.
same error code here:
julia> WinRPM.update()
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_42.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270085
WARNING: Retry 1/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_42.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270085
WARNING: Retry 2/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_42.2/repodata/repomd.xml
WARNING: Unknown download failure, error code: 2148270085
WARNING: Retry 3/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_42.2/repodata/repomd.xml
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_42.2/repodata/8c9628aaa81afc42e7d02ffc19ea6fce9484c8d74ac80db4e2b9420c841fcb4a-primary.xml.gz
WARNING: Unknown download failure, error code: 2148270085
WARNING: Retry 1/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_42.2/repodata/8c9628aaa81afc42e7d02ffc19ea6fce9484c8d74ac80db4e2b9420c841fcb4a-primary.xml.gz
WARNING: Unknown download failure, error code: 2148270085
WARNING: Retry 2/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win32/openSUSE_42.2/repodata/8c9628aaa81afc42e7d02ffc19ea6fce9484c8d74ac80db4e2b9420c841fcb4a-primary.xml.gz
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_42.2/repodata/repomd.xml
INFO: Downloading https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_42.2/repodata/cb547b27fb7340befef807b4ca16a1f42e722bf920d52f861155a3e7b701f660-primary.xml.gz
WARNING: Unknown download failure, error code: 2148270088
WARNING: Retry 1/5 downloading: https://cache.julialang.org/http://download.opensuse.org/repositories/windows:/mingw:/win64/openSUSE_42.2/repodata/cb547b27fb7340befef807b4ca16a1f42e722bf920d52f861155a3e7b701f660-primary.xml.gz
update:
I finally fixed this using a router to globally bypass the infamous GFW(I'm living in China). 🤔
