Bug: symbol lookup error: ... undefined symbol

[briochemc]

Unsure what is happening. Trying to run my code on a google cloud virtual machine, I get:

Do you want to download the dataset from URL to "local_path"?
[y/n]
y
/local_bin_path/julia: symbol lookup error: /home/local_user/.julia/packages/MbedTLS/XkQiX/deps/usr/lib/libmbedtls.so: undefined symbol: mbedtls_x509_crt_verify_restartable

and that stops the run... (Note I "masked" some stuff to only show the relevant info.) Any ideas?

[oxinabox]

This is a MbedTLS.jl issue. You should open an issue on their repo.

I think it might have already been fixed as it sounds similar to something I heard was fixed recently, so do a package update if you've not done one recently.

A workaround is to define a custom fetch method that uses Base.download should be stuff in docs about that. I should extend that bit of the docs with an example

[briochemc]

It seems there is an issue already (link) to which I added a :+1:

You mean that Base.download could be added with a try/catch somewhere? Could you explain (or point out to where I can find the info) why DataDeps uses something else (HTTPS.jl?) in the first place?

[oxinabox]

We originally used Base.download, but the problem is that determining filenames for things from HTTP is actually insane, and often has to be done using header information. If you go back digging into the issues (around #22) you can see how complicated that is using commandline tools, and it just wasn't robust enough on Windows, or differing unixen. It also wasn't robust against different servers that might not support the HEAD command. The better way to do it is as part of the GET, to get headers and content at same time.

Anyway the simple case, for when you don't need a header as part of the filename is

function fallback_download(remotepath, localdir)
    @assert(isdir(localdir))
    filename = basename(remotepath)  # only works for URLs with filename as last part of name
    localpath = joinpath(localdir, filename)
    Base.download(remotepath, localpath)
    return localpath
end


register(DataDep( ...
     fetch_method = fallback_download
)

[briochemc]

Thanks for the explanation!

Just to be sure I understand: Is the fallback_download function in DataDeps.jl at the moment? I.e., can I use the fetch_method = fallback_download directly? Or do I have to include the fallback_download function definition in my code?

[oxinabox]

It is not in DataDeps.jl right now.

I might add it to the docs at some point as an example of how to define your own,

[briochemc]

Is there a way for to write the same type of fallback (that uses Base.download) but that works when the URL does not end with the file name? (I am trying it on a file on FigShare and am not sure I can get the URL with the file name at the end)

[briochemc]

Oh and for the record, I just tried it now and got an UndefVarError: update_period not defined error. So I removed the update_period part — that is, instead, I used this

function fallback_download(remotepath, localdir)
    @assert(isdir(localdir))
    filename = basename(remotepath)  # only works for URLs with filename as last part of name
    localpath = joinpath(localdir, filename)
    Base.download(remotepath, localpath)
    return localpath
end

which seems to work 😄

[briochemc]

For the (my) record also, note that instead of the clickable URL that FigShare gives you, you can instead use a URL of the type http://files.figshare.com/<Project/file number>/<file name>. Useful if you want to use

filename = basename(remotepath)  # only works for URLs with filename as last part of name

[oxinabox]

Is there a way for to write the same type of fallback (that uses Base.download) but that works when the URL does not end with the file name? (I am trying it on a file on FigShare and am not sure I can get the URL with the file name at the end)

See around #22 but it is really unrobust.

Oh and for the record, I just tried it now and got an UndefVarError: update_period not defined error. So I removed the update_period part

Sorry, that was a typo on my part. Your code is the correct one.