unable to use thredds access on Windows, problem with cookies

[jmbeckers]

I try accessing a netcdf file using NCDatasets under Julia, with libraries compiled with mingw compiler (x86_64-w64-mingw32-gcc (GCC) 4.8.5 and NetCDF 4.7.4 (for windows).

It seems there is a problem with a cookiejar that cannot be created (see error messages below).

A look at the function ocinternal.c seems to suggest the directory or file cannot be created

Maybe it is due to a hard coded "/" in the path name which is not working under Windows ?

https://github.com/Unidata/netcdf-c/blob/b2097a77aaf6f237e7d9afe531731d93273fe157/oc2/ocinternal.c#L565

Any suggestion how to make it work would be welcomed.

NCDataset("https://erddap.ifremer.fr/erddap/griddap/SDC_GLO_CLIM_TS_V2_1") state->auth.curlflags.cookiejar != NULL

Assertion failed: ocpanic(("state->auth.curlflags.cookiejar != NULL")), file ocinternal.c, line 566

signal (22): SIGABRT in expression starting at REPL[2]:1 crt_sig_handler at /cygdrive/c/buildbot/worker/package_win64/build/src\signals-win.c:92 raise at C:\Windows\System32\msvcrt.dll (unknown line) abort at C:\Windows\System32\msvcrt.dll (unknown line) assert at C:\Windows\System32\msvcrt.dll (unknown line) ocset_curlproperties at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line) ocopen at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line) .text at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line) NCD2_open at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line) NC_open at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line) nc_open at C:\Users\jmbeckers.julia\artifacts\a81fe95ac632a7fa76c5e9cbe522c998aee9fa21\bin\libnetcdf-18.dll (unknown line) nc_open at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\netcdf_c.jl:266 unknown function (ip: 000000005f6f6b8d) #NCDataset#12 at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\dataset.jl:187 NCDataset at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\dataset.jl:157 [inlined] NCDataset at C:\Users\jmbeckers.julia\packages\NCDatasets\XVX8L\src\dataset.jl:157 jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined] do_call at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:126 eval_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:215 eval_stmt_value at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:166 [inlined] eval_body at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:583 jl_interpret_toplevel_thunk at /cygdrive/c/buildbot/worker/package_win64/build/src\interpreter.c:731 jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:885 jl_toplevel_eval_flex at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:830 jl_toplevel_eval at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:894 [inlined] jl_toplevel_eval_in at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:944 eval at .\boot.jl:373 [inlined] eval_user_input at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:150 repl_backend_loop at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:244 start_repl_backend at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:229 #run_repl#47 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:362 run_repl at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.7\REPL\src\REPL.jl:349 #930 at .\client.jl:394 jfptr_YY.930_35191.clone_1 at C:\Users\jmbeckers\AppData\Local\Programs\Julia-1.7.1\lib\julia\sys.dll (unknown line) jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined] jl_f__call_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:757 #invokelatest#2 at .\essentials.jl:716 [inlined] invokelatest at .\essentials.jl:714 [inlined] run_main_repl at .\client.jl:379 exec_options at .\client.jl:309 _start at .\client.jl:495 jfptr__start_43221.clone_1 at C:\Users\jmbeckers\AppData\Local\Programs\Julia-1.7.1\lib\julia\sys.dll (unknown line) jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1788 [inlined] true_main at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:559 jl_repl_entrypoint at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:701 mainCRTStartup at /cygdrive/c/buildbot/worker/package_win64/build/cli\loader_exe.c:42 BaseThreadInitThunk at C:\Windows\System32\KERNEL32.DLL (unknown line) RtlUserThreadStart at C:\Windows\SYSTEM32\ntdll.dll (unknown line) Allocations: 722840 (Pool: 722461; Big: 379); GC: 1

[DennisHeimbigner]

The / should get converted to \ by NC_mktmp.

   tmppath = NC_mktmp(path);

I did quite a lot of work on mingw support that will not show up until the next release. Are you in a position to pull and build the current netcdf-c main on github and try that?

[DennisHeimbigner]

In any case, one workaround should be this:

1. create a file called .dodsrc in you home directory
2. Insert this line in that .dodsrc file. HTTP>COOKIEJAR=<abs path to some temp file>

So e.g

HTTP.COOKIEJAR=/tmp/netcdf-cookies

[jmbeckers]

In any case, one workaround should be this:

1. create a file called .dodsrc in you home directory

2. Insert this line in that .dodsrc file. `HTTP>COOKIEJAR=<abs path to some temp file>`

So e.g

HTTP.COOKIEJAR=/tmp/netcdf-cookies

Thanks, I tried to translate that into Windows jargon and put a file .dodsrc into c:\Users\jmbeckers

with content HTTP.COOKIEJAR=C:\Users\jmbeckers\AppData\Local\Temp\netcdf-cookies

no joy. Same when using / instead of \ in the path to the file, including or not an empty file netcdf-cookies

As for pulling your new version and compiling myself, I would be far out of my comfort zone, as you might guess from my unsuccessful attempt right now.

[DennisHeimbigner]

ok. try putting the .dodsrc in the same directory in which you are working.

[jmbeckers]

yep, that worked ! I have put it in the C:\Users\jmbeckers\AppData\Local\Programs\Julia-1.7.1\bin directory where I launched julia.exe by hand from a cmd window in order to be able to see the error message. This also works if I run it directly with the Julia "icon", but not when I run in Jupyter (probably I need to copy the file also into a directory where Jupyter starts IJulia). Anyway, thanks

PS: edit, just copied the .dodsrc also into the place where Jupyter starts by default and it works now too.

[DennisHeimbigner]

It is probably that you do not have the environment variable HOME set on windows.

[jmbeckers]

That does not seem to be the case; at least when I search for environmental variable in "Settings", the panel which defines the variables contains no HOME. BTW I tried to include HTTP.COOKIEJAR=C:\Users\jmbeckers\AppData\Local\Temp\netcdf-cookies as an environmental variable there, but that did not work out either. The secrets of Windows ...

[Alexander-Barth]

Maybe this is helpful to get the home directory on Windows: https://stackoverflow.com/a/42700844

I am wondering if NetCDF can also look for USERPROFILE if HOME is undefined (on Windows)? https://github.com/Unidata/netcdf-c/blob/93742f8e7b77425400ceeb1791766bcd5b5ecb59/libdispatch/ddispatch.c#L79

(I can make a PR if this is helpful)

[jmbeckers]

%HOMEPATH% points me to \Users\jmbeckers where I first tried to place the .dodsrc file without success.

[DennisHeimbigner]

Netcdf-c uses "HOME" not "HOMEPATH", But apparently, I should be using the environment variable USERPROFILE instead of HOME.

[CyprienBosserelle]

I had to tweak this workaround to make it work for me. It may not help with isolating the ultimate issue but hopefully that helps other in the same situation as me.

This didn't work form me (with any folder explicitly written)

HTTP.COOKIEJAR=C:\Users\USERNAME\AppData\Local\Temp

I kept getting an error saying i do not have read write access to the folder .

However, somehow, this worked for me

HTTP.COOKIEJAR=$USERPROFILE