SciSharpCube
SciSharpCube copied to clipboard
SciSharp
Run an error in the K8s container
My k8s docker environment is as follows: Use up to 1 cores of cpu,1g memory，The following error occurred in the run：
2019-06-02T12:31:51.410400083Z display_data: [{"source":"","data":{"text/plain":"Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references\n at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.<Restore>g__RestoreAndCacheProjectFile|5_0(<>c__DisplayClass5_0& )\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code)\n at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 172\n at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 108\n at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49","text/html":"<p style="color:red;">Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references\n at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.<Restore>g__RestoreAndCacheProjectFile|5_0(<>c__DisplayClass5_0& )\n at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources)\n at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code)\n at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 172\n at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 108\n at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49
1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49
It seems like a nuget package downloading failure. Does your k8s environment allow outgoing traffic?
From Microsoft/dotnet, this should be change. https://devblogs.microsoft.com/dotnet/net-core-container-images-now-published-to-microsoft-container-registry/
Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Context.CachedRestorer.g__RestoreAndCacheProjectFile|5_0(<>c__DisplayClass5_0& ) at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources) at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code) at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 172 at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 108 at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 49
It's okay to manually go inside the docker and run the command line.
/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive# ls NuGet.Config obj script.csproj /tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive# dotnet restore Restore completed in 622.5 ms for /tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.2/icsharpcore/0.2.2/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj.
@geffzhang I tried running via commandline inside the container.... "using Tensorflow" completed however, whenever i start using tensorflow (e.g. tf.constant("test");), the kernel just dies... any workaround for this?
@whatevergeek which docker image version do you use?
Did you get same error with Geff Zhang?
@whatevergeek which docker image version do you use?
Did you get same error with Geff Zhang? I used scisharpstack/scisharpcube:0.4
Initially the container fails when loading (using Tensorflow;) After i've done @geffzhang commandline restore, it got completed. However, the container dies again when i do: tf.constant("test");
That means the native Tensorflow lib cannot be found. I will try to reproduce it.
Which kind host do you use? Windows, Linux or Mac OSX?
Windows. Using Docker Toolbox for hosting the image.
Btw, i'm also trying to use binder to help reproduce it online. However, the environment doesn't seem to launch. Anyway, placing my repo here in case anyone got suggestion on the dockerfile setup: https://github.com/whatevergeek/SciSharpCubeBinder
@whatevergeek Using the Try.NET repository as a starting base, I have made a working dockerfile for Binder:
https://github.com/javiercp/BinderTF.NET
I haven't tested it fully but it seems to work. The helloworld sample returns null but otherwise the objects seem right, and the linear regression test works just fine.
@kerryjiang I have the same problem with the dockerfile from this repo, but I'm running it from behind my corporate firewall. This maybe a problem to get the Nuget packages. I'll try it from home later.
No, just standard Windows 10, but unfortunately with corporate proxy/antivirus/...
At home I'll have a simpler connection. I'll check it when I get back.
@javiercp i tried the binderhub... it kinda works... thanks for sharing... i tried the following:
var str = "Hello, TensorFlow.NET!"; var hello = tf.constant(str);
` var str = "Hello, TensorFlow.NET!"; var hello = tf.constant(str);
// Start tf session using(var sess = tf.Session()) { var result = sess.run(hello); Console.WriteLine(result.GetType()); Console.WriteLine(result.ToString()); // Run the op return result; }`
-- result --
` NumSharp.NDArray
Hello, TensorFlow.NET!
<null>
`
Yup, that's what I see too. The return result shows null, although the result object seems fine.
If you try it with a int constant it works without problem: var foo = tf.constant(1); var bar = tf.constant(2);
// Start tf session using(var sess = tf.Session()) { // Run the op var result = sess.run(foo + bar);
Console.WriteLine(result.GetType());
Console.WriteLine(result.ToString());
return result;
}
-- result -- NumSharp.NDArray 3
| index | value |
|---|---|
| 0 | 3 |
So I guess it works, it's just something when it tries to show the result.
@kerryjiang A bit more info. This is the log from the docker container:
[...] 6bcea139de54: Pull complete Digest: sha256:6b518f79038b4eaf385f32a2ad823764ea94d116ac94f4d4c93b68a44366d44f Status: Downloaded newer image for scisharpstack/scisharpcube:latest [I 09:58:37.832 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret [I 09:58:38.018 NotebookApp] Serving notebooks from local directory: /scisharp/home [I 09:58:38.018 NotebookApp] The Jupyter Notebook is running at: [I 09:58:38.018 NotebookApp] http://87e79186a562:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250 [I 09:58:38.018 NotebookApp] or http://127.0.0.1:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250 [I 09:58:38.018 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 09:58:38.023 NotebookApp]
To access the notebook, open this file in a browser:
file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
Or copy and paste one of these URLs:
http://87e79186a562:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250
or http://127.0.0.1:8888/?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250
[I 12:34:56.180 NotebookApp] 302 GET /?token=c1a49993cbbf76792a7a2bc52658f6365bb7edabcefb6250 (172.17.0.1) 0.82ms
[I 12:35:00.604 NotebookApp] Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret
[W 12:35:00.604 NotebookApp] Notebook samples/HelloWorld.ipynb is not trusted
[I 12:35:00.723 NotebookApp] Kernel started: 2c31cc70-8aae-4b39-8790-cc33c3eac300
[W 12:35:00.759 NotebookApp] 404 GET /static/components/codemirror/mode/.netstandard/.netstandard.js?v=20191115095837 (172.17.0.1) 31.31ms referer=http://127.0.0.1:8888/notebooks/samples/HelloWorld.ipynb
Kernel connecting...
arg 0: /root/.local/share/jupyter/runtime/kernel-2c31cc70-8aae-4b39-8790-cc33c3eac300.json
arg 1: /scisharp/refs.txt
{"shell_port":33909,"iopub_port":59745,"stdin_port":54997,"control_port":35107,"hb_port":45979,"IP":"127.0.0.1","Key":"ef8cdb3b-c5d50a30318711a08f4c3346","Transport":"tcp","signature_scheme":"hmac-sha256","kernel_name":"csharpcore"}
Listening Shell @tcp://127.0.0.1:33909
Listening IOPub @tcp://127.0.0.1:59745
kernel_info_request: [NetMQMessage[ ?~qO,<IDS|MSG>,6d6688c88562e04933708634a15aa3a73587155c1e6ee13fbda8ebc138ed2fcc,{"date":"2019-11-15T12:35:00.764013Z","username":"username","msg_type":"kernel_info_request","session":"bf825497633848628df4be9a602a2a2f","version":"5.3","msg_id":"d71ebcf5-78b257e2cdcf146180cb8a55"},{},{},{}]]
kernel_info_request: [NetMQMessage[bf825497633848628df4be9a602a2a2f,<IDS|MSG>,07c43a7d1ddae3a9445a289345df1844c22000d4c32bdfc82d38d80f6a3e3255,{"username":"username","msg_type":"kernel_info_request","msg_id":"5a5b6ae49d9546ef88aebd906f906059","session":"bf825497633848628df4be9a602a2a2f","version":"5.2"},{},{},{}]]
comm_info_request: [NetMQMessage[bf825497633848628df4be9a602a2a2f,<IDS|MSG>,147639e491b474a863f349364c76351544d1b3d3ccb63735c220a0ecd53c6d13,{"username":"username","msg_type":"comm_info_request","msg_id":"50c9484b17664c338208be8c415f3992","session":"bf825497633848628df4be9a602a2a2f","version":"5.2"},{},{},{"target_name":"jupyter.widget"}]]
execute_request: [NetMQMessage[bf825497633848628df4be9a602a2a2f,<IDS|MSG>,3a5e7482f53a5dde679a5b74a9088dc0a3206a8694fa18c9d059b8c158abb00c,{"username":"username","msg_type":"execute_request","msg_id":"9213303500f14e13af545a2337127bbb","session":"bf825497633848628df4be9a602a2a2f","version":"5.2"},{},{},{"code":"using Tensorflow;","user_expressions":{},"store_history":true,"stop_on_error":true,"allow_stdin":true,"silent":false}]]
fail: ExecuteHandler[0]
Failed to run the code: using Tensorflow;
System.Exception: Unable to restore packages from '/tmp/scripts/root/.dotnet/tools/.store/icsharpcore/0.2.4/icsharpcore/0.2.4/tools/netcoreapp2.2/any/REPL/interactive/interactive/script.csproj'. Make sure that all script files contains valid NuGet references
at Dotnet.Script.DependencyModel.Context.DotnetRestorer.Restore(String pathToProjectFile, String[] packageSources)
at Dotnet.Script.DependencyModel.Context.CachedRestorer.<Restore>g__RestoreAndCacheProjectFile|5_0(<>c__DisplayClass5_0& )
at Dotnet.Script.DependencyModel.Context.CachedRestorer.Restore(String pathToProjectFile, String[] packageSources)
at Dotnet.Script.DependencyModel.Context.ProfiledRestorer.Restore(String pathToProjectFile, String[] packageSources)
at Dotnet.Script.DependencyModel.Runtime.RuntimeDependencyResolver.GetDependenciesForCode(String targetDirectory, ScriptMode scriptMode, String[] packageSources, String code)
at ICSharpCore.Script.InteractiveScriptEngine.TryLoadReferenceFromScript(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 170
at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/Script/InteractiveScriptEngine.cs:line 106
at ICSharpCore.RequestHandlers.ExecuteHandler1.Process(Message1 message) in /Users/kerryjiang/WorkShop/ICSharpCore/src/ICSharpCore/RequestHandlers/ExecuteHandler.cs:line 70
And this is what I see in the browser:
It seems the restore isn't working, that's why suppose there is a problem with my container reaching nuget.
@javiercp I can run it:
@kerryjiang Can you help update the Cube environment? Seems the syntax and package are out of date.
@javiercp Do you mind if we put your link on our SciSharpCube Readme? I thought it's so convenient to experience SciSharp stack for new developers.
@Oceania2018 Sure, no problem. I'll also update the repo readme to point that it's using SciSharp's Tensorflow.NET.