Neural Network workflow does not work behind proxy

[btbest]

A user who works behind a company proxy could not start the Neural Network workflow: https://forum.image.sc/t/ilastik-neural-network-local-failed-to-start-from-the-beginning/80907/8

Logs (see below) indicate that the proxy server blocks GRPC requests from the ilastik frontend to the tiktorch server, responding with 403 Forbidden.

My guess is that the proxy is set up to only allow HTTPS, and as far as I can see we are not SSL-encrypting GRPC requests.

The user was able to get the workflow to work by setting grpc.enable_http_proxy to 0:

[PATCH] NN: Explicitly turn off proxy
---
Index: lazyflow/operators/tiktorch/classifier.py
<+>UTF-8
===================================================================
diff --git a/lazyflow/operators/tiktorch/classifier.py b/lazyflow/operators/tiktorch/classifier.py
--- a/lazyflow/operators/tiktorch/classifier.py	(revision 629932dee9cddc420a07a10a43c11b8f9807837e)
+++ b/lazyflow/operators/tiktorch/classifier.py	(revision 92c32ce138121acee0dfb79cd5b66c4bcf1e2c3b)
@@ -391,7 +391,11 @@
         logger.debug("Trying to connect to tiktorch server using %s(%s):%s", host, addr, port),
         self._chan = grpc.insecure_channel(
             f"{addr}:{port}",
-            options=[("grpc.max_send_message_length", _100_MB), ("grpc.max_receive_message_length", _100_MB)],
+            options=[
+                ("grpc.max_send_message_length", _100_MB),
+                ("grpc.max_receive_message_length", _100_MB),
+                ("grpc.enable_http_proxy", 0),
+            ],
         )
         client = inference_pb2_grpc.InferenceStub(self._chan)
         upload_client = data_store_pb2_grpc.DataStoreStub(self._chan)

But I don't think we can solve the problem like this in general.

Expected behavior

The workflow should run whether behind a proxy or not.

To Reproduce

Steps to reproduce the behavior:

1. Be behind a proxy :)
2. Try to create a new Neural Network (Local) project

(I will have to try if I can replicate the setup to actually reproduce the problem)

Error message/traceback

Traceback (most recent call last):
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\lazyflow\operator.py”, line 117, in call
instance = ABCMeta.call(cls, *args, **kwargs)
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\ilastik\workflows\neuralNetwork_localWorkflow.py”, line 53, in init
super().init(shell, headless, workflow_cmdline_args, project_creation_args, *args, **kwargs)
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\ilastik\workflows\neuralNetwork_nnWorkflowBase.py”, line 91, in init
self._createClassifierApplet(headless=self._headless, conn_str=connection_string)
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\ilastik\workflows\neuralNetwork_localWorkflow.py”, line 67, in _createClassifierApplet
preferred_cuda_device_id, device_name = super()._configure_device(conn)
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\ilastik\workflows\neuralNetwork_nnWorkflowBase.py”, line 245, in _configure_device
devices = conn.get_devices()
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\lazyflow\operators\tiktorch\classifier.py”, line 355, in get_devices
resp = self._client.ListDevices(inference_pb2.Empty())
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\grpc_channel.py”, line 946, in call
return _end_unary_response_blocking(state, call, False, None)
File “C:\Program Files\ilastik-1.4.0-gpu\lib\site-packages\grpc_channel.py”, line 849, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = “failed to connect to all addresses”
debug_error_string = “{“created”:”@1683600100.468000000",“description”:“Failed to pick subchannel”,“file”:“src/core/ext/filters/client_channel/client_channel.cc”,“file_line”:3159,“referenced_errors”:[{“created”:“@1683600100.468000000”,“description”:“failed to connect to all addresses”,“file”:“src/core/lib/transport/error_utils.cc”,“file_line”:147,“grpc_status”:14}]}"

With GRPC_TRACE=all and GRPC_VERBOSITY=debug, we receive additional logs:

I0630 10:19:15.338000000 10824 src/core/ext/filters/client_channel/http_connect_handshaker.cc:333] Connecting to server 127.0.0.1:62817 via HTTP proxy ipv4:<proxy IP>:8080
(...)
I0630 10:19:15.355000000 10824 src/core/lib/channel/handshaker.cc:89] handshake_manager 0000024388134E10: error={"created":"@1688087955.355000000","description":"HTTP proxy returned response code 403","file":"src/core/ext/filters/client_channel/http_connect_handshaker.cc","file_line":253} shutdown=0 index=1, args={endpoint=(nil), args=(nil) {size=0: }, read_buffer=(nil) (length=0), exit_early=0}

Desktop:

• ilastik version: 1.4.0-gpu
• OS: Windows

[tibuch]

I encounter this issue as well. Behind the proxy I can´t start the workflow. After switching to a network without a proxy it works. OS: Ubuntu

[constantinpape]

I am pretty sure this is due to biomageio.core. It tries to download some files in the beginning, and this process "gets stuck" behind proxy / within a VPN. I have noticed this before myself.

cc @FynnBe

[FynnBe]

I'll update core soon to avoid additional downloads.

[imagejan]

(probably duplicate of #2680)