openwhisk-deploy-kube icon indicating copy to clipboard operation
openwhisk-deploy-kube copied to clipboard

Lean OW with K8s container Factory

Open MatheoAtche opened this issue 3 years ago • 0 comments

Hello !

I am trying to deploy the lean version of Openwhisk on a Kubernetes cluster. I have encountered an issue when I choose k8s as the container factory if .Values.controller,lean is set to true. The chart deployement seems to be fine, but when I try to invoke an action, it times out every time. Here are the controller logs related to an activation, it seems that the controller can't contact the action pod to perform the initialization. I didn't find why this request is timing out.

FYI when I deploy OW using lean OW and Docker container factory, or without lean OW and with k8s container factory, I can invoke an action.

[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [InvokerReactive]  [marker:invoker_activation_start:12]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [WhiskAction] [GET] serving from cache: CacheKey(whisk.system/samples/greeting) [marker:database_cacheHit_counter:13]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [ContainerPool] containerStart containerState: cold container: None activations: 1 of max 2 action: greeting namespace: guest activationId: a9870dad3a0440c2870dad3a04c0c2f1 [marker:invoker_containerStart.cold_counter:40]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [KubernetesClient] launching pod wsk0-2-guest-greeting (image:openwhisk/action-nodejs-v14:1.19.0, mem: 256) (timeout: 60s) [marker:invoker_kubeapi.create_start:43]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [KubernetesClient]  [marker:invoker_kubeapi.create_finish:2142:2099]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [KubernetesContainer] sending initialization to ContainerId(wsk0-2-guest-greeting) ContainerAddress(172.16.130.150,8080) [marker:invoker_activationInit_start:2184]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore] [GET] 'test_activations' finding document: 'id: guest/a9870dad3a0440c2870dad3a04c0c2f1' [marker:database_getDocument_start:15031]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore]  [marker:database_getDocument_finish:15061:30]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore] [GET] 'test_activations' finding document: 'id: guest/a9870dad3a0440c2870dad3a04c0c2f1' [marker:database_getDocument_start:30071]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore]  [marker:database_getDocument_finish:30081:10]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore] [GET] 'test_activations' finding document: 'id: guest/a9870dad3a0440c2870dad3a04c0c2f1' [marker:database_getDocument_start:45101]
 [INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore]  [marker:database_getDocument_finish:45112:10]
[INFO] [#tid_sid_invoker] [CouchDbRestStore] [QUERY] 'test_subjects' searching 'namespaceThrottlings/blockedNamespaces [marker:database_queryView_start:302151]
[INFO] [#tid_sid_invoker] [CouchDbRestStore]  [marker:database_queryView_finish:302179:28]
[INFO] [#tid_sid_unknown] [InvokerReactive] updated blacklist to 0 entries
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [ActionsApi]  [marker:controller_blockingActivation_finish:60030:60019]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [BasicHttpService] [marker:http_post.202_counter:60032:60032]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [KubernetesContainer] initialization result: ConnectionError(org.apache.http.conn.ConnectTimeoutException: Connect to 172.16.130.150:8080 [/172.16.130.150] failed: connect timed out) [marker:invoker_activationInit_finish:62247:60062]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [ContainerProxy]  [marker:invoker_collectLogs_start:62248]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [MessagingActiveAck] posted result of activation a9870dad3a0440c2870dad3a04c0c2f1
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [LeanBalancer] received result ack for 'a9870dad3a0440c2870dad3a04c0c2f1'
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [ContainerProxy]  [marker:invoker_collectLogs_finish:64271:2023]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore] [PUT] 'test_activations' saving document: 'id: guest/a9870dad3a0440c2870dad3a04c0c2f1, rev: null' [marker:database_saveDocument_start:64273]
[INFO] [#tid_sid_dbBatcher] [CouchDbRestStore] 'test_activations' saving 1 documents [marker:database_saveDocumentBulk_start:308093]
[ERROR] [#tid_sid_unknown] [ContainerProxy] Failed during init of cold container Some(ContainerId(wsk0-2-guest-greeting)), queued activations will be aborted.
[INFO] [#tid_sid_invokerNanny] [KubernetesClient] Deleting pod wsk0-2-guest-greeting [marker:invoker_kubeapi.delete_start:308094]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [MessagingActiveAck] posted completion of activation a9870dad3a0440c2870dad3a04c0c2f1
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [LeanBalancer] received completion ack for 'a9870dad3a0440c2870dad3a04c0c2f1', system error=false
[INFO] Message [org.apache.openwhisk.core.loadBalancer.InvocationFinishedMessage] to Actor[akka://controller-actor-system/user/$g#1140144961] was unhandled. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [#tid_sid_invokerNanny] [KubernetesClient]  [marker:invoker_kubeapi.delete_finish:308110:15]
[INFO] [#tid_sid_dbBatcher] [CouchDbRestStore]  [marker:database_saveDocumentBulk_finish:308118:24]
[INFO] [#tid_7si1aIddlolBDjS4qPklWJcWrk1T6enp] [CouchDbRestStore]  [marker:database_saveDocument_finish:64299:26]

Thanks for your time and help !

MatheoAtche avatar Aug 09 '22 04:08 MatheoAtche