lithops icon indicating copy to clipboard operation
lithops copied to clipboard

Published 20 hours ago • verified publisher icon lithops-cloud

[k8s] Job gets stuck in pending state

Open bystepii opened this issue 6 months ago • 16 comments

I have the following code that I am trying to run on a k8s cluster using lithops:

from lithops import FunctionExecutor
from time import sleep

def map_fn(i):
    sleep(3)

def red_fn(i):
    sleep(2)

fexec = FunctionExecutor()
for i in range(1000):
    print(f"i = {i}")
    fexec.map_reduce(map_fn, [1,2,3,4], red_fn)
    res = fexec.get_result()

At some random iteration, a job gets stuck in the pending state and never gets executed. It can be a mapper as well as a reducer.

Config

lithops:
 backend: k8s
 storage: minio
 log_level: DEBUG

k8s:
 docker_server: docker.io
 docker_user: stepii
 docker_password: ******
 runtime_memory: 10240
 runtime_cpu: 6
 worker_processes: 1
 runtime_timeout: 900
 namespace: indexing

minio:
 storage_bucket: stepan
 endpoint: http://192.168.5.24:9000
 access_key_id: ******
 secret_access_key: ******

Logs

In the example execution, the job R027 is stuck in the pending state.

Lithops logs:

[...]
i = 27
2024-07-31 13:04:49,542 [INFO] invokers.py:119 -- ExecutorID 7046b1-0 | JobID M027 - Selected Runtime: docker.io/stepii/lithops-kubernetes-default-v310:3.4.2.dev0 - 10240MB
2024-07-31 13:04:49,542 [DEBUG] storage.py:472 -- Runtime metadata found in local memory cache
2024-07-31 13:04:49,542 [DEBUG] job.py:242 -- ExecutorID 7046b1-0 | JobID M027 - Serializing function and data
2024-07-31 13:04:49,544 [DEBUG] serialize.py:75 -- Referenced Modules: __main__, time
2024-07-31 13:04:49,544 [DEBUG] module_dependency.py:172 -- Module 'time' is built-in. Skipping.
2024-07-31 13:04:49,544 [DEBUG] serialize.py:120 -- Modules to transmit: None
2024-07-31 13:04:49,544 [DEBUG] job.py:284 -- ExecutorID 7046b1-0 | JobID M027 - Function and modules found in local cache
2024-07-31 13:04:49,544 [DEBUG] job.py:302 -- ExecutorID 7046b1-0 | JobID M027 - Uploading data to the storage backend
2024-07-31 13:04:49,631 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/aggdata.pickle - Size: 84.0B - OK
2024-07-31 13:04:49,631 [INFO] invokers.py:186 -- ExecutorID 7046b1-0 | JobID M027 - Starting function invocation: map_fn() - Total: 4 activations
2024-07-31 13:04:49,631 [DEBUG] invokers.py:212 -- ExecutorID 7046b1-0 | JobID M027 - Worker processes: 1 - Chunksize: 1
2024-07-31 13:04:49,645 [DEBUG] k8s.py:632 -- ExecutorID 7046b1-0 | JobID M027 - Required Workers: 4
2024-07-31 13:04:49,653 [DEBUG] k8s.py:661 -- ExecutorID 7046b1-0 | JobID M027 - Going to run 4 activations in 4 workers
2024-07-31 13:04:49,668 [DEBUG] invokers.py:271 -- ExecutorID 7046b1-0 | JobID M027 - Job invoked (0.037s) - Activation ID: lithops-7046b1-0-m027
2024-07-31 13:04:49,668 [INFO] invokers.py:225 -- ExecutorID 7046b1-0 | JobID M027 - View execution logs at /tmp/lithops-stepii/logs/7046b1-0-M027.log
2024-07-31 13:04:49,669 [DEBUG] monitor.py:429 -- ExecutorID 7046b1-0 - Starting Storage job monitor
2024-07-31 13:04:49,669 [INFO] wait.py:105 -- ExecutorID 7046b1-0 - Waiting for 20% of 4 function activations to complete
2024-07-31 13:04:51,708 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 4 - Running: 0 - Done: 0
2024-07-31 13:04:52,141 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 3 - Running: 1 - Done: 0
2024-07-31 13:04:52,574 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 0 - Running: 4 - Done: 0
2024-07-31 13:04:55,681 [DEBUG] future.py:229 -- ExecutorID 7046b1-0 | JobID M027 - Got status from call 00000 - Activation ID: ce49b1360754 - Time: 3.20 seconds
2024-07-31 13:04:55,682 [DEBUG] future.py:229 -- ExecutorID 7046b1-0 | JobID M027 - Got status from call 00001 - Activation ID: 5e9ab076c1c4 - Time: 3.15 seconds
2024-07-31 13:04:55,682 [DEBUG] future.py:229 -- ExecutorID 7046b1-0 | JobID M027 - Got status from call 00003 - Activation ID: d603959a2909 - Time: 3.17 seconds
2024-07-31 13:04:55,683 [DEBUG] executors.py:366 -- ExecutorID 7046b1-0 | JobID M027 - 20% of map activations done. Spawning reduce stage
2024-07-31 13:04:55,683 [INFO] invokers.py:119 -- ExecutorID 7046b1-0 | JobID R027 - Selected Runtime: docker.io/stepii/lithops-kubernetes-default-v310:3.4.2.dev0 - 10240MB
2024-07-31 13:04:55,683 [DEBUG] storage.py:472 -- Runtime metadata found in local memory cache
2024-07-31 13:04:55,683 [DEBUG] job.py:185 -- Extra environment vars {'__LITHOPS_REDUCE_JOB': 'True'}
2024-07-31 13:04:55,683 [DEBUG] job.py:242 -- ExecutorID 7046b1-0 | JobID R027 - Serializing function and data
2024-07-31 13:04:55,684 [DEBUG] serialize.py:75 -- Referenced Modules: __main__, time
2024-07-31 13:04:55,684 [DEBUG] module_dependency.py:172 -- Module 'time' is built-in. Skipping.
2024-07-31 13:04:55,684 [DEBUG] serialize.py:120 -- Modules to transmit: None
2024-07-31 13:04:55,685 [DEBUG] job.py:284 -- ExecutorID 7046b1-0 | JobID R027 - Function and modules found in local cache
2024-07-31 13:04:55,685 [DEBUG] job.py:302 -- ExecutorID 7046b1-0 | JobID R027 - Uploading data to the storage backend
2024-07-31 13:04:55,741 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-R027/aggdata.pickle - Size: 13.6KiB - OK
2024-07-31 13:04:55,741 [INFO] invokers.py:186 -- ExecutorID 7046b1-0 | JobID R027 - Starting function invocation: red_fn() - Total: 1 activations
2024-07-31 13:04:55,741 [DEBUG] invokers.py:212 -- ExecutorID 7046b1-0 | JobID R027 - Worker processes: 1 - Chunksize: 1
2024-07-31 13:04:55,758 [DEBUG] k8s.py:632 -- ExecutorID 7046b1-0 | JobID R027 - Required Workers: 1
2024-07-31 13:04:55,766 [DEBUG] k8s.py:661 -- ExecutorID 7046b1-0 | JobID R027 - Going to run 1 activations in 1 workers
2024-07-31 13:04:55,782 [DEBUG] invokers.py:271 -- ExecutorID 7046b1-0 | JobID R027 - Job invoked (0.041s) - Activation ID: lithops-7046b1-0-r027
2024-07-31 13:04:55,782 [INFO] invokers.py:225 -- ExecutorID 7046b1-0 | JobID R027 - View execution logs at /tmp/lithops-stepii/logs/7046b1-0-R027.log
2024-07-31 13:04:55,782 [INFO] executors.py:494 -- ExecutorID 7046b1-0 - Getting results from 28 function activations
2024-07-31 13:04:55,783 [INFO] wait.py:101 -- ExecutorID 7046b1-0 - Waiting for 2 function activations to complete
2024-07-31 13:04:55,918 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 1 - Running: 0 - Done: 4
2024-07-31 13:04:56,786 [DEBUG] future.py:229 -- ExecutorID 7046b1-0 | JobID M027 - Got status from call 00002 - Activation ID: d48d456eee35 - Time: 3.20 seconds
2024-07-31 13:05:02,704 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 1 - Running: 0 - Done: 4
2024-07-31 13:05:09,433 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 1 - Running: 0 - Done: 4
2024-07-31 13:05:16,152 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 1 - Running: 0 - Done: 4
2024-07-31 13:05:22,877 [DEBUG] monitor.py:144 -- ExecutorID 7046b1-0 - Pending: 1 - Running: 0 - Done: 4

K8s logs for lithops master (pod lithops-master-2304d41a58-s7jsx):

[...]
10.244.140.136 - - [31/Jul/2024 13:04:51] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:51,829] INFO in lithopsentry: Sending range "1-2" to Host 10.244.235.216
10.244.235.216 - - [31/Jul/2024 13:04:51] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:51,911] INFO in lithopsentry: Sending range "2-3" to Host 10.244.140.172
10.244.140.172 - - [31/Jul/2024 13:04:51] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:51,930] INFO in lithopsentry: Sending range "3-4" to Host 10.244.235.194
10.244.235.194 - - [31/Jul/2024 13:04:51] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:51,935] INFO in lithopsentry: Sending range "-1" to Host 10.244.140.190
10.244.140.190 - - [31/Jul/2024 13:04:51] "GET /get-range/7046b1-0-M026/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:52,053] INFO in lithopsentry: Sending range "-1" to Host 10.244.235.228
10.244.235.228 - - [31/Jul/2024 13:04:52] "GET /get-range/7046b1-0-M026/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:52,112] INFO in lithopsentry: Sending range "-1" to Host 10.244.235.242
10.244.235.242 - - [31/Jul/2024 13:04:52] "GET /get-range/7046b1-0-M026/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:52,171] INFO in lithopsentry: Sending range "-1" to Host 10.244.140.143
10.244.140.143 - - [31/Jul/2024 13:04:52] "GET /get-range/7046b1-0-M026/4/1 HTTP/1.1" 200 -
[2024-07-31 13:04:57,983] INFO in lithopsentry: Sending range "0-1" to Host 10.244.140.142
10.244.140.142 - - [31/Jul/2024 13:04:58] "GET /get-range/7046b1-0-R027/1/1 HTTP/1.1" 200 -
[2024-07-31 13:04:58,184] INFO in lithopsentry: Sending range "-1" to Host 10.244.140.142
10.244.140.142 - - [31/Jul/2024 13:04:58] "GET /get-range/7046b1-0-R027/1/1 HTTP/1.1" 200 -
[2024-07-31 13:04:59,130] INFO in lithopsentry: Sending range "-1" to Host 10.244.140.171
10.244.140.171 - - [31/Jul/2024 13:04:59] "GET /get-range/7046b1-0-R026/1/1 HTTP/1.1" 200 -
[2024-07-31 13:05:05,414] INFO in lithopsentry: Sending range "-1" to Host 10.244.140.136
10.244.140.136 - - [31/Jul/2024 13:05:05] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -
[2024-07-31 13:05:06,656] INFO in lithopsentry: Sending range "-1" to Host 10.244.140.172
10.244.140.172 - - [31/Jul/2024 13:05:06] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -
[2024-07-31 13:05:06,712] INFO in lithopsentry: Sending range "-1" to Host 10.244.235.194
10.244.235.194 - - [31/Jul/2024 13:05:06] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -
[2024-07-31 13:05:06,714] INFO in lithopsentry: Sending range "-1" to Host 10.244.235.216
10.244.235.216 - - [31/Jul/2024 13:05:06] "GET /get-range/7046b1-0-M027/4/1 HTTP/1.1" 200 -

K8s logs for the job R027 (pod lithops-7046b1-0-r027-mjz9c):

2024-07-31 13:04:57,977 [INFO] lithopsentry.py:84 -- Lithops v3.4.2.dev0 - Starting kubernetes execution
2024-07-31 13:04:58,187 [INFO] lithopsentry.py:111 -- Received range: -1
2024-07-31 13:04:58,187 [INFO] lithopsentry.py:122 -- Finishing kubernetes execution

K8s logs for the mapper jobs M027:

pod lithops-7046b1-0-m027-57tn9:

2024-07-31 13:04:51,922 [INFO] lithopsentry.py:84 -- Lithops v3.4.2.dev0 - Starting kubernetes execution
2024-07-31 13:04:51,929 [INFO] lithopsentry.py:111 -- Received range: 3-4
2024-07-31 13:04:52,086 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:52,086 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:52,260 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:52,268 [INFO] utils.py:50 -- Getting function and modules
2024-07-31 13:04:52,268 [INFO] utils.py:62 -- Loading lithops.jobs/7046b1-0/c9195dee50e04507b9a5f2d1019afc17.func.pickle from storage
2024-07-31 13:04:52,288 [INFO] utils.py:107 -- Loading function data parameters from storage
2024-07-31 13:04:52,297 [INFO] handler.py:75 -- Tasks received: 1 - Worker processes: 1
2024-07-31 13:04:52,297 [INFO] handler.py:117 -- Worker process 0 started
2024-07-31 13:04:52,299 [INFO] handler.py:177 -- Lithops v3.4.2.dev0 - Starting k8s execution
2024-07-31 13:04:52,299 [INFO] handler.py:178 -- Execution ID: 7046b1-0-M027/00003
2024-07-31 13:04:52,299 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:52,299 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:52,306 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:52,312 [DEBUG] handler.py:194 -- Runtime: docker.io/stepii/lithops-kubernetes-default-v310:3.4.2.dev0 - Memory: 10240MB - Timeout: 895 seconds
2024-07-31 13:04:52,413 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00003/d603959a2909.init - Size: 0.0B - OK
2024-07-31 13:04:52,414 [DEBUG] handler.py:206 -- Starting JobRunner process
2024-07-31 13:04:52,432 [DEBUG] jobrunner.py:204 -- Process started
2024-07-31 13:04:52,434 [INFO] jobrunner.py:234 -- Going to execute 'map_fn()'
---------------------- FUNCTION LOG ----------------------
----------------------------------------------------------
2024-07-31 13:04:55,437 [INFO] jobrunner.py:240 -- Success function execution
2024-07-31 13:04:55,439 [INFO] jobrunner.py:312 -- Process finished
2024-07-31 13:04:55,467 [DEBUG] handler.py:217 -- JobRunner process finished
2024-07-31 13:04:55,468 [INFO] status.py:88 -- Storing execution stats - Size: 2.1KiB
2024-07-31 13:05:06,705 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00003/status.json - Size: 2.1KiB - OK
2024-07-31 13:05:06,705 [INFO] handler.py:292 -- Finished
2024-07-31 13:05:06,705 [INFO] handler.py:139 -- Worker process 0 finished
2024-07-31 13:05:06,712 [INFO] lithopsentry.py:111 -- Received range: -1
2024-07-31 13:05:06,712 [INFO] lithopsentry.py:122 -- Finishing kubernetes execution

pod lithops-7046b1-0-m027-g7k9k:

2024-07-31 13:04:51,693 [INFO] lithopsentry.py:84 -- Lithops v3.4.2.dev0 - Starting kubernetes execution
2024-07-31 13:04:51,700 [INFO] lithopsentry.py:111 -- Received range: 0-1
2024-07-31 13:04:51,817 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:51,817 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:51,950 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:51,958 [INFO] utils.py:50 -- Getting function and modules
2024-07-31 13:04:51,958 [INFO] utils.py:69 -- Loading lithops.jobs/7046b1-0/c9195dee50e04507b9a5f2d1019afc17.func.pickle from storage
2024-07-31 13:04:51,978 [INFO] utils.py:114 -- Loading function data parameters from storage
2024-07-31 13:04:52,049 [INFO] handler.py:75 -- Tasks received: 1 - Worker processes: 1
2024-07-31 13:04:52,049 [INFO] handler.py:117 -- Worker process 0 started
2024-07-31 13:04:52,051 [INFO] handler.py:177 -- Lithops v3.4.2.dev0 - Starting k8s execution
2024-07-31 13:04:52,051 [INFO] handler.py:178 -- Execution ID: 7046b1-0-M027/00000
2024-07-31 13:04:52,051 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:52,051 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:52,056 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:52,061 [DEBUG] handler.py:194 -- Runtime: docker.io/stepii/lithops-kubernetes-default-v310:3.4.2.dev0 - Memory: 10240MB - Timeout: 895 seconds
2024-07-31 13:04:52,191 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00000/ce49b1360754.init - Size: 0.0B - OK
2024-07-31 13:04:52,192 [DEBUG] handler.py:206 -- Starting JobRunner process
2024-07-31 13:04:52,213 [DEBUG] jobrunner.py:204 -- Process started
2024-07-31 13:04:52,215 [INFO] jobrunner.py:234 -- Going to execute 'map_fn()'
---------------------- FUNCTION LOG ----------------------
----------------------------------------------------------
2024-07-31 13:04:55,219 [INFO] jobrunner.py:240 -- Success function execution
2024-07-31 13:04:55,221 [INFO] jobrunner.py:312 -- Process finished
2024-07-31 13:04:55,248 [DEBUG] handler.py:217 -- JobRunner process finished
2024-07-31 13:04:55,250 [INFO] status.py:88 -- Storing execution stats - Size: 2.1KiB
2024-07-31 13:05:05,409 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00000/status.json - Size: 2.1KiB - OK
2024-07-31 13:05:05,409 [INFO] handler.py:292 -- Finished
2024-07-31 13:05:05,410 [INFO] handler.py:139 -- Worker process 0 finished
2024-07-31 13:05:05,417 [INFO] lithopsentry.py:111 -- Received range: -1
2024-07-31 13:05:05,417 [INFO] lithopsentry.py:122 -- Finishing kubernetes execution

pod lithops-7046b1-0-m027-kc4pr:

2024-07-31 13:04:51,821 [INFO] lithopsentry.py:84 -- Lithops v3.4.2.dev0 - Starting kubernetes execution
2024-07-31 13:04:51,829 [INFO] lithopsentry.py:111 -- Received range: 1-2
2024-07-31 13:04:51,975 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:51,976 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:52,148 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:52,157 [INFO] utils.py:50 -- Getting function and modules
2024-07-31 13:04:52,157 [INFO] utils.py:62 -- Loading lithops.jobs/7046b1-0/c9195dee50e04507b9a5f2d1019afc17.func.pickle from storage
2024-07-31 13:04:52,188 [INFO] utils.py:107 -- Loading function data parameters from storage
2024-07-31 13:04:52,199 [INFO] handler.py:75 -- Tasks received: 1 - Worker processes: 1
2024-07-31 13:04:52,199 [INFO] handler.py:117 -- Worker process 0 started
2024-07-31 13:04:52,201 [INFO] handler.py:177 -- Lithops v3.4.2.dev0 - Starting k8s execution
2024-07-31 13:04:52,201 [INFO] handler.py:178 -- Execution ID: 7046b1-0-M027/00001
2024-07-31 13:04:52,201 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:52,201 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:52,209 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:52,216 [DEBUG] handler.py:194 -- Runtime: docker.io/stepii/lithops-kubernetes-default-v310:3.4.2.dev0 - Memory: 10240MB - Timeout: 895 seconds
2024-07-31 13:04:52,289 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00001/5e9ab076c1c4.init - Size: 0.0B - OK
2024-07-31 13:04:52,290 [DEBUG] handler.py:206 -- Starting JobRunner process
2024-07-31 13:04:52,309 [DEBUG] jobrunner.py:204 -- Process started
2024-07-31 13:04:52,311 [INFO] jobrunner.py:234 -- Going to execute 'map_fn()'
---------------------- FUNCTION LOG ----------------------
----------------------------------------------------------
2024-07-31 13:04:55,315 [INFO] jobrunner.py:240 -- Success function execution
2024-07-31 13:04:55,317 [INFO] jobrunner.py:312 -- Process finished
2024-07-31 13:04:55,346 [DEBUG] handler.py:217 -- JobRunner process finished
2024-07-31 13:04:55,347 [INFO] status.py:88 -- Storing execution stats - Size: 2.1KiB
2024-07-31 13:05:06,705 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00001/status.json - Size: 2.1KiB - OK
2024-07-31 13:05:06,705 [INFO] handler.py:292 -- Finished
2024-07-31 13:05:06,705 [INFO] handler.py:139 -- Worker process 0 finished
2024-07-31 13:05:06,714 [INFO] lithopsentry.py:111 -- Received range: -1
2024-07-31 13:05:06,715 [INFO] lithopsentry.py:122 -- Finishing kubernetes execution

pod lithops-7046b1-0-m027-pvcfh:

2024-07-31 13:04:51,905 [INFO] lithopsentry.py:84 -- Lithops v3.4.2.dev0 - Starting kubernetes execution
2024-07-31 13:04:51,913 [INFO] lithopsentry.py:111 -- Received range: 2-3
2024-07-31 13:04:52,082 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:52,082 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:52,254 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:52,263 [INFO] utils.py:50 -- Getting function and modules
2024-07-31 13:04:52,263 [INFO] utils.py:69 -- Loading lithops.jobs/7046b1-0/c9195dee50e04507b9a5f2d1019afc17.func.pickle from storage
2024-07-31 13:04:52,290 [INFO] utils.py:114 -- Loading function data parameters from storage
2024-07-31 13:04:52,299 [INFO] handler.py:75 -- Tasks received: 1 - Worker processes: 1
2024-07-31 13:04:52,299 [INFO] handler.py:117 -- Worker process 0 started
2024-07-31 13:04:52,301 [INFO] handler.py:177 -- Lithops v3.4.2.dev0 - Starting k8s execution
2024-07-31 13:04:52,301 [INFO] handler.py:178 -- Execution ID: 7046b1-0-M027/00002
2024-07-31 13:04:52,302 [DEBUG] minio.py:38 -- Creating MinIO client
2024-07-31 13:04:52,302 [DEBUG] minio.py:43 -- Setting MinIO endpoint to http://192.168.5.24:9000
2024-07-31 13:04:52,308 [INFO] minio.py:62 -- MinIO client created - Endpoint: http://192.168.5.24:9000
2024-07-31 13:04:52,315 [DEBUG] handler.py:194 -- Runtime: docker.io/stepii/lithops-kubernetes-default-v310:3.4.2.dev0 - Memory: 10240MB - Timeout: 895 seconds
2024-07-31 13:04:52,440 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00002/d48d456eee35.init - Size: 0.0B - OK
2024-07-31 13:04:52,441 [DEBUG] handler.py:206 -- Starting JobRunner process
2024-07-31 13:04:52,458 [DEBUG] jobrunner.py:204 -- Process started
2024-07-31 13:04:52,461 [INFO] jobrunner.py:234 -- Going to execute 'map_fn()'
---------------------- FUNCTION LOG ----------------------
----------------------------------------------------------
2024-07-31 13:04:55,464 [INFO] jobrunner.py:240 -- Success function execution
2024-07-31 13:04:55,466 [INFO] jobrunner.py:312 -- Process finished
2024-07-31 13:04:55,496 [DEBUG] handler.py:217 -- JobRunner process finished
2024-07-31 13:04:55,497 [INFO] status.py:88 -- Storing execution stats - Size: 2.1KiB
2024-07-31 13:05:06,652 [DEBUG] minio.py:110 -- PUT Object lithops.jobs/7046b1-0-M027/00002/status.json - Size: 2.1KiB - OK
2024-07-31 13:05:06,652 [INFO] handler.py:292 -- Finished
2024-07-31 13:05:06,652 [INFO] handler.py:139 -- Worker process 0 finished
2024-07-31 13:05:06,659 [INFO] lithopsentry.py:111 -- Received range: -1
2024-07-31 13:05:06,659 [INFO] lithopsentry.py:122 -- Finishing kubernetes execution

bystepii avatar Jul 31 '24 13:07 bystepii