yapapi icon indicating copy to clipboard operation
yapapi copied to clipboard

Published 20 hours ago • verified publisher icon golemfactory

Rejecting multiple tasks

Open johny-b opened this issue 2 years ago • 2 comments

I run the blender example on the current yapapi master, with the task.accept_result() replaced with:

                import random
                if random.random() > 0.7:
                    task.accept_result()
                    print("ACCEPTED")
                else:
                    task.reject_result(retry=True)
                    print("REJECTED")

I'd expect the example to finish, just slower (~~ 10/3 times slower). Yet I've run it few times and have never seen it finish in 5 minutes, while non-modified blender takes ~ 30s.

This might be some temporary devnet problem or whatever. I'm just noting this here for further investigation.

johny-b avatar Mar 10 '22 10:03 johny-b

Sample screen logs (this lasted over 10 minutes):

[2022-03-10T11:45:13.079+0100 INFO yapapi.summary] Received proposals from 9 providers so far
[2022-03-10T11:45:14.061+0100 INFO yapapi.summary] [Job 1] Agreement proposed to provider 'cobalt-six'
[2022-03-10T11:45:14.981+0100 INFO yapapi.summary] [Job 1] Agreement confirmed by provider 'cobalt-six'
[2022-03-10T11:45:16.081+0100 INFO yapapi.summary] Received proposals from 15 providers so far
[2022-03-10T11:45:16.986+0100 INFO yapapi.summary] [Job 1] Agreement proposed to provider 'omx-testnode1'
[2022-03-10T11:45:17.368+0100 INFO yapapi.summary] [Job 1] Task started on provider 'cobalt-six', task data: 0
[2022-03-10T11:45:20.882+0100 INFO yapapi.summary] [Job 1] Agreement confirmed by provider 'omx-testnode1'
[2022-03-10T11:45:22.898+0100 INFO yapapi.summary] [Job 1] Agreement proposed to provider 'glen'
[2022-03-10T11:45:23.211+0100 INFO yapapi.summary] [Job 1] Agreement confirmed by provider 'glen'
[2022-03-10T11:45:26.833+0100 INFO yapapi.summary] [Job 1] Task started on provider 'glen', task data: 10
[2022-03-10T11:45:27.554+0100 INFO yapapi.summary] [Job 1] Terminated agreement with omx-testnode1
[2022-03-10T11:45:27.567+0100 INFO yapapi.summary] [Job 1] Agreement proposed to provider 'mbenke'
ACCEPTED
Task computed: Task(id=1, data=0), result: output_0.png, time: 0:00:10.618696
[2022-03-10T11:45:27.986+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'cobalt-six', task data: 0
[2022-03-10T11:45:27.986+0100 INFO yapapi.summary] [Job 1] Task started on provider 'cobalt-six', task data: 20
[2022-03-10T11:45:28.000+0100 INFO yapapi.summary] [Job 1] Agreement confirmed by provider 'mbenke'
[2022-03-10T11:45:30.952+0100 INFO yapapi.summary] [Job 1] Task started on provider 'mbenke', task data: 30
REJECTED
[2022-03-10T11:45:32.683+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'cobalt-six', task data: 20
[2022-03-10T11:45:32.683+0100 INFO yapapi.summary] [Job 1] Task started on provider 'cobalt-six', task data: 40
REJECTED
[2022-03-10T11:45:35.491+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'glen', task data: 10
[2022-03-10T11:45:35.491+0100 INFO yapapi.summary] [Job 1] Task started on provider 'glen', task data: 20
ACCEPTED
Task computed: Task(id=5, data=40), result: output_40.png, time: 0:00:04.469206
[2022-03-10T11:45:37.156+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'cobalt-six', task data: 40
[2022-03-10T11:45:37.157+0100 INFO yapapi.summary] [Job 1] Task started on provider 'cobalt-six', task data: 10
REJECTED
[2022-03-10T11:45:40.122+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'mbenke', task data: 30
[2022-03-10T11:45:40.122+0100 INFO yapapi.summary] [Job 1] Task started on provider 'mbenke', task data: 50
REJECTED
[2022-03-10T11:45:41.473+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'cobalt-six', task data: 10
[2022-03-10T11:45:41.474+0100 INFO yapapi.summary] [Job 1] Task started on provider 'cobalt-six', task data: 30
REJECTED
[2022-03-10T11:45:42.717+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'glen', task data: 20
REJECTED
[2022-03-10T11:45:46.052+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'cobalt-six', task data: 30
[2022-03-10T11:45:46.052+0100 INFO yapapi.summary] [Job 1] Task started on provider 'glen', task data: 30
ACCEPTED
Task computed: Task(id=6, data=50), result: output_50.png, time: 0:00:07.438030
[2022-03-10T11:45:47.561+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'mbenke', task data: 50
[2022-03-10T11:45:47.561+0100 INFO yapapi.summary] [Job 1] Task started on provider 'mbenke', task data: 10
ACCEPTED
Task computed: Task(id=4, data=30), result: output_30.png, time: 0:00:07.501669
[2022-03-10T11:45:53.554+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'glen', task data: 30
REJECTED
[2022-03-10T11:45:55.121+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'mbenke', task data: 10
[2022-03-10T11:45:55.122+0100 INFO yapapi.summary] [Job 1] Task started on provider 'mbenke', task data: 20
REJECTED
[2022-03-10T11:46:02.668+0100 INFO yapapi.summary] [Job 1] Task finished by provider 'mbenke', task data: 20
^CShutting down gracefully, please wait a short while or press Ctrl+C to exit immediately...
[2022-03-10T11:55:07.171+0100 INFO yapapi.executor] [Job 1] Waiting for 1 worker to finish...
[2022-03-10T11:55:08.764+0100 INFO yapapi.summary] [Job 1] Terminated agreement with mbenke
[2022-03-10T11:55:09.094+0100 INFO yapapi.summary] [Job 1] Terminated agreement with cobalt-six
[2022-03-10T11:55:09.223+0100 INFO yapapi.executor] [Job 1] Waiting for Executor services to finish...
[2022-03-10T11:55:09.224+0100 INFO yapapi.summary] [Job 1] Terminated agreement with glen
[2022-03-10T11:55:09.225+0100 INFO yapapi.executor] Golem is shutting down...
[2022-03-10T11:55:09.227+0100 WARNING yapapi.summary] [Job 1] Job cancelled
[2022-03-10T11:55:09.228+0100 INFO yapapi.summary] [Job 1] Negotiated 4 agreements with 4 providers
[2022-03-10T11:55:09.228+0100 INFO yapapi.summary] [Job 1] Provider 'cobalt-six' computed 5 tasks
[2022-03-10T11:55:09.228+0100 INFO yapapi.summary] [Job 1] Provider 'glen' computed 3 tasks
[2022-03-10T11:55:09.229+0100 INFO yapapi.summary] [Job 1] Provider 'mbenke' computed 4 tasks
[2022-03-10T11:55:09.229+0100 INFO yapapi.summary] [Job 1] Provider 'omx-testnode1' did not compute any tasks
[2022-03-10T11:55:09.231+0100 WARNING yapapi.summary] Execution interrupted by CancelledError
[2022-03-10T11:55:09.231+0100 INFO yapapi.executor] All jobs have finished
[2022-03-10T11:55:09.231+0100 INFO yapapi.executor] 3 agreements still unpaid, waiting for invoices...
[2022-03-10T11:55:09.833+0100 INFO yapapi.summary] [Job 1] Accepted invoice from 'mbenke', amount: 0.03235461809565000
[2022-03-10T11:55:10.226+0100 INFO yapapi.summary] [Job 1] Accepted invoice from 'cobalt-six', amount: 0.003516801118238889
[2022-03-10T11:55:10.353+0100 INFO yapapi.summary] [Job 1] Accepted invoice from 'glen', amount: 0.03185552980400000
[2022-03-10T11:55:10.353+0100 INFO yapapi.executor] Waiting for Golem services to finish...
[2022-03-10T11:55:10.381+0100 INFO yapapi.summary] Total cost: 0.067726949017888889
[2022-03-10T11:55:10.381+0100 INFO yapapi.summary] Golem engine has shut down

johny-b avatar Mar 10 '22 10:03 johny-b

hmm.... it would be good to see the task data for the ACCEPTED/REJECTED messages

shadeofblue avatar Mar 14 '22 16:03 shadeofblue