chore: update decorators from backoff python module

[jagadeesh545]

Description

backoff.on_exception() method in the logging script is failing due to invalid arguments. Please find the stack trace below.

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/run/determined/enrich_task_logs.py", line 121, in run
    self.ship()
  File "/opt/conda/lib/python3.7/site-packages/backoff/_sync.py", line 92, in retry
    wait = _init_wait_gen(wait_gen, wait_gen_kwargs)
  File "/opt/conda/lib/python3.7/site-packages/backoff/_common.py", line 30, in _init_wait_gen
    initialized.send(None)  # Initialize with an empty send
AttributeError: 'float' object has no attribute 'send'

After investigation, it was identified that the reason for this failure is that the first argument to the backoff.on_exception() method was a lambda function instead of a generator (shown below).

@backoff.on_exception(  # type: ignore
        lambda: backoff.full_jitter(SHIPPER_FAILURE_BACKOFF_SECONDS),
        errors.APIException,
        max_tries=3,
    )

Examples of correct usage of the backoff.on_exception() method can be found here. The first argument should be one of the generators - expo, fibo, constant, runtime, as shown in the documentation mentioned above. An example of the correct usage is provided below.

@backoff.on_exception(  # type: ignore
        backoff.constant,
        errors.APIException,
        max_tries=3,
    )

This PR provides corrections for the backoff.on_exception() method.

Test Plan

Tested manually.

Commentary (optional)

Checklist

• [ ] Changes have been manually QA'd
• [ ] User-facing API changes need the "User-facing API Change" label.
• [ ] Release notes should be added as a separate file under docs/release-notes/. See Release Note for details.
• [ ] Licenses should be included for new code which was copied and/or modified from any external code.
• [ ] If modifying /webui/react/src/shared/ verify make -C webui/react test-shared passes.

[netlify[bot]]

✅ Deploy Preview for determined-ui canceled.

Name	Link
🔨 Latest commit	3591f971bc34cfc7aa76957a04fb225eaec58335
🔍 Latest deploy log	https://app.netlify.com/sites/determined-ui/deploys/63fd2cb643bd560008846f2b

[netlify[bot]]

✅ Deploy Preview for storybook-det canceled.

Name	Link
🔨 Latest commit	c4be3fa5a34f54d067163b9b69c724cb41ecbb38
🔍 Latest deploy log	https://app.netlify.com/sites/storybook-det/deploys/6321ed1452f6ab000952a951

[stoksc]

included @azhou-determined so someone from mlsys is aware

[stoksc]

This library is unpinned in envs and v1 -> v2 had a breaking change

[stoksc]

I think we can close this. Iirc it was already fixed somehow or another.

[jagadeesh545]

This is still an open issue. I added a test in the file master/static/srv/test_enrich_task_logs.py

[jagadeesh545]

PR is ready to merge. Please merge it. @azhou-determined

[rb-determined-ai]

I'll be taking over the ml-sys review of this one for @azhou-determined.

@jagadeesh545, @stoksc what's the current status on this one? enrich_logs.py is gone now, for instance.

[azhou-determined]

the backoff module was removed in https://github.com/determined-ai/determined/pull/6590, since it was only doing constant backoff. i think this PR can be closed.

[rb-determined-ai]

cool, also I heard from @jagadeesh545 out-of-band, he agrees it can be closed.