Wct save interval

[KuuCi]

What does this PR do?

This PR attempts to support wall clock time as a interval metric. Users will be able to set wall clock time in training yamls to define their save interval. (https://docs.google.com/document/d/1zEs6YKN5YpnwQD-CXjyHlW7R98B3P8ORjEIqA5D8iwI/edit). Specifically, it introduces TimeUnit.SECOND, TimeUnit.MINUTE, TimeUnit.HOUR while keeping all time arithmetic in seconds (minutes and hours are converted to seconds).

manual test: wct-test-sec-v8msnl, wct-test-min-oUF2KU, wct-test-hr-A5zIE0

Checkpoints: https://cloud.oracle.com/object-storage/buckets/axhe5a72vzpp/vincent-test/objects?region=us-phoenix-1 -->

What issue(s) does this change relate to?

N/A

Before submitting

• [x] Have you read the contributor guidelines?
• [ ] Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
• [ ] Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
• [x] Did you update any related docs and document your change?
• [x] Did you update any related tests and add any new tests related to your change? (see testing)
• [x] Did you run the tests locally to make sure they pass?
• [x] Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

[KuuCi]

Working test: wct-test-2m-J59zpP Schedule WCT: wct-test-2m-lIRT2v Max Dur WCT: wct-test-2m-ADkgA4 Both in WCT: wct-test-2m-j8kokf

[KuuCi]

Working: wct-test-2m-UAMkGI Broken Scheduler: wct-test-2m-xJkqik Broken Max Duration: wct-test-2m-NupAgq