New Logger Added (ClearML)

[skinan]

Patch description

I have added ClearML Logger to log experiments using ClearML Task. ClearML logger logs everything to ClearML WebUI. ClearML logs Hyperparameter, Scalars, Models, Checkpoints, other necessary artifacts such as dictionaries, and also Test and Validation Dialogue Worlds as debug samples. Also, I have integrated ClearML Data that can be used to download datasets from ClearML Datasets. Downloading a dataset from ClearML Data is as easy and similar as downloading it from Google Drive or anywhere else. ClearML.

• Added ClearMLLogger Class (Similar to TensorboardLogger and WandbLogger) to parlai/core/logs.py
• Added download_from_clearml method in parlai/core/build_data.py which downloads data from ClearML Data.
• Additional code added to files under parlai/scripts/. These scripts are train_model.py , display_model.py, verify_data.py, eval_model.py, eval_wordstat.py.
• No other scripts or files are changed in terms of adding codes.

ClearML is a leading MLOps stack that can supercharge dialogues research with its state-of-the-art experiment tracking capability. ClearML: https://clear.ml/

What ClearML Experiment Manager can log? Everything! You just name it. Example Screenshots:

[skinan]

@stephenroller , Please have a look for reviewing.

[skinan]

@klshuster , Please have a look for reviews.

[bmartinn]

@skinan there are a bunch of changes to the CI files, any reason for that? for example:

    parallelism: 2 # changed to 4

or in requirements.txt

protobuf~=3.20  # changed to protobuf == 3.20.1

And the need to add python -m pip install -v -r requirements.txt to .circleci/config.yml is odd to me (I would assume this is not the first CI test we run)

wdyt?

[skinan]

@bmartinn , The changes in CircleCI configuration were done as some CircleCI tests were failing due to insufficient disk space error and "protobuf" version mismatch error. Increasing the value of "parallelism" solved the disk out of space issue. In case of installing the requirements.txt through config.yml, I actually forgot why I added this. But it was definitely to solve CircleCI run issues.

[bmartinn]

protobuf~=3.20 means pip is free to install any patch version it needs (so 3.20.1 should be supported) Are you saying all CI is failing due to that? Please revert any changes that are not a direct byproduct of the PR (for example CI improvements) Once we have that let's see what we can do regrading some CI failing

[skinan]

@bmartinn , done. Please have a look.

[bmartinn]

Hi @chiehminwei , it seems some of the CircleCI tests fail on random stuff, I wonder if your commit fixed the two tests that are failing here (1 , 2), any insight?

[chiehminwei]

Hi my internship ended yesterday, you can redirect further questions to Mojtaba or Kurt, thanks!

The commit just updated the test fixtures by running pytest—force-regen test.py

So it should fix the regression tests but shouldn’t affect other things.

On Sat, Aug 20, 2022 at 10:39 AM Martin.B @.***> wrote:

Hi @chiehminwei https://github.com/chiehminwei , it seems some of the CircleCI tests fail on random stuff, I wonder if your commit https://github.com/facebookresearch/ParlAI/pull/4757 fixed the two tests that are failing here (1 https://circleci.com/gh/facebookresearch/ParlAI/95938?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link , 2 https://circleci.com/gh/facebookresearch/ParlAI/95939?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link), any insight?

— Reply to this email directly, view it on GitHub https://github.com/facebookresearch/ParlAI/pull/4636#issuecomment-1221325705, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATHUJ6C6K2DPBRHHESOF4DV2DU37ANCNFSM52CWQRTQ . You are receiving this because you were mentioned.Message ID: @.***>

[bmartinn]

Thanks for the update @chiehminwei ! @mojtaba-komeili any help will be appreciated 😊 (see CI failing, or should it just be ignored in the context of this PR?)

[github-actions[bot]]

This PR has not had activity in 30 days. Closing due to staleness.

[thepycoder]

@klshuster Is there someone who can review this?

[skinan]

Closed as a new pull request has been made instead. Refer to this > #4896