[WIP] Error logging callback

[bmosaicml]

trafficstars

What does this PR do?

This PR adds a callback that logs ICL outputs during eval. It modifies the custom metrics to keep track of incorrect model outputs. Each metric is responsible for specifying the table schema for logging cached responses as well as specifying how to format the cached responses using the tokenizer.

The EvalOutputLogging callback is then responsible for logging the cached results in table format after each evaluation.

Design doc

What issue(s) does this change relate to?

Before submitting

• [x] Have you read the contributor guidelines?
• [ ] Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
• [ ] Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
• [x] Did you update any related docs and document your change?
• [x] Did you update any related tests and add any new tests related to your change? (see testing)
• [x] Did you run the tests locally to make sure they pass?
• [x] Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

[maxisawesome]

Looks good except for my one comment. Would be nice if there was an option to save results to a file or the cloud but that might be better suited for a different PR doing it for all evals

[bmosaicml]

Looks good except for my one comment. Would be nice if there was an option to save results to a file or the cloud but that might be better suited for a different PR doing it for all evals

I believe this will work out of the box as composer's logger functionality will include wandb/s3/etc in the logger destinations and then log them all simultaneously. I am going to mock up a test branch on foundry that tests s3 and wandb!

[maxisawesome]

Successfully merged here