Usability Improvements

[wvaske]

After the 1.0 submission we found that usability of the benchmark can be greatly improved. This issue will track the 'sub-issues' we intend to address for the 2.0 release.

Please add any items in comments and I will update this top level comment. Feel free to attend the sub-working group meeting (bi-weekly Wednesday morning starting on Nov 20th). Join the MLPerf Storage working group for the invite or message me.

Tasks

Rules Document

• [ ] Define filesystem caching rules in detail
• [ ] Define system json schema and creation process
• [ ] Define allowed time between runs
• [ ] Define rules that use local SSD for caching data
• [ ] Define rules for hyperconverged and local cache

benchmark[.py | .sh] script

• [ ] Unique names for files and directories with structure for benchmark, accelerator, count, run-sequence, run-number
• [ ] Better installer that manages dependencies
• [ ] Containerization
• • [ ] Ease of Deployment of Benchmark (just get it working)
• • [ ] Cgroups and resource limits (better cache management)
• [ ] Flush Cache before a run
• [ ] Validate inputs for –closed runs (eg: don’t allow runs against datasets that are too small)
• [ ] Reportgen should run validation against outputs
• [ ] Add better system.json creation to automate the system description for consistency
• • [ ] Add json schema checker for system documents that submitters create
• [ ] Automate execution of multiple runs
• [ ] Add support for code changes in closed to supported categories [ data loader, s3 connector, etc]
• • [ ] Add patches directory that gets applied before execution
• [ ] Add runtime estimation and --what-if or --dry-run flag
• [ ] Automate selection of minimum required dataset
• [ ] Determine if batch sizes in MLPerf Training are representative of batch sizes for realistically sized datasets
• [ ] Split system.json into automatically capturable (clients) and manual (storage)
• [ ] Define system.json schema and add schema checker to the tool for reportgen
• [ ] Add report-dir csv of results from tests as they are run
• [ ] Collect versions of all prerequisite packages for storage and dlio

DLIO Improvements

• [ ] Reduce verbosity of logging
• [ ] Add callback handler for custom monitoring
• • [ ] SPECStorage uses a “PRIME_MON_SCRIPT” environment variable that will execute at different times
• • [ ] Checkpoint_bench uses RPC to call execution which can be wrapped externally
• [ ] Add support for DIRECTIO
• [ ] Add seed for dataset creation so that distribution of sizes is the same for all submitters (file 1 = mean + x bytes, file 2 = mean + y bytes, etc)
• [ ] Determine if global barrier for each batch matches industry behavior

Results Presentation

• [ ] Better linking and presentation of system diagrams (add working links to system diagrams to supplementals)
• [ ] Define presentation and rules for hyperconverged or systems with local cache