Huihuo Zheng
Hello, I was using the unitrace to trace an AI application. Below is part of my output. ``` {"ph": "X", "tid": 4294950910, "pid": 4294950911, "name": "gen9_eltwise_bwd[SIMD32 {1568; 1; 1} {512;...
This allows h5bench to pause the async operations in Cache VOL and start them later after issuing out all the dataset write calls.
Check whether we can adopt the PyTorch S3 support: https://pytorch.org/data/main/generated/torchdata.datapipes.iter.S3FileLoader.html
The DLRM workload support is added here: https://github.com/argonne-lcf/dlio_benchmark/pull/114. But we still need to validate that. I am adding this issue to keep track of that.
Magatron-DeepSpeed is added in https://github.com/argonne-lcf/dlio_benchmark/pull/114. This issue is to keep track of the validating work.
Maybe we can consider the Apache Spark support.
This PR includes changes on * Computation time calculation, instead of putting the value from configuration file. The computation time will be actually the time spend on framework.compute function. *...
In this PR, we changed the per step output from info to debug to reduce the logging overhead. We also add support for changing logging level
In reconfigure, global shuffling is performed. The data loader only have the indices of the local samples. It still has the shuffling on the pytorch / dali data loader which...