Adam Moody

parallelize writing of layer checkpoint files across data parallel instances

10

This is work in progress, but I wanted to open it early for discussion. Also, I wrote this before MOE was added, and it will need to be updated to...

romio: hint to allow avoiding extraneous MPI_File_sync calls

14

When not running with atomicity mode enabled, MPI data consistency semantics requires one to call ``MPI_File_sync`` to expose newly written data to make those bytes visible to other processes. However,...

scr: add init/finalize and start/complete output

6

@franzpoeschel, @psychocoderHPC This is getting ahead of things, since it's not yet apparent that SCR would be needed/useful. However, I had started the integration work just in case in might...

bp4: integrate SCR calls

4

@pnorbert , @anagainaru This provides a partial, but workable SCR integration into ADIOS2. I'll open the PR so we have some place to discuss these changes. Most SCR calls are...

How to load a tensor+pipeline parallel checkpoint for inference tasks?

3

Following the training script here as a template: https://github.com/bigscience-workshop/bigscience/blob/master/train/tr1-13B-base/tr1-13B-round1.slurm I've trained some models using 2-way tensor parallelism and 4-way pipeline parallelism, which produces a number of checkpoints in directories like...

I have a PR open on Microsoft's DeepSpeed page that parallelizes the task of writing per-layer checkpoint files across data parallel instances: https://github.com/microsoft/DeepSpeed/pull/1419 On my system, I found that this...

extend preprocess_data_dist to handle jsonl files

9

This extends ``tools/preprocess_dataset_dist.py`` to handle JSONL files as an input dataset. It defines a new IndexedJSON class in ``tools/indexed_json.py`` that creates and uses an index for the JSONL file. The...

This is solid enough that I'll go ahead and post a WIP PR. It's based on https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/60, so this will look noisy until that PR is merged. Most of the...

Calling IndexedDatasetBuilder directly with a best_fit datatype fails

1

I don't know whether this is intended to work or not, but I found the following program: ``` from megatron.data.indexed_dataset import IndexedDatasetBuilder, best_fitting_dtype best_dtype = best_fitting_dtype(10_000) IndexedDatasetBuilder("testfile", dtype=best_dtype) ``` leads...

[FEEDSTOCK REQUEST] DeepSpeed v0.3.16

19

**Describe the package you'd like added** Are there plans to add Microsoft's DeepSpeed support for PyTorch? https://github.com/microsoft/DeepSpeed **Describe how this package fits in with the project** **Additional context** I'm trying...