hivemind

hivemind copied to clipboard

[WIP] Add support for quantization with bitsandbytes

2

This PR integrates blockwise quantization from [bitsandbytes](https://github.com/facebookresearch/bitsandbytes) as a new compression mechanism of Hivemind. The important part is that it is an *optional* compression protocol: the user should only install...

mryab

[Feature Request] MoE enhancements

1

## The nature of this issue During #470 review there was a list of thing that were not crucial for the PR but ideally they should be done. Find problems...

GreenFatGuy

Add RL examples

4

Current plan: - working with https://stable-baselines3.readthedocs.io - trying to make a minimalistic example that uses hivemind.Optimizer TODO: - [x] make a PPO run with more than 1 peer - [x]...

foksly

Roadmap

4

This is a global project roadmap that states our priorities for the nearest future. These priorities can and should be disputed here or elsewhere, after which we will update the...

justheuristic

Add DDP support to hivemind.optim

1

**Status:** This PR is an early draft intended to validate the design of `hivemind.DDPOptimizer`. I didn't run the code even once yet. **Co-authored-by:** @justheuristic

borzunov

[Feature Request] quality-of-life changes to examples/albert

1

This is a collection of miscellaneous small updates that would make examples/albert more efficient or easier to understand. __Note 1:__ if you're looking for a more advanced example where many...

justheuristic

optimize load_state_from_peers

1

__problem:__ if many peers join at once, they will all pick one averager (latest at the time) as a target for loading initial state. This is causes choke points as...

justheuristic

Retire prefetch_generator

1

We're using this dependency in one spot, where it can be replaced with ~5 lines of native code. Would be great to remove it

justheuristic

It's something we played with a few times but did not end up merging to master. I'm creating this issue so we wouldn't forget it. It would be great if...

justheuristic

Implement CenteredClip in averager

2

- [ ] (important) Reducers should work concurrently - [ ] (important) Test with .isfinite(x), test that no other tensor values may corrupt CClip - [ ] Rename to `MeanReducer`,...

borzunov