oneflow

oneflow copied to clipboard

OFCCL: semi-dynamic and asynchronous collective communication scheduling.

1

WIP, do not merge. If possible, do not waste resources compiling it, I make sure there is no compiling error before commit & push.

Panlichen

marigoold

support flash attention

4

https://github.com/Oneflow-Inc/OneTeam/issues/1733 使用方式： ``` cu_seqlens_q = flow.arange( 0, (batch_size + 1) * seqlen_q, step=seqlen_q, dtype=flow.int32, device="cuda" ) cu_seqlens_v = flow.arange( 0, (batch_size + 1) * seqlen_k, step=seqlen_k, dtype=flow.int32, device="cuda" ) out,...

guo-ran

### op graph init 是 pass 的主要开销 参见：https://github.com/Oneflow-Inc/libai/issues/407#issuecomment-1286776427 其内部各部分开销如下 ``` Maybe OpGraph::Init(const Job& job) { auto cost_ct = std::make_unique(true, true); InitNodes(job); cost_ct->Count("OpGraph0", 1); op_name2op_node_.reserve(job.net().op_size()); ForEachNode([&](OpNode* node) { CHECK(op_name2op_node_.emplace(node->op().op_name(), node).second) Count("OpGraph1",...

strint

flow.Tensor 包数据存在误差

2

## Summary flow.Tensor 包数据存在误差 ## Code to reproduce bug ```python import oneflow as flow flow.manual_seed(987342) for i in range(5): n, in_c, out_c = flow.randint(1, 500, (3,)).tolist() ops = n *...

hhhfccz

[Just for test]Rank task test

2

strint

Add nn.MultiHeadAttention

7

TODO: - [x] 实现 helper functions - [x] 最 naive 的 case：native_multi_head_attention 在 functor 层对齐并实现 - [x] 实现 python functional 层的 mha - [x] 实现 nn.Module 层的 mha - [x]...

marigoold

Let auto parallel give the fastest strategy under the limitation of memory.

Yipeng1994

## Summary - It raises: ``` NotImplementedError: nn.Graph.build()'s input/output item only support types: Tensor/None. ``` - Not helpful. It would be much useful if the type not supported or index...

jackalcooper

`nn.graph` compilation takes too long when it is a large module

13

## Summary - After some modification of model implementation. denoise unet of stable diffusion can be build as a `nn.graph`. - It has around 76000 ops. - Just to build...

jackalcooper