Sudhakar Singh

Add multi host pjit tests

7

Added multi host pjit tests to the existing test setup of 2 nodes and 1 process per GPU

[Need more info] `causal_conv1d` doesn't simulate "shifting of tokens by 1"

4

If the local conv (or "causal conv1d") is intended to shift the tokens by 1, then this should instead be `padding=d_conv` instead of `padding=d_conv - 1`, shouldn't it? (Or can...

Expose `rotary_base` as an arg instead of hardcoding

1

# Description Expose `rotary_base` as an arg instead of hardcoding to 10000 Fixes # (issue) https://github.com/NVIDIA/TransformerEngine/issues/849 ## Type of change - [ ] Documentation change (change only to the documentation,...

Fp8 model init factory

4

# Description Trying to bake in `fp8_model_init` into layer initialization. The `fp8_model_init` context manager needs to be then added/managed by the user. Baking it into TE layer initialization would allow...

OpenAI RateLimitError

1

It wasn't possible to run this tutorial https://help.getzep.com/graphiti/graphiti/lang-graph-agent consistently without adding this rate limiting API. ---- > [!IMPORTANT] > Adds rate limiting handling with exponential backoff to `generate_response()` in `openai_client.py`....

Enable SWA with CP for THD input format

11

# Description Sliding Window Attention with CP for THD format is enabled with A2A communication. Fixes # (issue) ## Type of change - [ ] Documentation change (change only to...

feat: Apply RoPE embedding with sequence offsets

4

# What does this PR do ? Allows applying rope embedding with "offsets" to sequences in packed (thd) or batch (bshd/sbhd) formats. The offset per sequence can be supplied as...