chakra

chakra copied to clipboard

How to i enable logging ( logging.debug ) prints

1

How can i enable logging.debug prints in all the chakra scripts

alokprasad

Progress bar for chakra trace link converter

1

## Problem Related to the Feature >chakra_trace_link while parsing large host and device trace seems hungs as there is no progress bar indicator ## Proposed Solution > A nice progress...

alokprasad

How to view the contents of the .et file directly

2

When I went through the Chakra Tutorial or the Basic Tutorial section, I learned that the .et file contains some node details, because I want to know all the communication...

tninth

NCCL:Broadcast collectives are missing from the converted trace but present in the trace_link

5

## Describe the Bug After running a ResNet50 or TinyLlama2 workload on 4 ranks I see that in the Kineto trace at least one nccl:broadcast collective is observed. In the...

alexseceks

Chakra Module Missing Issue

3

Hello! I encountered an issue while running astra-sim. Specifically, the problem occurred when using the Execution Trace Converter. After executing the following command, the following bug appeared:  It seems...

Swan331

what do I specify for the argument --rank when linking the traces on single node single gpu?

1

I am testing my chakra environment following the steps in wiki. when I link the pytorch trace and kineto trace, with the cmd "chakra_trace_link --chakra-host-trace pytorch_et.json --chakra-device-trace kineto_trace.json --output-file /path/to/chakra_host_device_trace.json...

811454567

chakra_converter PyTorch - conversion failed to due to Cyclic dependecny in DAG

1

## Describe the Bug While converting the ET+ pytroch trace to Protobuf format facing conversion error due Cyclic dependency detected. ## Steps to Reproduce > Run pytorch resnet18 model with...

alokprasad

Do we have to do tracing in a real system when we try to analyze a 10,000 GPUs training system?

3

Accoring to the Astra-sim 2.0 paper, simulates based on Chakra trace, to "decouple parallelization strategies from the ASTRAsim implementation" . Does that mean we have to trace a real 10,000...

basicmi

Lack of GPU comm to compute dependency.

1

In kineto traces we see GPU compute waits for communication to finish.  However, after merging with trace_link there is no dependency between GPU compute and communication, even though HTA...

taifanjum

Chakra's kineto trace add metadata to the NCCL kernels, including message size, PG attributes (name, description, ranks, etc). However in a trace I just recorded some kernels don't include any...

x41lakazam