mlcommons
About chakra_trace_link
Hi, I am following your tutorial in wiki for Chakra ET generation with PyTorch ET and Kineto wiki . But when i try to merge the two trace from matmul, it raises the following error:
root@x08j03287:/workspace/astra-sim/extern/graph_frontend/chakra/example# chakra_trace_link --chakra-host-trace pytorch_et.json --chakra-device-trace kineto_trace.json --rank 0 --output-file chakra_host_device_trace.et [2024-12-10 06:05:56,631] trace.py:328 [INFO]: /workspace/astra-sim/extern/graph_frontend/chakra/example [2024-12-10 06:05:56,632] trace_file.py:56 [ERROR]: File /workspace/astra-sim/extern/graph_frontend/chakra/example/kineto_trace.json and file /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json has the same rank. Will use /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json as the path to rank: 0. [2024-12-10 06:05:56,632] trace_file.py:94 [INFO]: Rank to trace file map: {0: '/workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json'} [2024-12-10 06:05:56,632] trace.py:474 [INFO]: ranks=[0] [2024-12-10 06:05:56,632] trace_parser.py:107 [WARNING]: Parsed /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json time = 0.00 seconds [2024-12-10 06:05:56,633] trace_parser.py:430 [WARNING]: Parsed /workspace/astra-sim/extern/graph_frontend/chakra/example/pytorch_et.json backend=json in 0.00 seconds; current PID:58601 Traceback (most recent call last): File "/usr/local/bin/chakra_trace_link", line 8, in
sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/chakra/src/trace_link/trace_link.py", line 47, in main linker.link(args.rank, args.chakra_host_trace, args.chakra_device_trace, args.output_file) File "/usr/local/lib/python3.10/dist-packages/chakra/src/trace_link/trace_linker.py", line 70, in link sync_deps = self.load_sync_dependencies(rank, chakra_device_trace) File "/usr/local/lib/python3.10/dist-packages/chakra/src/trace_link/trace_linker.py", line 120, in load_sync_dependencies trace_analysis = TraceAnalysis(trace_dir=trace_dir) File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/trace_analysis.py", line 37, in init self.t.load_traces(include_last_profiler_step) File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 352, in load_traces self.parse_traces() File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 477, in parse_traces self.parse_multiple_ranks(ranks, use_multiprocessing and len(ranks) > 1) File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 404, in parse_multiple_ranks result = parse_trace_file(self.trace_files[rank]) File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 228, in parse_trace_file add_fwd_bwd_links(df) File "/workspace/astra-sim/extern/graph_frontend/chakra/HolisticTraceAnalysis/hta/common/trace.py", line 244, in add_fwd_bwd_links if df.cat.eq("fwdbwd").sum() == 0: File "/usr/local/lib/python3.10/dist-packages/pandas/core/generic.py", line 6299, in getattr return object.getattribute(self, name) AttributeError: 'DataFrame' object has no attribute 'cat'. Did you mean: 'at'?
How to solve this problem?
