yuleiqin

Results 7 comments of yuleiqin

[Rank 7] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=20879, OpType=ALLREDUCE, NumelIn=406069248, NumelOut=406069248, Timeout(ms)=1800000) ran for 1800115 milliseconds before timing out.

Any new updates? I am facing the same problem; The first 40 steps are fine and then collapse (notice no data exchange occured via RDMA starting at the step 40;...

Turn on USE_MESA: True Execution command fails

> hi, when I run the program headless rendering with a CentOS7 station (via ssh), I got the error: (I already followed 1. conda install -c conda-forge libstdcxx-ng=12 2. export...

[Informative] Having trouble with OpenGL in miniconda environment (Ubuntu 22)

I run all the instructions above but still get errors with my CentOS7 system (headless rendering): Can anyone helps please? Thank you in advance. ``` File "AnimatedDrawings/animated_drawings/view/shaders/shader.py", line 29, in...

Help on MultiGPU setup

I am experiencing the same error +1

Help on MultiGPU setup

> @guijinSON @yuleiqin what model are you experiencing this with? I seem to have trouble replicating this using > > ``` > accelerate launch -m lm_eval --model hf --tasks lambada_openai...

spacy.strings.StringStore size changed, may indicate binary incompatibility

> I was using spacy 3.6.0. Problem solved downgrading it. Try: > > ``` > pip install spacy==2.3.9 > ``` thanks, you save my day

Active Intent Labeling Issues

Yes, I am also very curious about the definition of the active intent. I am also wondering if we could do to extract true annotations from these active intents of...