Brian Hirsh

Results 146 comments of Brian Hirsh

@eellison do you have any other recommendations? I made a paste of the logs above [here](https://www.internalfb.com/intern/everpaste/?handle=GOIdXBwR8fvelGoDANziK4ut0WdMbsIXAAAz&phabricator_paste_number=1721215485), it looks like cudagraph trees are recording 200+ graphs, even when the user included...

A bit of a shot in the dark, but if you are using DDP with `find_unused_parameters=True`, can you try setting it to false? We saw an example of this causing...

I can repro! Will try to take a look soon when I have some bandwidth

@gcp we should still come up with a real fix, but for now can you try this config workaround? ``` torch._inductor.config.triton.cudagraph_support_input_mutation = False ``` With it, I see the mem...

cc @eellison @BoyuanFeng now that we have a repro and we know it's related to cudagraphs + input mutations, can either of you take a look? Separate note - what...

It would be helpful to get a repro (or at least a full stacktrace of the error). At the very least, the error here is only telling us the metadata...