Priyank Patel
Priyank Patel
Investigating the training slowing down to a crawl as it progresses.
> Nice! Does DistributedDataParallel work also? Not yet, work in progress. A few tricky things and more C++ code than I'd like, but close to something running, just need to...
Issue narrowed down to torch backend not supporting views for out or dest tensors. Working on fix. DDP uses slices of larger tensors to accumulate grads from different layers.
This PR now works in the sense that DDP is training mnist with the correct accuracy, but still very much a WIP. I need to implement some small fixes and...
Shifted my attention some tricky test_ops failures but still have plans for this. Part of these changes are already merged with the detach fix, just want to simplify the rest....
If my bisect is right, some behavior changed in #9845. Loss diverges to nan now, looking into why...
I got this working in ComfyUI (https://github.com/tocubed/ComfyUI-EvTexture). One result: 