zcrypt0

I am also seeing this, but not with every model. I do see it when using the tutorial model as well though.

@mvdan Note my second edit, which should probably be a separate issue. The var `max_path_size` is a package level var declared in another file and reverse (not build) seems to...

@RezaYazdaniAminabadi I was testing with the newly merged code last night but still hit the illegal memory accesses intermittently on the larger batch sizes. It wasn't like throwing a dice...

This might be related to the weird output I see with `bigscience/bloom-350m` because I am using 1080ti in those tests.

@mrwyattii I saw weird output with other models too, including the inference tutorial. I switched over to Volta now and everything is working well.

@RezaYazdaniAminabadi I installed from your PR commit. `pip install git+https://github.com/microsoft/DeepSpeed@73fc0303bf723386df95be0e55259197e540506e` With the `bigscience/bloom-350m` model I don't see any change in the output, it still doesn't make any sense. In fact,...

@RezaYazdaniAminabadi I just saw #2194 and am thinking my issue may be related to that as I ran on 1080tis. I am going to test out this script on a...

I tried using this method on a Z97-WS (Haswell) and successfully exposed the larger memory space at the PCI root, but am running into a snag with the PLX bridge...

@xCuri0 I can get up to 8gb that way, but only when I take out some of the RAM. To enable GPU p2p communication via pcie (thanks to geohotz recent...

I set the driver argument in that case and confirmed it was set with the appropriate commands (I don't have the link readily available, something like `NVReg_EnableResizableBar=1` in a modprobe.d...