nimlgen

`sudo rmmod amdgpu` run with: `sudo -E DEBUG=2 AM=1 python3 test/test_ops.py TestOps.test_tiny_add`

implemented in kfd

@rggs what commit/version you getting this on?

You can try running with DEBUG=2 to see if there is a reproducible kernel that causes the issue.

@dirkcgrunwald Can you please post your dmesg right after the crash? The gpu can't execute init commands which looks strange.

Cool! Got a bit different output on comma, but something changing in the buffers :) ``` ****** create eye Tensor and copy it to the GPU ****** ****** initial values...

@vpachkov, you're welcome to join our Monday meetings (9:40 PT) on Discord to discuss this pr and hcq-related things.

@sixEyees, closing this issue as duplicate of #6089. If you’re experiencing the same problem, please continue the discussion there.

Kernel produces different output. AQL works, so something related to PM4 setup. Looks like something related to flat mem. ``` ast = LazyOp(op=BufferOps.STORE, src=(LazyOp(op=ReduceOps.MAX, src=(LazyOp(op=BinaryOps.ADD, src=(LazyOp(op=BinaryOps.MUL, src=(LazyOp(op=UnaryOps.CAST, src=(LazyOp(op=BinaryOps.MUL, src=(LazyOp(op=BinaryOps.ADD, src=(LazyOp(op=UnaryOps.CAST,...

Let's hold this, since it might be backend related. Just noticed that with `UOPS_REWRITE=0` loss is a bit different from run to run. Some kernel with same input gives slightly...