a bunch of hangs and crashes

[camel-cdr]

I noticed that a bunch of the programs (benchmarks) I want to execute on ara start hanging at some point, but since that isn't that reproducible, I've tried to find some short examples that are definitely broken.

My approach to finding the bugs below was to shuffle a list of all instructions, see if it breaks, and try to minimize the code while keeping the breakage. This sound time-consuming, but it wasn't because it almost always started hanging with just a few instructions.

I'm using the llvm toolchain, added the implemented foo in a new file asm.S and modified the hello_world main to the following:

int main(void) {
        extern void foo(void);
        printf("BEGIN\n");
        foo();
        printf("END\n");
        return 0;
}

Hangs

.text
.global foo
foo:
    vsetvli t0, x0, e32, m1, ta, ma
    vfirst.m t0,v8,v0.t
    ret

foo:
        vsetvli t0, x0, e32, m1, ta, ma
        vmseq.vi v8,v16,13
        vcpop.m t0,v8,v0.t
        ret

foo:
        vsetvli t0, x0, e32, m1, ta, ma
        vredmax.vs v8,v16,v24,v0.t
        vslidedown.vx v8,v16,t0,v0.t
        vmv.v.v v8,v16
        ret

foo:
        vsetvli t0, x0, e32, m1, ta, ma
        vredor.vs v8,v16,v24
        vslidedown.vi v8,v16,3
        vmxnor.mm v0,v8,v16
        ret

foo:
        vsetvli t0, x0, e32, m1, ta, ma
        vid.v v0
        vid.v v8
        vid.v v16
        vid.v v24
        vadd.vv v0, v8, v16 # these can be almost any two instructions
        vsub.vv v0, v8, v16 # and it will also hang, e.g: vsub&vmul,vmseq&vxor, ...
        ret

foo:
        vsetvli t0, x0, e32, m1, ta, ma
        vrsub.vx v8,v16,t0
        vmsbf.m v8,v16
        vsrl.vi v8,v16,13,v0.t
        vslideup.vi v8,v16,3,v0.t
        vsbc.vvm v8,v16,v24,v0
        ret

Crashes

foo:
        vsetvli t0, x0, e16, m8, ta, ma
        vid.v v0
        ret

Full Reproduction

FROM ubuntu:22.04

RUN apt-get update \
	&& apt-get install -y build-essential wget git unzip python3 verilator make cmake zstd python3 ninja-build texi2html \
	texinfo device-tree-compiler autoconf autotools-dev curl lsb-release clang flex libfl-dev libelf-dev bison libmpc-dev htop  \
	help2man vim \
	&& apt-get clean \
	&& rm -rf /var/lib/apt/lists/*

RUN git clone https://github.com/pulp-platform/ara/

WORKDIR ara

COPY main.c apps/vfmv_test/main.c

RUN git submodule update --init --recursive

RUN make toolchain-llvm
RUN make riscv-isa-sim
RUN make verilator

RUN /usr/bin/install -c /ara/install/verilator/bin/verilator_bin /ara/install/verilator/share/verilator/

RUN make -C hardware apply-patches && make -C hardware verilate

$ # modify apps/hello_world/main.c and  apps/hello_world/asm.S
$ make bin/hello_world && app=hello_world make -C ../hardware/ simv

Note: I haven't looked at floating point, widening and narrowing instructions yet.

I also ran the built-in tests and rv64uv-ara-vl_nocheck hangs, and rv64uv-ara-vssub fails.

PS: Is there a kinda stable version of ara with rvv 1.0 or 0.7.1 support, I can run my benchmarks on for now?

[mp-17]

Hi @camel-cdr, thanks a lot for reporting. We really need to integrate random sequence generation tests in our verification flow since, thus far, we only used hand-assembled tests and vectorized benchmarks that do not expose more exotic patterns. The repo has been silent for a while because of internal exploration and evaluation, I will fix this asap (I think I will be able to dedicate regular time in two weeks) :-)

[quswarabid]

Hey @mp-17 I think this might be a good opportunity to explore few options regarding test suites/generator already available.

I have made a riscof plugin for ARA and got it up and running as well. The plugin can help use tests from a test generator rvvatg and is based on signature comparison. You can take a look at my local dev branch and meanwhile I will try to cleanup the code and make a PR soon.

It's really good work that can we use from Shenwei Hu from RIOS Lab. Tests generated from his generator can really cover a lot of ground on vtype settings and different VLENs as well.

Hope to get in touch with you on this, very soon.

[mp-17]

Hello @quswarabid, thank you very much; that would be fantastic! If you want and have time, we can schedule a meeting and discuss it.

[quswarabid]

Yes, sure. I'll drop you a mail soon to schedule a meetup. Looking forward to it!

Meanwhile you have a great weekend @mp-17 😊

[mp-17]

Thank you, @quswarabid, waiting for it! Have a great weekend too! :-)

[mp-17]

Hey @camel-cdr,

I am now dedicating some time every week to verification and bug fixes. This branch should solve all the hangs/crashes you posted: https://github.com/pulp-platform/ara/tree/mperotti/debug

I want to start solving all the bugs highlighted in the repo Issues. If you like, we can also have a call and discuss the problems you encountered. I see you have a repo with benchmarks https://github.com/camel-cdr/rvv-bench; I have not tried them yet, but I guess you have met problems running them. Let me know!

EDIT: I have also added a script to generate the instruction sequences, as you suggested. I copied your instruction list for now, let me know if the credit in the header is enough for you, otherwise I can change it as you like).

[camel-cdr]

Hi @mp-17, thats great to hear. Yes, I run into problems trying my benchmark, and created the small script so I can share reproducible bugs. (The credit in the header is enough :+1:)

I haven't tested on ara for some time now, just followed the commits, but I'll look into it again after I'm back from the rv summit.