Tests for testing correct implementation?

[madig]

Hi! First, thanks for the guide, I enjoyed it :smiley: I am wondering if there are test-images or something that extensively test if things are implemented correctly? The code I came up with does not match your example code exactly and I'd like to see if would make a difference in a hypothetical "compliance test suite".

[justinmeiners]

Other than the two games we don't have a test suite. It is a good idea though since there are a few instructions and trap calls we do not use.

One thought I have about writing such a suite is that it needs to be careful about what it decides to enforce. My LC-3 does not follow the LC-3 specification. For example trap codes are not implemented in assembly, and so they behave internally differently. One could write a program that depended on this behavior and it would not work.

We want to encourage flexibility of implementation, with a reasonable expectation of program compatibility. With this in mind I think the test suite should just verify the basic operations of the trap codes and instructions.

@rpendleton are you interested in this?

[rpendleton]

I'm interested in this. I don't know exactly how we'd want to implement these tests (comparing output? or writing a test runner in LC-3?) but I'll look into it.

[justinmeiners]

I now have an opinion on implementing tests. The tests should just be an LC-3 program which tries to use each instruction and mode and outputs some results. We can then write a simple shell script to verify output is correct. Of course, failure to pass tests will probably result in things like infinite loops.

Is anyone interested in writing an LC-3 program like this? Perhaps there is one out there?

[justinmeiners]

We want to do this. Here is what I want

• [ ] Create a src/test.asm program in lc3 assembly which tests basic operations of each instruction and variant based on the specification included in the supplies.
• [ ] At each test step make logs to stdout using Puts which demonstrate that the test is working. For example, addition might add 2 numbers, and print the resulting sum.
• [ ] Create a phony makefile target which runs each unix variant of the lc3 program and uses cmp(1) to test that stdout matches a previously verified implementation.
• [ ] Add some comments to the makefile explaining that this test does.
• [ ] Change the end of the tutorial to mention that the test is available.

[fizaan]

"..trap codes are not implemented in assembly, and so they behave internally differently. One could write a program that depended on this behavior and it would not work."

Hmm..this program when run on the official LC-3 simulator will never HALT (unless R7 is saved first then restored). But it does halt when tested in this implementation of the lc3-vm.

.orig x3000

jsr print
halt

print
 ; st r7, temp
 ld r0, char 
 out
 ; ld r7, temp
 ret

char   .stringz "5"
temp   .fill x0000
.end

[rpendleton]

Hmm..this program when run on the official LC-3 simulator will never HALT (unless R7 is saved first then restored). But it does halt when tested in this implementation of the lc3-vm.

@fizaan Good catch. I've created #52 to fix this and will have a PR up soon.

Even though we're implementing the TRAPs directly in the VM (as opposed to relying on the assembly-based implementations), we should still be changing R7 as part of the TRAP instruction since that's part of the specification. The fact that R7 doesn't change in our implementation is a pretty obvious difference in behavior and, as you mentioned, leads to programs that work in the article's implementation of the VM but not the official one. So I think it's important to fix.