Add test commands to makefile

[Jepson2k]

What is the use case for the feature? Currently there is no easy way to test locally that mimics the testing setup by the github runners. Ideally there should be a make test-all command that runs the same tests which are run by the github runners. This would allow new developers to easily test their changes without copying the commands one by one from the test-all.yml workflow file.

Does the feature contain any proprietary information about another company's intellectual property? No

How would you implement this feature? Essentially add the same sequence of commands used in the test-all.yml workflow file to the Makefile under a command titled "test-all".

Are there any (reasonable) alternative approaches? Yes

Are you interested in implementing it yourself? Yes

[whyitfor]

When you use build-image.py to create a Docker image, it builds a Makefile that includes a test target for every OFRAK package. Here is an example:

# cat /Makefile
develop:
	$(MAKE) -C ofrak_type develop
	$(MAKE) -C ofrak_io develop
	$(MAKE) -C ofrak_patch_maker develop
	$(MAKE) -C ofrak_core develop
	$(MAKE) -C ofrak_ghidra develop
	$(MAKE) -C frontend develop

.PHONY: test test_ofrak_type test_ofrak_io test_ofrak_patch_maker test_ofrak_core test_ofrak_ghidra test_frontend
test: test_ofrak_type test_ofrak_io test_ofrak_patch_maker test_ofrak_core test_ofrak_ghidra test_frontend
test_ofrak_type:
	$(MAKE) -C ofrak_type test
test_ofrak_io:
	$(MAKE) -C ofrak_io test
test_ofrak_patch_maker:
	$(MAKE) -C ofrak_patch_maker test
test_ofrak_core:
	$(MAKE) -C ofrak_core test
test_ofrak_ghidra:
	$(MAKE) -C ofrak_ghidra test
test_frontend:
	$(MAKE) -C frontend test

Is there a reason you cannot use this Makefile target in your testing?

[rbs-jacob]

It is perhaps also worth recommending that folks who want to run the exact GitHub Actions jobs use the following tool that does just that:

https://github.com/nektos/act

I haven't personally used it myself, but it seems popular and well-supported.

[Jepson2k]

When you use build-image.py to create a Docker image, it builds a Makefile that includes a test target for every OFRAK package. Here is an example:

# cat /Makefile
develop:
	$(MAKE) -C ofrak_type develop
	$(MAKE) -C ofrak_io develop
	$(MAKE) -C ofrak_patch_maker develop
	$(MAKE) -C ofrak_core develop
	$(MAKE) -C ofrak_ghidra develop
	$(MAKE) -C frontend develop

.PHONY: test test_ofrak_type test_ofrak_io test_ofrak_patch_maker test_ofrak_core test_ofrak_ghidra test_frontend
test: test_ofrak_type test_ofrak_io test_ofrak_patch_maker test_ofrak_core test_ofrak_ghidra test_frontend
test_ofrak_type:
	$(MAKE) -C ofrak_type test
test_ofrak_io:
	$(MAKE) -C ofrak_io test
test_ofrak_patch_maker:
	$(MAKE) -C ofrak_patch_maker test
test_ofrak_core:
	$(MAKE) -C ofrak_core test
test_ofrak_ghidra:
	$(MAKE) -C ofrak_ghidra test
test_frontend:
	$(MAKE) -C frontend test

Is there a reason you cannot use this Makefile target in your testing?

True, might be worth just doing this route instead

[Jepson2k]

It is perhaps also worth recommending that folks who want to run the exact GitHub Actions jobs use the following tool that does just that:

https://github.com/nektos/act

I haven't personally used it myself, but it seems popular and well-supported.

This technique falls short if testing on a different architecture such as Arm (or on an airplane which was the case for me)

[ANogin]

Actually, I do not think that github setup is necessarily sufficient. Here is my go-to, which is a lot more thorough (but slower):

#!/bin/bash -e
REV="`git rev-parse --short=8 HEAD`"
for f in ofrak*.yml; do
    if grep -q ofrak_binary_ninja $f; then :
    else
        for target in install develop; do
                if DOCKER_BUILDKIT=1 python3 build_image.py "$@" --config $f --base --finish --target=$target; then :
                else
                        echo "DOCKER_BUILDKIT=1 python3 build_image.py --config $f --base --finish --target=$target FAILED"
                        exit 1
                fi
        done
    fi
done
for f in ofrak*.yml; do
    if grep -q ofrak_binary_ninja $f; then :
    else
        IMAGE_NAME="`grep '^image_name:' $f|sed -e 's|^.*: *"*||' -e 's|"*,* *$||'`"
        (docker kill ${IMAGE_NAME}_${REV} && sleep 3) || true
        docker run -it --name ${IMAGE_NAME}_${REV} \
                -d --rm redballoonsecurity/ofrak/${IMAGE_NAME}:${REV}
    fi
done
sleep 60
for f in ofrak*.yml; do
    if grep -q ofrak_binary_ninja $f; then :
    else
        IMAGE_NAME="`grep '^image_name:' $f|sed -e 's|^.*: *"*||' -e 's|"*,* *$||'`"
        for TEST in "pip3 check" "ofrak license --community --i-agree" "make -k test"; do
                if docker exec -it ${IMAGE_NAME}_${REV} ${TEST}; then :
                else
                        echo "docker exec -it ${IMAGE_NAME}_${REV} ${TEST} FAILED"
                        docker kill ${IMAGE_NAME}_${REV}
                        exit 1
                fi
        done
        docker kill ${IMAGE_NAME}_${REV}
    fi
done