Error Running Golden Test for remap_acl: ModuleNotFoundError & AttributeError

[iting0321]

Hi, I'm new to contributing to ATS, and while working on #11917 , I encountered some errors and could use some help.

I tried running a golden test with:

autest run -f remap_acl --ats-bin /usr/bin

Here’s what I observed:

• When running the test under ~/trafficserver/tests/gold_tests, I get the following error:

Generating Report: --------------
 Test: remap_acl: Exception
    File: [remap_acl.test.py](http://remap_acl.test.py/)
    Directory: /home/iting0321/trafficserver/tests/gold_tests/remap
     Reason: Traceback (most recent call last):
       File "/home/iting0321/.local/lib/python3.10/site-packages/autest/core/runtesttask.py", line 34, in __call__
         tl = self.__logic.Run(self.__test)
       File "/home/iting0321/.local/lib/python3.10/site-packages/autest/runlogic/runlogic.py", line 18, in Run
         if not tmp.Start(obj):
       File "/home/iting0321/.local/lib/python3.10/site-packages/autest/runlogic/test.py", line 90, in Start
         loadTest(self.__test)
       File "/home/iting0321/.local/lib/python3.10/site-packages/autest/core/test.py", line 292, in loadTest
         execFile(fileName, locals, locals)
       File "/home/iting0321/.local/lib/python3.10/site-packages/autest/common/execfile.py", line 16, in execFile
         exec(safeCompile(f.read(), fname), globals, locals)
       File "/home/iting0321/trafficserver/tests/gold_tests/remap/[remap_acl.test.py](http://remap_acl.test.py/)", line 29, in <module>
         from deactivate_ip_allow import all_deactivate_ip_allow_tests
     ModuleNotFoundError: No module named 'deactivate_ip_allow'

Total of 1 test
  Unknown: 0
  Exception: 1
  Failed: 0
  Warning: 0
  Skipped: 0
  Passed: 0

• When running the test under ~/trafficserver/tests/gold_tests/remap, I get the following error:

error: AttributeError: 'TestRun' object has no attribute 'MakeATSProcess'

Any help or guidance would be greatly appreciated. Thanks!

[masaori335]

Hello. Thanks for taking a look at the issue!

You need to set the path to the module. Would you try autest.sh instead of autest command directly? It's a wrapper of autest command and has a trick to set the path to the module.

https://github.com/apache/trafficserver/blob/ca56c225fac7539de963296d7e11d4ee895dc16e/tests/autest.sh#L33

$ cd tests
$ ./autest.sh -f remap_acl --ats-bin /usr/bin

[iting0321]

I was able to successfully run the test, but I’m still encountering an issue that I haven't been able to resolve. Could there be any additional configuration that needs to be set?

The test fails because the watcher process finishes before it is ready, causing the test to repeatedly fail:

Test: remap_acl: Failed
File: remap_acl.test.py
    Directory: /home/iting0321/trafficserver/tests/gold_tests/remap
   Starting Test remap_acl : No issues found - Passed
      Reason: Started!
   Run: Verify allow is reject in modern policy: Failed
     Starting TestRun 0-tr : Waiting for Process "watcher" to become ready - Failed
        Reason: Process finished before it was ready
     Process: ts-old-action-0: Failed
# ...
Test : Checking that any test passes - Failed
          Reason: None of the tests passed
   
   Test : Checking that ReturnCode == 33 - Failed
            Reason: Returned Value 0 != 33
         
   Test : Checking that ReturnCode == 70 - Failed
            Reason: Returned Value 0 != 70
         

Additionally, I noticed that the Jenkins CI badge: ubuntu22.04 indicates failures on Ubuntu 22.04. Could this be related to my issue? If so, would upgrading my OS to 23.04 be necessary to run ATS properly?

[masaori335]

It looks like making the watcher process is failed. https://github.com/apache/trafficserver/blob/9a1ef119f3b7a017583c9aa5d088b2437101b92a/tests/gold_tests/remap/remap_acl.test.py#L183-L190

Is there any error logs in your _sandbox? The watcher process outputs should be in the _sandbox/remap_acl/_output/0-tr-watcher/ dir and ATS logs are in _sandbox/remap_acl/ts-old-action-0/log/ like below.

$ ls _sandbox/remap_acl/_output/0-tr-watcher/
command.txt  stream.all.txt    stream.error.txt   stream.stdout.txt   stream.warning.txt
replay.sh    stream.debug.txt  stream.stderr.txt  stream.verbose.txt  

$ ls _sandbox/remap_acl/ts-old-action-0/log/
diags.log  traffic.out

The ubuntu22.04 CI job is failed by a flaky unit test. It intermittently happens and the job is back now. It's not relates to this issue.

[iting0321]

I ran the gold test under tests/gold_tests again using the same command, but it still failed, and there is nothing under tests/gold_tests/_sandbox/remap_acl.

I also tried running the regression test (sudo bin/traffic_server -R 1 under /opt/ts) multiple times. However, I only got the logs you mentioned under tests/_sandbox and tests/gold_tests/_sandbox/remap_acl the first time. After that, nothing showed up.

Are there any additional environment variables I need to export before running the test?

[masaori335]

Regression Test is completely different thing from AuTest. The tests/gold_tests/_sandbox/remap_acl must be made by AuTest you ran before. Please check logs in the directory.

[JosiahWI]

@iting0321 You can pass a --sandbox <path> option to the autesh.sh command to specify a test output directory. I think the path @masaori335 gave you is the default path if you do not specify one with that option. Every time you run the AuTest, it will overwrite any previous logs in the sandbox for each test that it runs.