expectedError doesn't read stderr

[vasylmalanka]

During running of a simple test:

schemaVersion: "2.0.0"
commandTests:
  - name: "command not found"
    command: "composer"
    args: ["-v"]
    expectedError: [".*executable file not found.*"]

I am getting:

=================================
====== Test file: php.yaml ======
=================================
=== RUN: Command Test: command not found
--- FAIL
Error: Error creating container: API error (400): OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"composer\": executable file not found in $PATH": unknown


=================================
============ RESULTS ============
=================================
Passes:      0
Failures:    1
Total tests: 1

FAIL
FATA[0001] FAIL   

which is definitely incorrect.

[Aisuko]

Same issue with you, my situation below:

schemaVersion: '2.0.0'
commandTests:
  - name: "test-output"
    # setup: 
    command: "ls /app"
    # args: [""]
    expectedOutput: ["/app"]

17:57:02 › skaffold debug -p test-image
WARN[0000] k8s/*.yaml did not match any file            
Generating tags...
Tags generated in 8.031µs
Starting build...
Build complete in 229.147µs
Starting test...

==========================================
====== Test file: command-test.yaml ======
==========================================
=== RUN: Command Test: test-output
--- FAIL
Error: Error creating container: API error (400): OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"ls /app\": stat ls /app: no such file or directory": unknown


===========================================
================= RESULTS =================
===========================================
Passes:      0
Failures:    1
Total tests: 1

FAIL
FATA[0001] FAIL                                         
Pruning images...
Image prune complete in 18.613µs
FATA[0001] exiting dev mode because first run failed: test failed: running structure tests: running container-structure-test: exit status 1 

18:00:11 › docker run -it --rm aisuko/skaffold:v1 ls /app
/app

Does not read the stdout caused that, I think.

container-structure-test test -v debug --image aisuko/skaffold:v1 --config /Users/bowen/Documents/tektoncd-containers/example/test/command-test.yaml
INFO[0000] container-structure-test &{Version:v1.8.0 GitVersion: BuildDate:2019-02-19T21:40:36Z GoVersion:go1.9.7 Compiler:gc Platform:darwin/amd64}

==========================================
====== Test file: command-test.yaml ======
==========================================
DEBU[0000] Command Test: test-output
INFO[0001] stdout: app
bin
dev
etc
home
lib
media
mnt
proc
root
run
sbin
srv
sys
tmp
usr
var
DEBU[0001] Docker driver does not support teardown commands, since each test gets a new driver. Skipping commands.
DEBU[0001] Skipping empty metadata test
=== RUN: Command Test: test-output
--- FAIL
stdout: app
bin
dev
etc
home
lib
media
mnt
proc
root
run
sbin
srv
sys
tmp
usr
var
Error: Expected string 'stdout: app' not found in output 'app
bin
dev
etc
home
lib
media
mnt
proc
root
run
sbin
srv
sys
tmp
usr
var
'


===========================================
================= RESULTS =================
===========================================
Passes:      0
Failures:    1
Total tests: 1

FAIL
FATA[0001] FAIL

[donmccasland]

Thanks for this bug! I'll be trying to reproduce locally..

[Aisuko]

Feel free ping me, I'd like to contribute to this.❤️

[gtirloni]

@Aisuko the test will check the stderr output of running the command, not from Docker. You could run it like bash -c your_command though.

I think you're better off using a fileExistence test if you want to ensure the command doesn't exist.

[tatitati]

I have the same problem. I tried with bash -c your_command, but no luck for now. Is a basic one:

commandTests:
  - name: java version
    command: bash
    args: ["-c", "java -version"]
    expectedOutput: [".*openjdk.*"]
    exitCode: 0

output:

=====================================================
=== RUN: Command Test: java version
--- FAIL
duration: 488.724752ms
stderr: openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-b08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)

Error: Expected string '.*openjdk.*' not found in output ''