garden

garden copied to clipboard

https://kubernetes.slack.com/archives/CKM7CP8P9/p1657812033956049

Bug

Hey all. I have a weird problem using gardens artifacts. Tests fail which they dont if I comment out the artifacts. In addition, I think the testing container never gets shutdown and the garden process hangs indefinitely.

The artifact file is around 600MB. I can find the file inside the .garden/artifacts directory. It seems to be completely downloaded.

Expected behavior

test task should not fail and garden successfully finish its job.

Reproducible example

Not really, this fails as soon as I add the artifacts to the yml file. Without artifacts everything is working fine.

Your environment

Happens on macOS as well as Linux with latest garden version.

garden output

   Status of Pod test-backend-core-image-test-be-integration-f6d2c3 is "ready"
   ℹ Resources ready
   Execing command in dev-magnus/Pod/test-backend-core-image-test-be-integration-f6d2c3/main: sh -c tar --help
   Execing command in dev-magnus/Pod/test-backend-core-image-test-be-integration-f6d2c3/main: sh -c exec >/tmp/output; "/bin/bash" "-c" "cargo nextest run -P ci -j 32 && grcov . -s . --binary-path ./target/debug/ -t cobertura --branch --ignore-not-existing --ignore \"$HOME/.cargo/*\" -o ./target/debug/coverage-cobertura.xml && sleep 1\n"
   Execing command in dev-magnus/Pod/test-backend-core-image-test-be-unit-69eb33/main: sh -c 
rm -rf /tmp/.garden-artifacts-6afappns >/dev/null || true
mkdir -p /tmp/.garden-artifacts-6afappns
cd /tmp/.garden-artifacts-6afappns
touch .garden-placeholder
mkdir -p coverage/unit/
cp -r /app/target/debug/coverage-cobertura.xml coverage/unit/ >/dev/null || true
tar -c -z -f - . | cat
rm -rf /tmp/.garden-artifacts-6afappns >/dev/null || true
    
   tar: got entry ./
   tar: got entry ./.garden-placeholder
   tar: got entry ./coverage/
   tar: got entry ./coverage/unit/
   tar: got entry ./coverage/unit/coverage-cobertura.xml
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(Use `garden --trace-warnings ...` to show where the warning was created)
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
(node:84278) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [ReadEntry]. Use emitter.setMaxListeners() to increase limit
   Updating annotations and labels on namespace garden-system
   → Artifact: .garden/artifacts/coverage/unit/coverage-cobertura.xml
✖ backend-core-image-test   → Running be-unit tests → Failed! (took 296.8 sec)

Failed running be-unit tests in module backend-core-image-test. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Command exited with code 1:
    Finished test [unoptimized + debuginfo] target(s) in 1.76s
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error: Command exited with code 1:
    Finished test [unoptimized + debuginfo] target(s) in 1.76s

    at wrapped.process (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/build/src/tasks/test.js:0)
    at TaskNode.process (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/build/src/task-graph.js:0)
    at wrapped.processNode (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/build/src/task-graph.js:0)
✖ tasks                     → Failed task test.backend-core-image-test.be-unit.044432c0-0387-11ed-8d64-15d9c591caf5

error.log

[2022-07-14T15:15:09.795Z] Error: Command exited with code 1:
    Finished test [unoptimized + debuginfo] target(s) in 1.76s

    at wrapped.process (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/build/src/tasks/test.js:0)
    at TaskNode.process (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/build/src/task-graph.js:0)
    at wrapped.processNode (/snapshot/project/tmp/pkg/cli/node_modules/@garden-io/core/build/src/task-graph.js:0)

Thanks for reporting this @magnusja! It would be really nice if you can share a reproducible example. You can consider reducing the size of the artifacts. Please let us know if that fixes the issue.

If you need an artifact this big, probably you need to use a different way of storing it. For example, S3.

Hey @vvagaytsev

unfortunately I am unable to provide an example. I tried to adapt the demo project, downloaded an 1.1GB file, specified it as test artifact. Works flawlessly!! Not sure why I have this problem within my project.

With latest garden version I am getting this error

Failed running be-unit tests in module backend-core-image-test. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Test 'be-unit' in container module 'backend-core-image-test' specifies artifacts to export, but the image doesn't contain the tar binary. In order to copy 
artifacts out of Kubernetes containers, both sh and tar need to be installed in the image.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

I sshed into pod during test, tar is definitely installed.

If using log level silly I am getting this.

   Execing command in dev-magnus/Pod/test-backend-core-image-test-be-integration-cd12cc/main: sh -c exec &>/tmp/output; "/bin/bash" "-c" "cargo nextest run -P ci -j 32 && grcov . -s . --binary-path ./target/debug/ -t cobertura --branch --ignore-not-existing --ignore $HOME/.cargo/* -o ./target/debug/coverage-cobertura.xml\n"
   Execing command in dev-magnus/Pod/test-backend-core-image-test-be-integration-cd12cc/main: sh -c 
rm -rf /tmp/.garden-artifacts-ety3u9dl >/dev/null || true
mkdir -p /tmp/.garden-artifacts-ety3u9dl
cd /tmp/.garden-artifacts-ety3u9dl
touch .garden-placeholder
mkdir -p coverage/integration/
cp -r /app/target/debug/coverage-cobertura.xml coverage/integration/ >/dev/null || true
tar -c -z -f - . | cat
rm -rf /tmp/.garden-artifacts-ety3u9dl >/dev/null || true
    
✖ backend-core-image-test   → Running be-integration tests

Failed running be-integration tests in module backend-core-image-test. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TAR_BAD_ARCHIVE: Unrecognized archive format
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error: TAR_BAD_ARCHIVE: Unrecognized archive format
    at Unpack.warn (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/warn-mixin.js:19:40)
    at Unpack.warn (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/unpack.js:220:18)
    at Unpack.<anonymous> (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/parse.js:81:14)
    at Unpack.emit (events.js:412:35)
    at Unpack.emit (domain.js:475:12)
    at Unpack.[emit] (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/parse.js:285:12)
    at Unpack.[maybeEnd] (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/parse.js:401:17)
    at Unpack.[consumeChunk] (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/parse.js:409:21)
    at Unpack.write (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/parse.js:364:25)
    at Unpack.end (/snapshot/project/tmp/pkg/cli/node_modules/tar/lib/parse.js:477:14)
    at Function.handleStandardStreams (/snapshot/project/tmp/pkg/cli/node_modules/@kubernetes/client-node/dist/web-socket-handler.js:26:24)
    at /snapshot/project/tmp/pkg/cli/node_modules/@kubernetes/client-node/dist/exec.js:38:70
    at WebSocket.<anonymous> (/snapshot/project/tmp/pkg/cli/node_modules/@kubernetes/client-node/dist/web-socket-handler.js:146:47)
    at WebSocket.onMessage (/snapshot/project/tmp/pkg/cli/node_modules/ws/lib/event-target.js:132:16)
    at WebSocket.emit (events.js:400:28)
    at WebSocket.emit (domain.js:475:12)
    at Receiver.receiverOnMessage (/snapshot/project/tmp/pkg/cli/node_modules/ws/lib/websocket.js:1059:20)
    at Receiver.emit (events.js:400:28)
    at Receiver.emit (domain.js:475:12)
    at Receiver.dataMessage (/snapshot/project/tmp/pkg/cli/node_modules/ws/lib/receiver.js:502:14)
    at Receiver.getData (/snapshot/project/tmp/pkg/cli/node_modules/ws/lib/receiver.js:435:17)
    at Receiver.startLoop (/snapshot/project/tmp/pkg/cli/node_modules/ws/lib/receiver.js:143:22)

Error Details:

recoverable: false
cwd: /tmp/tmp-231385-gbFO82x2T8Hc
code: TAR_BAD_ARCHIVE
tarCode: TAR_BAD_ARCHIVE

✖ tasks                     → Failed task test.backend-core-image-test.be-integration.6917c310-0cf1-11ed-8f88-c3c2fdcdfbc4
ℹ Remaining tasks 1

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

This issue has been automatically marked as stale because it hasn't had any activity in 90 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to tag a maintainer and ask them to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!

Closing this as stale. If you still have this issue, feel free to reach us on Discord.

Are you using discord now as an issue tracker? Oo

@magnusja sorry for the confusion, we are tracking issues on github of course. So far we failed to reproduce this issue.

Do you still experience this issue with the latest version of Garden (0.13)?

If yes, please share more information with us.

Some of these could be helpful to reproduce this issue on our side:

• silly loglevel logs (-l5 option)
• the exact Garden command you are running
• The zip file from garden get debug-info (You can send it to me also privately via email or discord, if you like.)

Our community engineers and the engineering team are also happy to help troubleshoot this issue on Discord.

Thank you so much for taking the time to report errors like these. I am really sorry that we were unable to help you so far, but I hope we can still figure this out.