defang icon indicating copy to clipboard operation
defang copied to clipboard

AI debugger fails when no logs: not_found: etag not found

Open lionello opened this issue 5 months ago • 1 comments

It failed because the CD task failed to run (exec format error) and didn't actual log anything, so the CLI sent empty logs to Fabric, which was looking for the logs by etag in Playground.

? Would you like to debug the deployment with AI? Yes
 - tracking event "Debug Prompt Accepted": [{etag ohbq7qhurjh5} {loadErr <nil>}]e66ae-2b12-4771-98e6-ab9f47ffcb69","event":"Debug Prompted","properties":{"etag":"ohbq7qhurjh5","failedServices":"[]","reason":"deployment failed: Task d
 - Invoking AI debugger for deployment "ohbq7qhurjh5"er exited with an error."},"os":"darwin","arch":"arm64"}
 - fabricClient j80x60296xqb /io.defang.v1.FabricController/Track {"anon_id":"a2de66ae-2b12-4771-98e6-ab9f47ffcb69","event":"Debug Prompt Accepted","properties":{"etag":"ohbq7qhurjh5","loadErr":"\u003cnil\u003e"},"os":"darwin","arch":"arm64"}
 - Reading .dockerignore file from .dockerignore
 - file limit reached, stopping search
 - Reading .dockerignore file from .dockerignore
 - file limit reached, stopping search
 - Querying logs with filter: 
 (
logName=~"logs/run.googleapis.com%2F(stdout|stderr)$" OR
logName="projects/liotest-443018/logs/cloudbuild" OR
logName="projects/liotest-443018/logs/cos_containers" OR
logName="projects/liotest-443018/logs/docker-logs"
) AND (timestamp >= "2025-07-31T18:16:22.041787Z") AND (
(
  resource.type = "cloud_run_job"
  labels."run.googleapis.com/execution_name" = "defang-cd-t65b2"
) OR (
  resource.type = "cloud_run_job"
  labels."defang-project" = "crew-django-redis-postgres"
  labels."defang-etag" = "ohbq7qhurjh5"
) OR (
  resource.type="cloud_run_revision"
  labels."defang-project" = "crew-django-redis-postgres"
  labels."defang-etag" = "ohbq7qhurjh5"
) OR (
  resource.type="build"
  labels.build_tags =~ "_crew-django-redis-postgres_[a-zA-Z0-9-]{1,63}_ohbq7qhurjh5"
) OR (
  protoPayload.methodName="google.cloud.run.v2.Jobs.UpdateJob" OR "google.cloud.run.v2.Jobs.CreateJob"
  protoPayload.request.job.template.labels."defang-project"="crew-django-redis-postgres"
  protoPayload.request.job.template.labels."defang-etag"="ohbq7qhurjh5"
) OR (
  protoPayload.methodName="/Jobs.RunJob" OR "/Jobs.CreateJob" OR "/Jobs.UpdateJob"
  protoPayload.response.spec.template.metadata.labels."defang-project"="crew-django-redis-postgres"
  protoPayload.response.spec.template.metadata.labels."defang-etag"="ohbq7qhurjh5"
) OR (
  protoPayload.methodName="google.cloud.run.v2.Services.CreateService" OR "google.cloud.run.v2.Services.UpdateService" OR "google.cloud.run.v2.Services.ReplaceService" OR "google.cloud.run.v2.Services.DeleteService"
  protoPayload.request.service.template.labels."defang-project"="crew-django-redis-postgres"
  protoPayload.request.service.template.labels."defang-etag"="ohbq7qhurjh5"
) OR (
  protoPayload.methodName="/Services.CreateService" OR "/Services.UpdateService" OR "/Services.ReplaceService" OR "/Services.DeleteService"
  protoPayload.response.spec.template.metadata.labels."defang-project"="crew-django-redis-postgres"
  protoPayload.response.spec.template.metadata.labels."defang-etag"="ohbq7qhurjh5"
)
)
 - Closing log lister client
 - 
 - fabricClient 4ryh1bfn5wfg /io.defang.v1.FabricController/Debug {"files":[{"name":"compose.yaml","content":"services:\n  postgres:\n    image: pgvector/pgvector:pg16\n    environment:\n      POSTGRES_PASSWORD:\n    x-defang-postgres: true\n    ports:\n      - mode: host\n        target: 5432\n        published: 5432\n\n  redis:\n    image: valkey/valkey:7.2\n    x-defang-redis: true\n    ports:\n      - mode: host\n        target: 6379\n        published: 6379\n\n  app:\n    build:\n      context: ./app\n      dockerfile: Dockerfile\n    command: ./run.sh\n    ports:\n      - \"8000:8000\"\n    depends_on:\n      - postgres\n      - redis\n    environment:\n      DJANGO_SETTINGS_MODULE: config.settings\n      DATABASE_URL: postgres://postgres:${POSTGRES_PASSWORD}@postgres:5432/postgres?sslmode=${SSL_MODE}\n      REDIS_URL: redis://redis:6379/0\n      CELERY_BROKER_URL: redis://redis:6379/0\n      CELERY_RESULT_BACKEND: redis://redis:6379/0\n      DJANGO_SECRET_KEY:\n    healthcheck:\n      test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8000/\"]\n      interval: 30s\…
 ! Failed to debug deployment: not_found: etag not found

To see the logs of the failed service, do:

  defang tail --since=2025-07-31T18:16:22.041787Z --deployment=ohbq7qhurjh5 --type=ALL

Error: deployment failed: Task defang-cd-t65b2-task0 failed with message: The container exited with an error.
deployment failed: Task defang-cd-t65b2-task0 failed with message: The container exited with an error.

lionello avatar Jul 31 '25 18:07 lionello

We should be able to

  • [ ] detect this on the CLI to prevent the error and
  • [ ] have better messaging when this occurs

nullfunc avatar Aug 09 '25 05:08 nullfunc