dbt-labs
[CT-106] [Bug] Docker image build process is broken
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
On building I get:
ERROR: Could not find a version that satisfies the requirement dbt-postgres (unavailable) (from versions: 0.13.0a1, 0.13.0a2, 0.13.0rc1, 0.13.0, 0.13.1a1, 0.13.1a2, 0.13.1, 0.14.0a1, 0.14.0a2, 0.14.0rc1, 0.14.0, 0.14.1a1, 0.14.1rc1, 0.14.1rc2, 0.14.1, 0.14.2, 0.14.3rc1, 0.14.3, 0.14.4, 0.15.0b1, 0.15.0b2, 0.15.0b3, 0.15.0rc1, 0.15.0rc2, 0.15.0, 0.15.1rc1, 0.15.1rc2, 0.15.1, 0.15.2, 0.15.3rc1, 0.15.3, 0.16.0b1, 0.16.0b2, 0.16.0b3, 0.16.0rc1, 0.16.0rc2, 0.16.0rc3, 0.16.0rc4, 0.16.0, 0.16.1rc1, 0.16.1, 0.17.0b1, 0.17.0b2, 0.17.0rc1, 0.17.0rc2, 0.17.0rc3, 0.17.0rc4, 0.17.0, 0.17.1rc1, 0.17.1rc2, 0.17.1rc3, 0.17.1rc4, 0.17.1, 0.17.2b1, 0.17.2rc1, 0.17.2, 0.18.0b1, 0.18.0b2, 0.18.0rc1, 0.18.0rc2, 0.18.0, 0.18.1b1, 0.18.1b2, 0.18.1b3, 0.18.1rc1, 0.18.1, 0.18.2rc1, 0.18.2, 0.19.0b1, 0.19.0rc1, 0.19.0rc2, 0.19.0rc3, 0.19.0, 0.19.1b2, 0.19.1rc1, 0.19.1rc2, 0.19.1, 0.19.2rc1, 0.19.2rc2, 0.19.2, 0.20.0b1, 0.20.0rc1, 0.20.0rc2, 0.20.0, 0.20.1rc1, 0.20.1, 0.20.2rc1, 0.20.2rc2, 0.20.2, 0.21.0b1, 0.21.0b2, 0.21.0rc1, 0.21.0rc2, 0.21.0, 0.21.1rc1, 0.21.1rc2, 0.21.1, 1.0.0b1, 1.0.0b2, 1.0.0rc1, 1.0.0rc2, 1.0.0rc3, 1.0.0, 1.0.1rc1, 1.0.1)
ERROR: No matching distribution found for dbt-postgres (unavailable)
Expected Behavior
I'd expect to have docker image
Steps To Reproduce
- Clone fresh git repo
- docker build -t tmp -f docker/Dockerfile
Relevant log output
[3/9] STEP 2/2: RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres"
Collecting dbt-postgres
Cloning https://github.com/dbt-labs/ to /tmp/pip-install-epfngjko/dbt-postgres_8d7eb22fbdbe4b06bf3a20348f72b0c7
Running command git clone --filter=blob:none -q https://github.com/dbt-labs/ /tmp/pip-install-epfngjko/dbt-postgres_8d7eb22fbdbe4b06bf3a20348f72b0c7
remote: Not Found
fatal: repository 'https://github.com/dbt-labs/' not found
WARNING: Discarding git+https://github.com/dbt-labs/#egg=dbt-postgres&subdirectory=plugins/postgres. Command errored out with exit status 128: git clone --filter=blob:none -q https://github.com/dbt-labs/ /tmp/pip-install-epfngjko/dbt-postgres_8d7eb22fbdbe4b06bf3a20348f72b0c7 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement dbt-postgres (unavailable) (from versions: 0.13.0a1, 0.13.0a2, 0.13.0rc1, 0.13.0, 0.13.1a1, 0.13.1a2, 0.13.1, 0.14.0a1, 0.14.0a2, 0.14.0rc1, 0.14.0, 0.14.1a1, 0.14.1rc1, 0.14.1rc2, 0.14.1, 0.14.2, 0.14.3rc1, 0.14.3, 0.14.4, 0.15.0b1, 0.15.0b2, 0.15.0b3, 0.15.0rc1, 0.15.0rc2, 0.15.0, 0.15.1rc1, 0.15.1rc2, 0.15.1, 0.15.2, 0.15.3rc1, 0.15.3, 0.16.0b1, 0.16.0b2, 0.16.0b3, 0.16.0rc1, 0.16.0rc2, 0.16.0rc3, 0.16.0rc4, 0.16.0, 0.16.1rc1, 0.16.1, 0.17.0b1, 0.17.0b2, 0.17.0rc1, 0.17.0rc2, 0.17.0rc3, 0.17.0rc4, 0.17.0, 0.17.1rc1, 0.17.1rc2, 0.17.1rc3, 0.17.1rc4, 0.17.1, 0.17.2b1, 0.17.2rc1, 0.17.2, 0.18.0b1, 0.18.0b2, 0.18.0rc1, 0.18.0rc2, 0.18.0, 0.18.1b1, 0.18.1b2, 0.18.1b3, 0.18.1rc1, 0.18.1, 0.18.2rc1, 0.18.2, 0.19.0b1, 0.19.0rc1, 0.19.0rc2, 0.19.0rc3, 0.19.0, 0.19.1b2, 0.19.1rc1, 0.19.1rc2, 0.19.1, 0.19.2rc1, 0.19.2rc2, 0.19.2, 0.20.0b1, 0.20.0rc1, 0.20.0rc2, 0.20.0, 0.20.1rc1, 0.20.1, 0.20.2rc1, 0.20.2rc2, 0.20.2, 0.21.0b1, 0.21.0b2, 0.21.0rc1, 0.21.0rc2, 0.21.0, 0.21.1rc1, 0.21.1rc2, 0.21.1, 1.0.0b1, 1.0.0b2, 1.0.0rc1, 1.0.0rc2, 1.0.0rc3, 1.0.0, 1.0.1rc1, 1.0.1)
ERROR: No matching distribution found for dbt-postgres (unavailable)
error building at STEP "RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres"": error while running runtime: exit status 1
### Environment
```markdown
- OS: linux
- Python: -
- dbt: main
What database are you using dbt with?
No response
Additional Context
No response
Hi @korjavin , sorry you're having some issues with the docker image.
Here's a few things that might help sort this out.
-
The command provided
docker build -t tmp -f docker/Dockerfileisn't a valid docker command because it lacks a path. If you want to build the full image you're probably looking for something like this:docker build -t <tag_name> path/to/folder/that/contains/the/Dockerfile. The-fflag is meant to be used when you want to break the Dockerfiles-must-be-named-Dockerfilenaming convention. -
While I wasn't able to reproduce your error, the log message makes it look as if the build arg
dbt_postgres_refhas been somehow unset or never set in the first place. That's pretty odd given that it's hard coded directly into the dockerfile. Is there by chance some way that could have been broken in your environment?
Hi @iknox-fa, I experienced the same issue as described above and it seems logical to me.
As far as I am aware, ARGs are only persisted within the scope of the preceding FROM statement, unless an ARG is referenced in a FROM line (i.e. version number). See also this.
https://github.com/dbt-labs/dbt-core/blob/aafa1c7f473988bb1771b7cf5b120efab44a8144/docker/Dockerfile#L59
From this line onwards, all ARGs declared above are out of scope again.
This works for me:
##
# dbt-core
##
FROM base as dbt-core
ARG [email protected]
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core"
##
# dbt-snowflake
##
FROM base as dbt-snowflake
ARG [email protected]
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_snowflake_ref}#egg=dbt-snowflake"```
Hi @thenaturalist -- Can you provide the exact docker build command you're running? I still can't reproduce this error. As far as I can tell, args are passed from stage to stage as long as you inherit one stage from the previous (as we do, everything inherits from base).
As an example, here's my laptop building the dbt-all target. You can see in the later steps where the various git refs are populated by the args:
⚙ Projects/dbt-core docker_release_CT-3 docker build -t foobar --target dbt-all docker
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 3.60kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.9.9-slim-bullseye 0.8s
=> [base 1/4] FROM docker.io/library/python:3.9.9-slim-bullseye@sha256:f4efbe5d1eb52c221fded79ddf18e4baa0606e7766afe2f07b0b330a9e79564a 4.8s
=> => resolve docker.io/library/python:3.9.9-slim-bullseye@sha256:f4efbe5d1eb52c221fded79ddf18e4baa0606e7766afe2f07b0b330a9e79564a 0.0s
=> => sha256:f4efbe5d1eb52c221fded79ddf18e4baa0606e7766afe2f07b0b330a9e79564a 1.86kB / 1.86kB 0.0s
=> => sha256:615ceb9962b182726198d4e71c1b8ecfa4904f83957fb8f4742e573e18e990f7 1.37kB / 1.37kB 0.0s
=> => sha256:8ace3a02b8421dd3afe87c552673fb21ebba878c29977856b27a9e68714e16bb 7.88kB / 7.88kB 0.0s
=> => sha256:a2abf6c4d29d43a4bf9fbb769f524d0fb36a2edab49819c1bf3e76f409f953ea 31.36MB / 31.36MB 1.3s
=> => sha256:27003db43ed41a599ee4f212684be296e644c6a82868f0f34dc9bdc39dc93348 1.08MB / 1.08MB 0.1s
=> => sha256:eba080ca5a70cc6fa2bcbab4c3ba67efb762171f7e63ea47c9e73c23e1b0d14d 11.01MB / 11.01MB 0.7s
=> => sha256:00ceeca3d803e82aef2f42658d514b7f567a9aac28a45238a3e8891b6c372b1b 235B / 235B 0.3s
=> => sha256:8980b900ecd203d1d844aea46144bda4a95ef3ebf22245279e14abdfa710840d 2.64MB / 2.64MB 0.8s
=> => extracting sha256:a2abf6c4d29d43a4bf9fbb769f524d0fb36a2edab49819c1bf3e76f409f953ea 1.9s
=> => extracting sha256:27003db43ed41a599ee4f212684be296e644c6a82868f0f34dc9bdc39dc93348 0.2s
=> => extracting sha256:eba080ca5a70cc6fa2bcbab4c3ba67efb762171f7e63ea47c9e73c23e1b0d14d 0.6s
=> => extracting sha256:00ceeca3d803e82aef2f42658d514b7f567a9aac28a45238a3e8891b6c372b1b 0.0s
=> => extracting sha256:8980b900ecd203d1d844aea46144bda4a95ef3ebf22245279e14abdfa710840d 0.3s
=> [base 2/4] RUN apt-get update && apt-get dist-upgrade -y && apt-get install -y --no-install-recommends git ssh-client software-properties-co 27.8s
=> [base 3/4] RUN python -m pip install --upgrade pip setuptools wheel --no-cache-dir 5.8s
=> [base 4/4] WORKDIR /usr/app/dbt/ 0.0s
=> [dbt-all 1/6] RUN apt-get update && apt-get dist-upgrade -y && apt-get install -y --no-install-recommends python-dev libsasl2-dev gcc un 10.2s
=> [dbt-all 2/6] RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/[email protected]#egg=dbt-redshift" 21.2s
=> [dbt-all 3/6] RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/[email protected]#egg=dbt-bigquery" 8.6s
=> [dbt-all 4/6] RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/[email protected]#egg=dbt-snowflake" 7.8s
=> [dbt-all 5/6] RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/[email protected]#egg=dbt-spark[all]" 20.1s
=> [dbt-all 6/6] RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/[email protected]#egg=dbt-postgres&subdirectory=plugins/postgres" 5.7s
=> exporting to image 4.7s
=> => exporting layers 4.7s
=> => writing image sha256:631bdee89cbb2ea20aa7029136a87bc9a17bf6b81592b7265146a27ea5063877 0.0s
=> => naming to docker.io/library/foobar ```
+1. @iknox-fa, what tool are you using for docker builds (e.g. Engine, Desktop*, podman, etc)? I'm reproducing this issue on Docker Engine 20.10.12.
From the ref [0]:
An ARG instruction goes out of scope at the end of the build stage where it was defined. To use an arg in multiple stages, each stage must include the ARG instruction.
Same as with @thenaturalist, just moving the ARG lines into their respective stages worked for me, since the base stage wasn't using those.
[0] - https://docs.docker.com/engine/reference/builder/#scope
@Khouderchah-Alex Thanks for chiming in and sorry for the delayed reply. I've been doing all of my testing in Docker Desktop v 20.10.8 which should be functionally almost identical to Docker Engine 20.10.12.
Can you provide the exact docker build command and whatever pertinent env vars you're using that causes the error?
No special vars, just a plain docker build -t $TAG ..
Given the documentation about ARG scoping, I'm tempted to say the bug is with Docker Desktop rather than Engine, but I'm not familiar enough with the project to say for sure.
@korjavin Can you try the following? docker build -t $TAG --target dbt-all . The docker file was only intended to be used in conjunction with a target specified. This is documented, but sadly the PR for said docs has been held up and it's not on the site yet.
Additionally, I've tested this in linux, windows, and macos and I still can't reproduce it. Even building without a tag as you indicated creates a valid image (although it forces docker to do a TON of extra work, so avoid if possible).
Trying:
$git clone ...
$cd <root>
$docker build -t tmp --target dbt-all ./docker/
<skip a lot>
Step 19/39 : RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core"
---> Running in d384693f9402
Collecting dbt-core
Cloning https://github.com/dbt-labs/ to /tmp/pip-install-h42yrkxl/dbt-core_fa3e85a74e6b4072ba794ea699793158
Running command git clone --filter=blob:none --quiet https://github.com/dbt-labs/ /tmp/pip-install-h42yrkxl/dbt-core_fa3e85a74e6b4072ba794ea699793158
remote: Not Found
fatal: repository 'https://github.com/dbt-labs/' not found
error: subprocess-exited-with-error
× git clone --filter=blob:none --quiet https://github.com/dbt-labs/ /tmp/pip-install-h42yrkxl/dbt-core_fa3e85a74e6b4072ba794ea699793158 did not run successfully.
│ exit code: 128
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× git clone --filter=blob:none --quiet https://github.com/dbt-labs/ /tmp/pip-install-h42yrkxl/dbt-core_fa3e85a74e6b4072ba794ea699793158 did not run successfully.
│ exit code: 128
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
sigh. Well, I'm out of ideas here. I'll see if I can wrangle some better docker types to help out tomorrow.
I think I found the error. Is Dockerfile config. Due Docker create a new build stage, isn't able to keep the ARG variables after a FROM statement. So what I've done, is declare every variable again before every RUN statement. Dockerfile as below:
##
# Generic dockerfile for dbt image building.
# See README for operational details
##
# Top level build args
ARG build_for=linux/amd64
##
# base image (abstract)
##
FROM --platform=$build_for python:3.10.3-slim-bullseye as base
# N.B. The refs updated automagically every release via bumpversion
# N.B. dbt-postgres is currently found in the core codebase so a value of dbt-core@<some_version> is correct
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
ARG [email protected]
# special case args
ARG dbt_spark_version=all
ARG dbt_third_party
# System setup
RUN apt-get update \
&& apt-get dist-upgrade -y \
&& apt-get install -y --no-install-recommends \
git \
ssh-client \
software-properties-common \
make \
build-essential \
ca-certificates \
libpq-dev \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*
# Env vars
ENV PYTHONIOENCODING=utf-8
ENV LANG=C.UTF-8
# Update python
RUN python -m pip install --upgrade pip setuptools wheel --no-cache-dir
# Set docker basics
WORKDIR /usr/app/dbt/
VOLUME /usr/app
ENTRYPOINT ["dbt"]
##
# dbt-core
##
FROM base as dbt-core
ARG [email protected]
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core"
##
# dbt-postgres
##a
FROM base as dbt-postgres
ARG [email protected]
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres"
##
# dbt-redshift
##
FROM base as dbt-redshift
ARG [email protected]
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_redshift_ref}#egg=dbt-redshift"
##
# dbt-bigquery
##
FROM base as dbt-bigquery
ARG [email protected]
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_bigquery_ref}#egg=dbt-bigquery"
##
# dbt-snowflake
##
FROM base as dbt-snowflake
ARG [email protected]
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_snowflake_ref}#egg=dbt-snowflake"
##
# dbt-spark
##
FROM base as dbt-spark
RUN apt-get update \
&& apt-get dist-upgrade -y \
&& apt-get install -y --no-install-recommends \
python-dev \
libsasl2-dev \
gcc \
unixodbc-dev \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*
ARG [email protected]
ARG dbt_spark_version=all
RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_spark_ref}#egg=dbt-spark[${dbt_spark_version}]"
##
# dbt-third-party
##a
FROM dbt-core as dbt-third-party
ARG dbt_third_party
RUN python -m pip install --no-cache-dir "${dbt_third_party}"
##
# dbt-all
##
FROM base as dbt-all
RUN apt-get update \
&& apt-get dist-upgrade -y \
&& apt-get install -y --no-install-recommends \
python-dev \
libsasl2-dev \
gcc \
unixodbc-dev \
&& apt-get clean \
&& rm -rf \
/var/lib/apt/lists/* \
/tmp/* \
/var/tmp/*
ARG [email protected]
RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/${dbt_redshift_ref}#egg=dbt-redshift"
ARG [email protected]
RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/${dbt_bigquery_ref}#egg=dbt-bigquery"
ARG [email protected]
RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/${dbt_snowflake_ref}#egg=dbt-snowflake"
ARG [email protected]
ARG dbt_spark_version=all
RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/${dbt_spark_ref}#egg=dbt-spark[${dbt_spark_version}]"
ARG [email protected]
RUN python -m pip install --no-cache "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres"
I had run into the same issue. I guess @iknox-fa has "BuildKit" enabled, and that allows extra feature in Dockerfile, like reusing ARG across multiple stages, skipping not-required stages.
I have a fix to Dockerfile, but I don't want to sign CLA, so I post it here.
- Move the "base" image FROM statement after
ARG dbt_third_party. - Insert
ARG dbt_<component>_refbefore each RUN statement referencing that argument. - Insert all seven
ARG dbt_<component>_refin "dbt-all" target. - Move
dbt-third-partytarget to the end of Dockerfile, so it won't be called unless specified in the command line.
Note: an "ARG" after second FROM without values will get the default value, see reference 1.
The test.sh also need a fix to work with the latest Dockerfile.
- Either bump the version to latest 1.3.0b1
- or remove those version specific tests.
With the fixes above, this Dockerfile will work with both docker and podman, BuildKit(env variable DOCKER_BUILDKIT=1) is no longer needed.
References:
- Understand how ARG and FROM interact, see "An ARG declared before a FROM is outside of a build stage, so it can’t be used in any instruction after a FROM. To use the default value of an ARG declared before the first FROM use an ARG instruction without a value inside of a build stage"
- Containerfile(5) man page, see "Note that a second FROM in a Containerfile sets the values associated with an Arg variable to nil and they must be reset if they are to be used later in the Containerfile"
- What is Podman? Podman is a daemonless, open source, Linux native tool designed to make it easy to find, run, build, share and deploy applications using Open Containers Initiative (OCI) Containers and Container Images.
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@iknox-fa cc @iknox any update on this?
I think the issue has been scoped out, the root cause found.
Could either a) the Dockerfile be updated or b) the Docs be ammended to make it clear that Buildkit needs to be enabled?
Hi @iknox-fa, I experienced the same issue as described above and it seems logical to me.
As far as I am aware,
ARGs are only persisted within the scope of the precedingFROMstatement, unless anARGis referenced in aFROMline (i.e. version number). See also this.https://github.com/dbt-labs/dbt-core/blob/aafa1c7f473988bb1771b7cf5b120efab44a8144/docker/Dockerfile#L59
From this line onwards, all
ARGs declared above are out of scope again.This works for me:
## # dbt-core ## FROM base as dbt-core ARG [email protected] RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" ## # dbt-snowflake ## FROM base as dbt-snowflake ARG [email protected] RUN python -m pip install --no-cache-dir "git+https://github.com/dbt-labs/${dbt_snowflake_ref}#egg=dbt-snowflake"```
Hello,
I had the same issue yesterday using your Dockerfile structure. I have resolved the issue using the solution provided by @thenaturalist (moving arguments after the "FROM base" definition line)
Regards
I have a PR to fix this issue https://github.com/dbt-labs/dbt-core/pull/9590
This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.
Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.