`LocalDagRunner` ignores output artifacts from container components

[IanTayler]

System information

• Have I specified the code to reproduce the issue (Yes): (see below)
• Environment in which the code is executed: Local(Linux) amd64
• TensorFlow version: 2.6.2 (also 2.7.0)
• TFX Version: 1.4.0 (also failing with 1.5.0)
• Python version: 3.7.12
• Python dependencies (from poetry export --without-hashes output. Not using pip freeze because it points to locally cached wheels when using poetry):

absl-py==0.11.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
aiodns==3.0.0; python_version >= "3.6"
aiohttp==3.8.1; python_version >= "3.6"
aiosignal==1.2.0; python_version >= "3.6"
apache-beam==2.34.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
appnope==0.1.2; sys_platform == "darwin" and python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and platform_system == "Darwin"
argcomplete==1.12.3; python_version >= "3.7" and python_version < "3.8.0" and python_full_version >= "3.7.1"
argon2-cffi-bindings==21.2.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
argon2-cffi==21.3.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
astunparse==1.6.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
async-timeout==4.0.1; python_version >= "3.6"
asynctest==0.13.0; python_version < "3.8" and python_version >= "3.6"
attrs==20.3.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6") and (python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0")
avro-python3==1.9.2.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
backcall==0.2.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
bleach==4.1.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
boto3==1.20.23; python_version >= "3.6"
botocore==1.23.23; python_version >= "3.6"
brotli==1.0.9; python_version >= "3.6"
bugsnag==4.1.1; python_version >= "3.5" and python_version < "4"
cached-property==1.5.2; python_version < "3.8" and python_version >= "3.6"
cachetools==4.2.4; python_version >= "3.7" and python_version < "3.9" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6") and python_full_version >= "3.7.1"
cchardet==2.1.7; python_version >= "3.6"
certifi==2021.10.8; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
cffi==1.15.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and implementation_name == "pypy"
charset-normalizer==2.0.9; python_full_version >= "3.6.0" and python_version >= "3.6"
clang==5.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
click==7.1.2; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
cloudpickle==2.0.0; python_version >= "3.6" and python_full_version >= "3.6.1"
colorama==0.4.4; sys_platform == "win32" and python_version >= "3.7" and python_full_version >= "3.7.1" and python_version < "3.9"
comet-ml==3.23.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.5.0")
configobj==5.0.6; python_version > "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version > "3.5"
crcmod==1.7; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
debugpy==1.5.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
decorator==5.1.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
defusedxml==0.7.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
deprecated==1.2.13; python_full_version >= "3.6.1"
dill==0.3.1.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
docker==4.4.4; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
docopt==0.6.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
docstring-parser==0.13; python_version >= "3.6" and python_full_version >= "3.6.1"
dulwich==0.20.26; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
entrypoints==0.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
everett==2.0.1; python_version > "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version > "3.5"
fastavro==1.4.7; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
fasteners==0.16.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
fire==0.4.0; python_full_version >= "3.6.1"
flatbuffers==1.12; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
frozenlist==1.2.0; python_version >= "3.6"
future==0.18.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
gast==0.4.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-api-core==1.31.4; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
google-api-python-client==1.12.8; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-apitools==0.5.31; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-auth-httplib2==0.1.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-auth-oauthlib==0.4.6; python_version >= "3.6"
google-auth==1.35.0; python_full_version >= "3.7.1" and python_version >= "3.7" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6") and python_version < "3.9"
google-cloud-aiplatform==1.8.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-appengine-logging==1.1.0; python_version >= "3.6"
google-cloud-audit-log==0.2.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6"
google-cloud-bigquery-storage==2.10.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-bigquery==2.31.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-bigtable==1.7.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-core==1.7.2; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
google-cloud-datastore==1.15.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-dlp==1.0.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-language==1.3.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-logging==2.7.0; python_version >= "3.6"
google-cloud-pubsub==1.7.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-recommendations-ai==0.2.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-secret-manager==2.8.0; python_version >= "3.6"
google-cloud-spanner==1.19.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-storage==1.43.0; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.6.0")
google-cloud-videointelligence==1.16.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-cloud-vision==1.0.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-crc32c==1.3.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
google-pasta==0.2.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
google-resumable-media==2.1.0; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
googleapis-common-protos==1.54.0; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
grpc-google-iam-v1==0.12.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
grpcio-gcp==0.2.2
grpcio==1.42.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6")
h5py==3.1.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
hdfs==2.6.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
httplib2==0.19.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
idna==3.3; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6"
imbalanced-learn==0.8.1
importlib-metadata==4.8.2; python_full_version >= "3.7.1" and python_version == "3.7"
ipykernel==6.6.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
ipython-genutils==0.2.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
ipython==7.30.1; python_version >= "3.7"
ipywidgets==7.6.5; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jedi==0.18.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jinja2==3.0.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jmespath==0.10.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6"
joblib==0.14.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jsonschema==3.2.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jupyter-client==7.1.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jupyter-core==4.9.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jupyterlab-pygments==0.1.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
jupyterlab-widgets==1.0.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
keras-preprocessing==1.1.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
keras-tuner==1.1.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
keras==2.6.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
kfp-pipeline-spec==0.1.13; python_full_version >= "3.6.1"
kfp-server-api==1.7.1; python_full_version >= "3.6.1"
kfp==1.8.10; python_full_version >= "3.6.1"
kt-legacy==1.0.4; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
kubernetes==12.0.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
libcst==0.3.23; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
markdown==3.3.6; python_version >= "3.6"
markupsafe==2.0.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
matplotlib-inline==0.1.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
mistune==0.8.4; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
ml-metadata==1.4.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
ml-pipelines-sdk==1.4.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
multidict==5.2.0; python_version >= "3.6"
mypy-extensions==0.4.3; python_version >= "3.6"
nbclient==0.5.9; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
nbconvert==6.3.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
nbformat==5.1.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
nest-asyncio==1.5.4; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
notebook==6.4.6; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
numpy==1.19.5; python_version >= "3.6"
nvidia-ml-py3==7.352.0; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
oauth2client==4.1.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
oauthlib==3.1.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
opt-einsum==3.3.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
orjson==3.6.5; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
packaging==20.9; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6") and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
pandas==1.3.3; python_full_version >= "3.7.1"
pandocfilters==1.5.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
parso==0.8.3; python_version >= "3.7"
pexpect==4.8.0; sys_platform != "win32" and python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
pickleshare==0.7.5; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
portpicker==1.5.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
prometheus-client==0.12.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
prompt-toolkit==3.0.24; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
proto-plus==1.19.8; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
protobuf==3.19.1; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6") and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
psutil==5.8.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
ptyprocess==0.7.0; sys_platform != "win32" and python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and os_name != "nt"
py==1.11.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and implementation_name == "pypy"
pyarrow==5.0.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
pyasn1-modules==0.2.8; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
pyasn1==0.4.8; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6")
pycares==4.1.2; python_version >= "3.6"
pycparser==2.21; python_version >= "3.6" and python_full_version < "3.0.0" or python_version >= "3.6" and python_full_version >= "3.4.0"
pydantic==1.8.2; python_full_version >= "3.6.1"
pydot==1.4.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
pygments==2.10.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
pymongo==3.12.3; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
pyparsing==2.4.7; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
pyrsistent==0.18.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.6"
python-dateutil==2.8.2; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
python-dotenv==0.19.2; python_version >= "3.5"
pytz==2021.3; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
pywin32==227; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and sys_platform == "win32" and platform_python_implementation != "PyPy"
pywinpty==1.1.6; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and os_name == "nt"
pyyaml==5.4.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6")
pyzmq==22.3.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
requests-oauthlib==1.3.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.4.0" and python_version >= "3.6")
requests-toolbelt==0.9.1; python_full_version >= "3.6.1"
requests==2.26.0; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
rsa==4.8; python_version >= "3.7" and python_version < "3.9" and (python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.6.0" and python_version >= "3.6") and python_full_version >= "3.7.1"
s3transfer==0.5.0; python_version >= "3.6"
scikit-learn==1.0.1; python_version >= "3.7"
scipy==1.6.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
semantic-version==2.8.5; python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.5.0"
send2trash==1.8.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
six==1.15.0; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9" and (python_version > "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version > "3.5")
strip-hints==0.1.10; python_full_version >= "3.6.1"
tabulate==0.8.9; python_full_version >= "3.6.1"
tensorboard-data-server==0.6.1; python_version >= "3.6"
tensorboard-plugin-wit==1.8.0; python_version >= "3.6"
tensorboard==2.6.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow-data-validation==1.4.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow-estimator==2.6.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow-hub==0.12.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow-metadata==1.4.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow-model-analysis==0.35.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow-serving-api==2.6.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow-transform==1.4.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tensorflow==2.6.2
termcolor==1.1.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
terminado==0.12.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
testpath==0.5.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tfx-bsl==1.4.0; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
tfx==1.4.0; python_full_version >= "3.7.1" and python_version < "3.9"
threadpoolctl==3.0.0; python_version >= "3.7"
tornado==6.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
traitlets==5.1.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
typer==0.4.0; python_version >= "3.6" and python_full_version >= "3.6.1"
typing-extensions==3.7.4.3; python_version < "3.8" and python_full_version >= "3.7.1" and python_version >= "3.7" and (python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6")
typing-inspect==0.7.1; python_version >= "3.6"
uritemplate==3.0.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
urllib3==1.26.7; python_full_version >= "3.7.1" and python_version < "3.9" and python_version >= "3.7"
wcwidth==0.2.5; python_full_version >= "3.6.2" and python_version >= "3.7"
webencodings==0.5.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
webob==1.8.7; python_version >= "3.5" and python_full_version < "3.0.0" and python_version < "4" or python_version >= "3.5" and python_version < "4" and python_full_version >= "3.3.0"
websocket-client==1.2.3; python_full_version >= "3.7.1" and python_version >= "3.7" and python_version < "3.9"
werkzeug==2.0.2; python_version >= "3.6"
widgetsnbextension==3.5.2; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
wrapt==1.12.1; python_version >= "3.7" and python_version < "3.9" and python_full_version >= "3.7.1"
wurlitzer==3.0.2; python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5"
yarl==1.7.2; python_version >= "3.6"
zipp==3.6.0; python_version >= "3.6" and python_full_version < "3.0.0" and python_version < "3.8" or python_full_version >= "3.5.0" and python_version < "3.8" and python_version >= "3.6"

Describe the current behavior

Even though output artifacts are correctly written inside the container, these files are never writter to the pipeline root and are never passed to the subsequent components that need them.

After running the code I wrote below, I run docker container ls --all and can check that I had two container runs but the second one got an empty value for output x even though by doing docker cp <first_container_hash>:/tmp/output/ first_out I can check that my container is correctly writing the value inside the provided path in /tmp/output.

I have also tried changing the InputValuePlaceholder for the String to an InputUriPlaceholder (while changing echo to cat) and that makes the second container fail with a No such file or directory error message from cat.

Of course, the original pipeline I got this problem with was more complicated than the one shown below. In that one, I also saw this problem with tfx.types.experimental.simple_artifacts.Metrics artifacts. These are the only two types of artifacts I tested the LocalDagRunner with. I don't want to give the impression I saw it working for other artifact types.

docker inspect shows there are no volumes in the containers created by the LocalDagRunner.

Describe the expected behavior

The file written in the docker file to the output location should be copied to the host pipeline_root for inspection and passed to all components that need it. Probably the most reasonable way to get this would be by using volumes.

Standalone code to reproduce the issue

import click
from tfx.dsl.component.experimental.container_component import (
    create_container_component,
)
from tfx.dsl.component.experimental.placeholders import (
    ConcatPlaceholder,
    InputValuePlaceholder,
    OutputUriPlaceholder,
)
from tfx.orchestration.metadata import sqlite_metadata_connection_config
from tfx.types.standard_artifacts import String
from tfx.v1.dsl import Pipeline
from tfx.v1.orchestration import LocalDagRunner

PIPELINE_NAME = "mlops-dummy-pipeline"
PIPELINE_ROOT = "/tmp/output/"
METADATA_CONNECTION_CONFIG = sqlite_metadata_connection_config(
    "/tmp/pipeline_metadata.db"
)

python_script_cmd = ConcatPlaceholder(
    [
        """
import os

os.makedirs(os.path.dirname('""",
        OutputUriPlaceholder("x"),
        """'), exist_ok=True)
with open('""",
        OutputUriPlaceholder("x"),
        """', 'w') as f:
    f.write('someoutput')
""",
    ]
)


@click.command(help=__doc__)
def _broken_localdag_runner():
    component_image = "python:3.8"
    component_a = create_container_component(
        name="A",
        image=component_image,
        outputs={"x": String},
        command=["python", "-c", python_script_cmd],
    )()
    component_b = create_container_component(
        name="B",
        image=component_image,
        inputs={"x": String},
        command=["echo", InputValuePlaceholder("x")],
    )(x=component_a.outputs["x"])
    components = [
        component_a,
        component_b,
    ]
    pipeline = Pipeline(
        pipeline_name=PIPELINE_NAME,
        pipeline_root=PIPELINE_ROOT,
        components=components,
        metadata_connection_config=METADATA_CONNECTION_CONFIG,
    )
    runner = LocalDagRunner()
    out = runner.run(pipeline)
    print(out)


if __name__ == "__main__":
    _broken_localdag_runner()

Other info / logs

Containers after running:

Value of output inside container (run number 7):

Let me know if there's any other information I can provide or any other way I can help with this issue.

[charlesccychen]

Hi @IanTayler, thank you for the report and apologies for the late reply.

Currently, you are right that the current implementation does not use volumes to achieve value-passing here. In current usage (and in usage on cloud-based runners), users set their pipeline root to be a storage bucket (say on GCS or S3). Then, subsequent writes to the intermediate files are accessible by all containers when run in a certain cloud project. We currently have a limitation that this scheme does not work as well for local execution, and have not yet implemented the local volume type approach.

[SoleRivas]

Hi @charlesccychen, thank you very much for the response. Soledad here, I work on the same team that @IanTayler and he's OOO today.

Is the local volume implementation on your development roadmap? If that's the case, do you have an estimate of when this will be Implemented?

Thanks in advance!

[IanTayler]

For future reference, I was actually able to work around this by using a BeamDagRunner and attaching a platform_config_pb2.DockerPlatformConfig() to the components, setting the volumes field to the local pipeline root (which should be an absolute path for it to work properly).

[singhniraj08]

@IanTayler,

Could you please confirm if this issue can be closed since the workaround of using BeamDagRunner and setting volumes field to local pipeline works for you. Thank you!

[IanTayler]

The workaround works, although it's a bit hacky.

Whether that means this issue can be closed depends on whether the tfx team thinks local pipeline roots should be implemented for LocalDagRunner or not. On our side, we have a working alternative and would not mind too much either way.