NVTabular
NVTabular copied to clipboard
NVIDIA-Merlin
Scaling Criteo updates + Multi-GPU TensorFlow version
- Fixed glob pattern for converting CSV to Parquet
- Updated ETL text about HugeCTR data types to be more precise
- Updated Training HugeCTR notebook to use variable for file names
- Added notebook for Multi-GPU TensorFlow training
Check out this pull request on 
See visual diffs & provide feedback on Jupyter Notebooks.
Powered by ReviewNB
![review-notebook-app[bot] avatar](https://avatars.githubusercontent.com/in/15474?v=4 "Published by a pub.dev verified publisher"){kind=small}
Click to view CI Results
GitHub pull request #1278 of commit 2948bbb9934fd383ff92656e6f8eb291bef08e37, no merge conflicts.
Running as SYSTEM
Setting status of 2948bbb9934fd383ff92656e6f8eb291bef08e37 to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/3802/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
> git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1278/*:refs/remotes/origin/pr/1278/* # timeout=10
> git rev-parse 2948bbb9934fd383ff92656e6f8eb291bef08e37^{commit} # timeout=10
Checking out Revision 2948bbb9934fd383ff92656e6f8eb291bef08e37 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 2948bbb9934fd383ff92656e6f8eb291bef08e37 # timeout=10
Commit message: "tf multi gpu"
> git rev-list --no-walk 6b72f89a4a60b9dacfa06f3b95abd3943db9ce8e # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins3283621764864235266.sh
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in /var/jenkins_home/.local/lib/python3.8/site-packages (21.3.1)
Requirement already satisfied: setuptools in /var/jenkins_home/.local/lib/python3.8/site-packages (59.4.0)
Requirement already satisfied: wheel in /var/jenkins_home/.local/lib/python3.8/site-packages (0.37.0)
Requirement already satisfied: pybind11 in /var/jenkins_home/.local/lib/python3.8/site-packages (2.8.1)
Requirement already satisfied: numpy==1.20.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (1.20.3)
running develop
running egg_info
creating nvtabular.egg-info
writing nvtabular.egg-info/PKG-INFO
writing dependency_links to nvtabular.egg-info/dependency_links.txt
writing requirements to nvtabular.egg-info/requires.txt
writing top-level names to nvtabular.egg-info/top_level.txt
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
warning: no files found matching '*.h' under directory 'cpp'
warning: no files found matching '*.cu' under directory 'cpp'
warning: no files found matching '*.cuh' under directory 'cpp'
adding license file 'LICENSE'
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
running build_ext
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c flagcheck.cpp -o flagcheck.o -std=c++17
building 'nvtabular_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/cpp
creating build/temp.linux-x86_64-3.8/cpp/nvtabular
creating build/temp.linux-x86_64-3.8/cpp/nvtabular/inference
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+41.g2948bbb -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+41.g2948bbb -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+41.g2948bbb -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/categorify.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+41.g2948bbb -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/fill.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -std=c++17 -fvisibility=hidden -g0
creating build/lib.linux-x86_64-3.8
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -o build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so ->
Generating nvtabular/inference/triton/model_config_pb2.py from nvtabular/inference/triton/model_config.proto
Creating /var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link (link to .)
nvtabular 0.7.1+41.g2948bbb is already the active version in easy-install.pth
Installed /var/jenkins_home/workspace/nvtabular_tests/nvtabular
Processing dependencies for nvtabular==0.7.1+41.g2948bbb
Searching for protobuf==3.18.0
Best match: protobuf 3.18.0
Adding protobuf 3.18.0 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for tensorflow-metadata==1.2.0
Best match: tensorflow-metadata 1.2.0
Processing tensorflow_metadata-1.2.0-py3.8.egg
tensorflow-metadata 1.2.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow_metadata-1.2.0-py3.8.egg
Searching for pyarrow==4.0.1
Best match: pyarrow 4.0.1
Adding pyarrow 4.0.1 to easy-install.pth file
Installing plasma_store script to /var/jenkins_home/.local/bin
Using /usr/local/lib/python3.8/dist-packages
Searching for tqdm==4.61.2
Best match: tqdm 4.61.2
Processing tqdm-4.61.2-py3.8.egg
tqdm 4.61.2 is already the active version in easy-install.pth
Installing tqdm script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tqdm-4.61.2-py3.8.egg
Searching for numba==0.54.0
Best match: numba 0.54.0
Processing numba-0.54.0-py3.8-linux-x86_64.egg
numba 0.54.0 is already the active version in easy-install.pth
Installing pycc script to /var/jenkins_home/.local/bin
Installing numba script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages/numba-0.54.0-py3.8-linux-x86_64.egg
Searching for pandas==1.3.3
Best match: pandas 1.3.3
Processing pandas-1.3.3-py3.8-linux-x86_64.egg
pandas 1.3.3 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/pandas-1.3.3-py3.8-linux-x86_64.egg
Searching for distributed==2021.7.1
Best match: distributed 2021.7.1
Processing distributed-2021.7.1-py3.8.egg
distributed 2021.7.1 is already the active version in easy-install.pth
Installing dask-ssh script to /var/jenkins_home/.local/bin
Installing dask-scheduler script to /var/jenkins_home/.local/bin
Installing dask-worker script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages/distributed-2021.7.1-py3.8.egg
Searching for dask==2021.7.1
Best match: dask 2021.7.1
Adding dask 2021.7.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for googleapis-common-protos==1.53.0
Best match: googleapis-common-protos 1.53.0
Processing googleapis_common_protos-1.53.0-py3.8.egg
googleapis-common-protos 1.53.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/googleapis_common_protos-1.53.0-py3.8.egg
Searching for absl-py==0.12.0
Best match: absl-py 0.12.0
Processing absl_py-0.12.0-py3.8.egg
absl-py 0.12.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/absl_py-0.12.0-py3.8.egg
Searching for numpy==1.20.3
Best match: numpy 1.20.3
Adding numpy 1.20.3 to easy-install.pth file
Installing f2py script to /var/jenkins_home/.local/bin
Installing f2py3 script to /var/jenkins_home/.local/bin
Installing f2py3.8 script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for setuptools==59.4.0
Best match: setuptools 59.4.0
Adding setuptools 59.4.0 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for llvmlite==0.37.0
Best match: llvmlite 0.37.0
Processing llvmlite-0.37.0-py3.8-linux-x86_64.egg
llvmlite 0.37.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/llvmlite-0.37.0-py3.8-linux-x86_64.egg
Searching for pytz==2021.1
Best match: pytz 2021.1
Adding pytz 2021.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for python-dateutil==2.8.2
Best match: python-dateutil 2.8.2
Adding python-dateutil 2.8.2 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for zict==2.0.0
Best match: zict 2.0.0
Processing zict-2.0.0-py3.8.egg
zict 2.0.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg
Searching for tornado==6.1
Best match: tornado 6.1
Processing tornado-6.1-py3.8-linux-x86_64.egg
tornado 6.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg
Searching for toolz==0.11.1
Best match: toolz 0.11.1
Processing toolz-0.11.1-py3.8.egg
toolz 0.11.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/toolz-0.11.1-py3.8.egg
Searching for tblib==1.7.0
Best match: tblib 1.7.0
Processing tblib-1.7.0-py3.8.egg
tblib 1.7.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg
Searching for sortedcontainers==2.4.0
Best match: sortedcontainers 2.4.0
Processing sortedcontainers-2.4.0-py3.8.egg
sortedcontainers 2.4.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg
Searching for PyYAML==5.4.1
Best match: PyYAML 5.4.1
Processing PyYAML-5.4.1-py3.8-linux-x86_64.egg
PyYAML 5.4.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg
Searching for psutil==5.8.0
Best match: psutil 5.8.0
Processing psutil-5.8.0-py3.8-linux-x86_64.egg
psutil 5.8.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg
Searching for msgpack==1.0.2
Best match: msgpack 1.0.2
Processing msgpack-1.0.2-py3.8-linux-x86_64.egg
msgpack 1.0.2 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/msgpack-1.0.2-py3.8-linux-x86_64.egg
Searching for cloudpickle==1.6.0
Best match: cloudpickle 1.6.0
Processing cloudpickle-1.6.0-py3.8.egg
cloudpickle 1.6.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/cloudpickle-1.6.0-py3.8.egg
Searching for click==8.0.1
Best match: click 8.0.1
Processing click-8.0.1-py3.8.egg
click 8.0.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/click-8.0.1-py3.8.egg
Searching for packaging==21.0
Best match: packaging 21.0
Adding packaging 21.0 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for partd==1.2.0
Best match: partd 1.2.0
Processing partd-1.2.0-py3.8.egg
partd 1.2.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg
Searching for fsspec==2021.11.1
Best match: fsspec 2021.11.1
Adding fsspec 2021.11.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for six==1.15.0
Best match: six 1.15.0
Adding six 1.15.0 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for HeapDict==1.0.1
Best match: HeapDict 1.0.1
Processing HeapDict-1.0.1-py3.8.egg
HeapDict 1.0.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg
Searching for pyparsing==2.4.7
Best match: pyparsing 2.4.7
Adding pyparsing 2.4.7 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for locket==0.2.1
Best match: locket 0.2.1
Processing locket-0.2.1-py3.8.egg
locket 0.2.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg
Finished processing dependencies for nvtabular==0.7.1+41.g2948bbb
Running black --check
All done! ✨ 🍰 ✨
149 files would be left unchanged.
Running flake8
Running isort
Skipped 2 files
Running bandit
Running pylint
************* Module nvtabular.dispatch
nvtabular/dispatch.py:605:11: I1101: Module 'numpy.random.mtrand' has no 'RandomState' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
************* Module nvtabular.ops.categorify
nvtabular/ops/categorify.py:496:15: I1101: Module 'nvtabular_cpp' has no 'inference' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
************* Module nvtabular.ops.fill
nvtabular/ops/fill.py:67:15: I1101: Module 'nvtabular_cpp' has no 'inference' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
Running flake8-nb
examples/scaling-criteo/03-Training-with-TF-multi-GPU.ipynb#In[ ]:1:121: E501 line too long (124 > 120 characters)
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[nvtabular_tests] $ /bin/bash /tmp/jenkins5732520750049128604.sh
- Added column tagging API feature
- Fixed TensorFlow example to use numerical input features
Click to view CI Results
GitHub pull request #1278 of commit 13eccb0fb1b9eaada1fb455a7fdb477e93ed8596, no merge conflicts.
Running as SYSTEM
Setting status of 13eccb0fb1b9eaada1fb455a7fdb477e93ed8596 to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/3927/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
> git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1278/*:refs/remotes/origin/pr/1278/* # timeout=10
> git rev-parse 13eccb0fb1b9eaada1fb455a7fdb477e93ed8596^{commit} # timeout=10
Checking out Revision 13eccb0fb1b9eaada1fb455a7fdb477e93ed8596 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 13eccb0fb1b9eaada1fb455a7fdb477e93ed8596 # timeout=10
Commit message: "added column tagging api"
> git rev-list --no-walk be3d3a91138c649ca494570c7f47c43597efe1e1 # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins3848714141918175700.sh
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in /var/jenkins_home/.local/lib/python3.8/site-packages (21.3.1)
Requirement already satisfied: setuptools in /var/jenkins_home/.local/lib/python3.8/site-packages (59.6.0)
Requirement already satisfied: wheel in /var/jenkins_home/.local/lib/python3.8/site-packages (0.37.0)
Requirement already satisfied: pybind11 in /var/jenkins_home/.local/lib/python3.8/site-packages (2.8.1)
Requirement already satisfied: numpy==1.20.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (1.20.3)
running develop
running egg_info
creating nvtabular.egg-info
writing nvtabular.egg-info/PKG-INFO
writing dependency_links to nvtabular.egg-info/dependency_links.txt
writing requirements to nvtabular.egg-info/requires.txt
writing top-level names to nvtabular.egg-info/top_level.txt
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
warning: no files found matching '*.h' under directory 'cpp'
warning: no files found matching '*.cu' under directory 'cpp'
warning: no files found matching '*.cuh' under directory 'cpp'
adding license file 'LICENSE'
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
running build_ext
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c flagcheck.cpp -o flagcheck.o -std=c++17
building 'nvtabular_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/cpp
creating build/temp.linux-x86_64-3.8/cpp/nvtabular
creating build/temp.linux-x86_64-3.8/cpp/nvtabular/inference
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+42.g13eccb0 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+42.g13eccb0 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+42.g13eccb0 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/categorify.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+42.g13eccb0 -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/fill.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -std=c++17 -fvisibility=hidden -g0
creating build/lib.linux-x86_64-3.8
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -o build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so ->
Generating nvtabular/inference/triton/model_config_pb2.py from nvtabular/inference/triton/model_config.proto
Creating /var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link (link to .)
nvtabular 0.7.1+42.g13eccb0 is already the active version in easy-install.pth
Installed /var/jenkins_home/workspace/nvtabular_tests/nvtabular
Processing dependencies for nvtabular==0.7.1+42.g13eccb0
Searching for protobuf==3.19.1
Best match: protobuf 3.19.1
Adding protobuf 3.19.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for tensorflow-metadata==1.2.0
Best match: tensorflow-metadata 1.2.0
Processing tensorflow_metadata-1.2.0-py3.8.egg
tensorflow-metadata 1.2.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow_metadata-1.2.0-py3.8.egg
Searching for pyarrow==4.0.1
Best match: pyarrow 4.0.1
Adding pyarrow 4.0.1 to easy-install.pth file
Installing plasma_store script to /var/jenkins_home/.local/bin
Using /usr/local/lib/python3.8/dist-packages
Searching for tqdm==4.61.2
Best match: tqdm 4.61.2
Processing tqdm-4.61.2-py3.8.egg
tqdm 4.61.2 is already the active version in easy-install.pth
Installing tqdm script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tqdm-4.61.2-py3.8.egg
Searching for numba==0.54.1
Best match: numba 0.54.1
Adding numba 0.54.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for pandas==1.3.3
Best match: pandas 1.3.3
Processing pandas-1.3.3-py3.8-linux-x86_64.egg
pandas 1.3.3 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/pandas-1.3.3-py3.8-linux-x86_64.egg
Searching for distributed==2021.7.1
Best match: distributed 2021.7.1
Processing distributed-2021.7.1-py3.8.egg
distributed 2021.7.1 is already the active version in easy-install.pth
Installing dask-ssh script to /var/jenkins_home/.local/bin
Installing dask-scheduler script to /var/jenkins_home/.local/bin
Installing dask-worker script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages/distributed-2021.7.1-py3.8.egg
Searching for dask==2021.7.1
Best match: dask 2021.7.1
Processing dask-2021.7.1-py3.8.egg
dask 2021.7.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/dask-2021.7.1-py3.8.egg
Searching for googleapis-common-protos==1.53.0
Best match: googleapis-common-protos 1.53.0
Processing googleapis_common_protos-1.53.0-py3.8.egg
googleapis-common-protos 1.53.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/googleapis_common_protos-1.53.0-py3.8.egg
Searching for absl-py==0.12.0
Best match: absl-py 0.12.0
Processing absl_py-0.12.0-py3.8.egg
absl-py 0.12.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/absl_py-0.12.0-py3.8.egg
Searching for numpy==1.20.3
Best match: numpy 1.20.3
Adding numpy 1.20.3 to easy-install.pth file
Installing f2py script to /var/jenkins_home/.local/bin
Installing f2py3 script to /var/jenkins_home/.local/bin
Installing f2py3.8 script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for llvmlite==0.37.0
Best match: llvmlite 0.37.0
Processing llvmlite-0.37.0-py3.8-linux-x86_64.egg
llvmlite 0.37.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/llvmlite-0.37.0-py3.8-linux-x86_64.egg
Searching for setuptools==59.6.0
Best match: setuptools 59.6.0
Adding setuptools 59.6.0 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for pytz==2021.3
Best match: pytz 2021.3
Adding pytz 2021.3 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for python-dateutil==2.8.2
Best match: python-dateutil 2.8.2
Adding python-dateutil 2.8.2 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for zict==2.0.0
Best match: zict 2.0.0
Processing zict-2.0.0-py3.8.egg
zict 2.0.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg
Searching for tornado==6.1
Best match: tornado 6.1
Processing tornado-6.1-py3.8-linux-x86_64.egg
tornado 6.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg
Searching for toolz==0.11.1
Best match: toolz 0.11.1
Processing toolz-0.11.1-py3.8.egg
toolz 0.11.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/toolz-0.11.1-py3.8.egg
Searching for tblib==1.7.0
Best match: tblib 1.7.0
Processing tblib-1.7.0-py3.8.egg
tblib 1.7.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg
Searching for sortedcontainers==2.4.0
Best match: sortedcontainers 2.4.0
Processing sortedcontainers-2.4.0-py3.8.egg
sortedcontainers 2.4.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg
Searching for PyYAML==5.4.1
Best match: PyYAML 5.4.1
Processing PyYAML-5.4.1-py3.8-linux-x86_64.egg
PyYAML 5.4.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg
Searching for psutil==5.8.0
Best match: psutil 5.8.0
Processing psutil-5.8.0-py3.8-linux-x86_64.egg
psutil 5.8.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg
Searching for msgpack==1.0.2
Best match: msgpack 1.0.2
Processing msgpack-1.0.2-py3.8-linux-x86_64.egg
msgpack 1.0.2 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/msgpack-1.0.2-py3.8-linux-x86_64.egg
Searching for cloudpickle==1.6.0
Best match: cloudpickle 1.6.0
Processing cloudpickle-1.6.0-py3.8.egg
cloudpickle 1.6.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/cloudpickle-1.6.0-py3.8.egg
Searching for click==8.0.1
Best match: click 8.0.1
Processing click-8.0.1-py3.8.egg
click 8.0.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/click-8.0.1-py3.8.egg
Searching for partd==1.2.0
Best match: partd 1.2.0
Processing partd-1.2.0-py3.8.egg
partd 1.2.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg
Searching for packaging==21.2
Best match: packaging 21.2
Adding packaging 21.2 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for fsspec==2021.11.1
Best match: fsspec 2021.11.1
Adding fsspec 2021.11.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for six==1.15.0
Best match: six 1.15.0
Adding six 1.15.0 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for HeapDict==1.0.1
Best match: HeapDict 1.0.1
Processing HeapDict-1.0.1-py3.8.egg
HeapDict 1.0.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg
Searching for locket==0.2.1
Best match: locket 0.2.1
Processing locket-0.2.1-py3.8.egg
locket 0.2.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg
Searching for pyparsing==2.4.7
Best match: pyparsing 2.4.7
Adding pyparsing 2.4.7 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Finished processing dependencies for nvtabular==0.7.1+42.g13eccb0
Running black --check
All done! ✨ 🍰 ✨
149 files would be left unchanged.
Running flake8
Running isort
Skipped 2 files
Running bandit
Running pylint
************* Module nvtabular.dispatch
nvtabular/dispatch.py:605:11: I1101: Module 'numpy.random.mtrand' has no 'RandomState' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
************* Module nvtabular.ops.categorify
nvtabular/ops/categorify.py:496:15: I1101: Module 'nvtabular_cpp' has no 'inference' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
************* Module nvtabular.ops.fill
nvtabular/ops/fill.py:67:15: I1101: Module 'nvtabular_cpp' has no 'inference' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
Running flake8-nb
Building docs
make: Entering directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.7) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
make: Leaving directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 1570 items / 1 skipped / 1569 selected
tests/unit/test_dask_nvt.py ............................................ [ 2%]
....................................................................... [ 7%]
tests/unit/test_io.py .................................................. [ 10%]
........................................................................ [ 15%]
.................ssssssss............................................... [ 19%]
........ [ 20%]
tests/unit/test_notebooks.py F
=================================== FAILURES ===================================
___________________________ test_criteo_tf_notebook ____________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-2/test_criteo_tf_notebook0')
def test_criteo_tf_notebook(tmpdir):
tor = pytest.importorskip("tensorflow") # noqa
# create a toy dataset in tmpdir, and point environment variables so the notebook
# will read from it
os.system("mkdir -p " + os.path.join(tmpdir, "converted/criteo"))
for i in range(24):
df = _get_random_criteo_data(1000)
df.to_parquet(os.path.join(tmpdir, "converted/criteo", f"day_{i}.parquet"))
os.environ["BASE_DIR"] = str(tmpdir)
def _nb_modify(line):
# Disable LocalCUDACluster
line = line.replace("client.run(_rmm_pool)", "# client.run(_rmm_pool)")
line = line.replace("if cluster is None:", "if False:")
line = line.replace("client = Client(cluster)", "# client = Client(cluster)")
line = line.replace(
"workflow = nvt.Workflow(features, client=client)", "workflow = nvt.Workflow(features)"
)
line = line.replace("client", "# client")
line = line.replace("NUM_GPUS = [0, 1, 2, 3, 4, 5, 6, 7]", "NUM_GPUS = [0]")
line = line.replace("part_size = int(part_mem_frac * device_size)", "part_size = '128MB'")
return line
_run_notebook(
tmpdir,
os.path.join(
dirname(TEST_PATH),
"examples/scaling-criteo/",
"02-ETL-with-NVTabular.ipynb",
),
# disable rmm.reinitialize, seems to be causing issues
transform=_nb_modify,
)
tests/unit/test_notebooks.py:62:
tests/unit/test_notebooks.py:307: in _run_notebook
subprocess.check_output([sys.executable, script_path])
/usr/lib/python3.8/subprocess.py:415: in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '/tmp/pytest-of-jenkins/pytest-2/test_criteo_tf_notebook0/notebook.py'],)
kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f2978581460>
stdout = b'feature {\n name: "C1"\n type: INT\n int_domain {\n name: "C1"\n min: 0\n max: 227\n is_categorical: ...eo/day_22.parquet']\n['/tmp/pytest-of-jenkins/pytest-2/test_criteo_tf_notebook0/converted/criteo/day_23.parquet']\n'
stderr = None, retcode = 1
def run(*popenargs,
input=None, capture_output=False, timeout=None, check=False, **kwargs):
"""Run command with arguments and return a CompletedProcess instance.
The returned instance will have attributes args, returncode, stdout and
stderr. By default, stdout and stderr are not captured, and those attributes
will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
If check is True and the exit code was non-zero, it raises a
CalledProcessError. The CalledProcessError object will have the return code
in the returncode attribute, and output & stderr attributes if those streams
were captured.
If timeout is given, and the process takes too long, a TimeoutExpired
exception will be raised.
There is an optional argument "input", allowing you to
pass bytes or a string to the subprocess's stdin. If you use this argument
you may not also use the Popen constructor's "stdin" argument, as
it will be used internally.
By default, all communication is in bytes, and therefore any "input" should
be bytes, and the stdout and stderr will be bytes. If in text mode, any
"input" should be a string, and stdout and stderr will be strings decoded
according to locale encoding, or by "encoding" if set. Text mode is
triggered by setting any of text, encoding, errors or universal_newlines.
The other arguments are the same as for the Popen constructor.
"""
if input is not None:
if kwargs.get('stdin') is not None:
raise ValueError('stdin and input arguments may not both be used.')
kwargs['stdin'] = PIPE
if capture_output:
if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
raise ValueError('stdout and stderr arguments may not be used '
'with capture_output.')
kwargs['stdout'] = PIPE
kwargs['stderr'] = PIPE
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
retcode = process.poll()
if check and retcode:
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
E subprocess.CalledProcessError: Command '['/usr/bin/python3', '/tmp/pytest-of-jenkins/pytest-2/test_criteo_tf_notebook0/notebook.py']' returned non-zero exit status 1.
/usr/lib/python3.8/subprocess.py:516: CalledProcessError
----------------------------- Captured stderr call -----------------------------
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (1) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
Traceback (most recent call last):
File "/tmp/pytest-of-jenkins/pytest-2/test_criteo_tf_notebook0/notebook.py", line 164, in
[x.name for x in schema.select_by_tag(nvt.graph.tags.Tags.CATEGORICAL)]
AttributeError: module 'nvtabular' has no attribute 'graph'
=============================== warnings summary ===============================
tests/unit/test_dask_nvt.py: 3 warnings
tests/unit/test_io.py: 24 warnings
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (1) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/test_dask_nvt.py: 2 warnings
tests/unit/test_io.py: 36 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/workflow/workflow.py:89: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler will be used for execution. Please use the client argument to initialize a Workflow object with distributed-execution enabled.
warnings.warn(
tests/unit/test_dask_nvt.py: 2 warnings
tests/unit/test_io.py: 52 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dask.py:375: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler will be used for this write operation. Please use the client argument to initialize a Dataset and/or Workflow object with distributed-execution enabled.
warnings.warn(
tests/unit/test_io.py: 96 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/init.py:38: DeprecationWarning: ColumnGroup is deprecated, use ColumnSelector instead
warnings.warn("ColumnGroup is deprecated, use ColumnSelector instead", DeprecationWarning)
tests/unit/test_io.py: 20 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:521: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler is being used for this shuffle operation. Please use the client argument to initialize a Dataset and/or Workflow object with distributed-execution enabled.
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
nvtabular/init.py 18 0 0 0 100%
nvtabular/columns/init.py 2 0 0 0 100%
nvtabular/columns/schema.py 120 24 59 9 75% 38, 51, 65, 107, 116->118, 119, 121, 124-126, 129->132, 164-166, 170, 172, 193, 196-208
nvtabular/columns/schema_io/schema_writer_base.py 8 0 2 0 100%
nvtabular/columns/schema_io/schema_writer_pbtxt.py 122 11 58 12 87% 45, 61->68, 75, 92->97, 95->97, 118->133, 121-122, 124-127, 129, 169->185, 177, 181
nvtabular/columns/selector.py 67 14 36 9 74% 50, 62-67, 90, 97, 103, 105, 114, 117, 121, 130
nvtabular/dispatch.py 339 188 164 18 40% 37-39, 42-46, 51-53, 59-69, 76-77, 101-112, 116-120, 128-130, 135-138, 142-147, 154, 173, 184, 190, 195->197, 200->202, 207-210, 223-226, 231-234, 239-250, 265-266, 274, 278-280, 285-288, 294, 303, 311, 318, 346-362, 371-377, 382, 393-403, 408-411, 415-418, 422-431, 435-437, 441-450, 455-481, 516, 521, 537-540, 545, 547, 553-555, 568-589, 593-599, 603-605
nvtabular/framework_utils/init.py 0 0 0 0 100%
nvtabular/framework_utils/tensorflow/init.py 1 0 0 0 100%
nvtabular/framework_utils/tensorflow/feature_column_utils.py 134 125 90 0 4% 28-32, 69-286
nvtabular/framework_utils/tensorflow/layers/init.py 4 0 0 0 100%
nvtabular/framework_utils/tensorflow/layers/embedding.py 153 130 89 0 11% 27, 31-32, 38-39, 48-75, 79-93, 103-115, 119-124, 173-185, 188-211, 214-228, 231-239, 242, 249-265, 307-311, 314-344, 347-360, 363-364, 367
nvtabular/framework_utils/tensorflow/layers/interaction.py 47 39 22 0 14% 48-52, 55-71, 74-103, 106-110, 113
nvtabular/framework_utils/tensorflow/layers/outer_product.py 30 24 12 0 19% 37-38, 41-60, 71-84, 87
nvtabular/framework_utils/tensorflow/tfrecords_to_parquet.py 58 58 30 0 0% 16-111
nvtabular/framework_utils/torch/init.py 0 0 0 0 100%
nvtabular/framework_utils/torch/layers/init.py 2 0 0 0 100%
nvtabular/framework_utils/torch/layers/embeddings.py 32 25 18 0 22% 37-46, 49-54, 74-82, 85-95
nvtabular/framework_utils/torch/models.py 45 40 30 0 9% 52-82, 85-110
nvtabular/framework_utils/torch/utils.py 75 66 34 0 12% 22, 25-33, 42-44, 47-55, 58-76, 106-140
nvtabular/inference/init.py 0 0 0 0 100%
nvtabular/inference/triton/init.py 36 23 14 0 26% 37-56, 60-63, 67, 71, 75-81
nvtabular/inference/triton/benchmarking_tools.py 52 52 10 0 0% 2-103
nvtabular/inference/triton/data_conversions.py 87 73 58 0 10% 32-33, 52-84, 88-94, 98-105, 109-115, 119-136, 140-150
nvtabular/inference/triton/ensemble.py 303 271 112 0 8% 66-97, 137-170, 191-214, 259-303, 307-338, 355-382, 395-403, 419-465, 480-517, 545-585, 591-607, 611-678, 682-708, 712-723, 727-737, 741-742, 748-773, 779-798, 804-828, 833-836
nvtabular/inference/triton/model.py 24 24 6 0 0% 27-66
nvtabular/inference/triton/model_config_pb2.py 299 0 2 0 100%
nvtabular/inference/triton/model_pt.py 101 101 42 0 0% 27-220
nvtabular/inference/triton/runners/init.py 0 0 0 0 100%
nvtabular/inference/triton/runners/base.py 104 104 64 0 0% 27-210
nvtabular/inference/triton/runners/hugectr.py 40 40 16 0 0% 27-90
nvtabular/inference/triton/runners/pytorch.py 15 15 6 0 0% 27-54
nvtabular/inference/triton/runners/tensorflow.py 31 31 10 0 0% 27-73
nvtabular/io/init.py 4 0 0 0 100%
nvtabular/io/avro.py 88 88 32 0 0% 16-189
nvtabular/io/csv.py 57 6 22 5 86% 22-23, 99, 103->107, 108, 110, 124
nvtabular/io/dask.py 183 9 74 12 92% 111, 114, 150, 226, 401, 411, 428->431, 439, 443->445, 445->441, 450, 452
nvtabular/io/dataframe_engine.py 61 5 30 6 88% 19-20, 50, 69, 88->92, 92->97, 94->97, 97->116, 125
nvtabular/io/dataset.py 364 45 178 29 85% 48-49, 268, 270, 283, 308-322, 446->520, 451-454, 459->469, 476->474, 477->481, 494->498, 509, 520->529, 580-581, 582->586, 634, 762, 764, 766, 772, 776-778, 780, 840-841, 868, 875-876, 882, 888, 985-986, 1012, 1104-1109, 1115, 1127-1128, 1216
nvtabular/io/dataset_engine.py 31 2 6 1 92% 48, 74
nvtabular/io/fsspec_utils.py 115 101 64 0 8% 26-27, 42-98, 103-114, 151-198, 220-270, 275-291, 295-297, 311-322
nvtabular/io/hugectr.py 45 2 26 2 92% 34, 74->97, 101
nvtabular/io/parquet.py 590 52 218 34 87% 34-35, 58, 80->156, 87, 101, 113-127, 140-153, 176, 205-206, 223->248, 234->248, 242->248, 279->295, 285-293, 313, 319, 337->339, 343->345, 351-353, 371->381, 374, 427, 435, 549-554, 592-597, 713->720, 781->786, 787-788, 908, 912, 916, 922, 954, 971, 975, 982->984, 1092->exit, 1096->1093, 1103->1108, 1113->1123, 1128, 1150, 1177, 1181
nvtabular/io/shuffle.py 31 7 18 4 73% 42, 44-45, 49, 62-64
nvtabular/io/writer.py 184 13 78 5 92% 24-25, 51, 79, 125, 128, 212, 221, 224, 267, 299-301
nvtabular/io/writer_factory.py 18 2 8 2 85% 35, 60
nvtabular/loader/init.py 0 0 0 0 100%
nvtabular/loader/backend.py 365 297 152 0 14% 27-28, 45, 68-75, 78, 82, 86, 89, 93-100, 108-120, 124-149, 153-161, 165-169, 172, 176-183, 209-242, 246-251, 255-258, 261-265, 268-271, 274-277, 281-283, 288-296, 300-309, 313-318, 321-338, 341, 344-348, 363-390, 394-474, 482-485, 492-501, 532-538, 547-586, 590-617
nvtabular/loader/tensorflow.py 169 129 58 0 19% 41-72, 76-98, 234-266, 274-275, 283, 291-293, 305-309, 312, 319, 323, 327, 330-339, 342-344, 347-379, 388-398, 402, 406-414, 417-420, 423-427, 430-435, 443-444, 447-460
nvtabular/loader/tf_utils.py 57 27 22 6 46% 32->35, 35->37, 42->44, 46, 47->68, 53-54, 62-64, 70-74, 89-94, 104-117
nvtabular/loader/torch.py 87 56 26 0 33% 28-30, 33-39, 90, 110, 113-115, 118-120, 123-131, 134-136, 139, 142, 146, 150, 154-166, 169, 174-179, 182-187, 190-191, 202, 205
nvtabular/ops/init.py 23 0 0 0 100%
nvtabular/ops/add_metadata.py 17 1 4 1 90% 45
nvtabular/ops/bucketize.py 37 19 20 2 39% 53-55, 59->exit, 60-65, 72-88, 91, 94
nvtabular/ops/categorify.py 638 175 344 78 68% 244, 246, 263, 267, 271, 275, 279, 283, 285, 289, 297, 312, 315-320, 331-332, 352-356, 359, 362-387, 412, 422, 443, 446, 466-467, 473-476, 491-496, 563-593, 598->600, 601, 602->604, 608, 610, 619, 695, 697->700, 703, 720, 729-734, 765, 799, 830->833, 834-836, 843-844, 859-863, 864->828, 882, 890, 897-898, 915-916, 921, 924->927, 953, 972->974, 977, 979-982, 987, 990, 992->995, 1000->969, 1008-1015, 1016->1018, 1020-1023, 1035, 1039, 1043, 1050, 1055-1058, 1136, 1138, 1200, 1208->1230, 1214->1230, 1231-1236, 1250, 1253-1261, 1264, 1275-1283, 1288, 1293->1298, 1297, 1306-1311, 1312->1304, 1319, 1322, 1327-1341, 1351->1356, 1362-1370
nvtabular/ops/clip.py 18 2 8 3 81% 44, 52->54, 55
nvtabular/ops/column_similarity.py 121 86 40 0 23% 19-20, 29-30, 73-79, 82-89, 93-115, 120-124, 127-129, 132, 135, 138, 164-193, 202-203, 212-214, 222-238, 247-272, 276-279, 283-284
nvtabular/ops/data_stats.py 56 40 24 0 22% 28-32, 35, 39-77, 80-99, 102
nvtabular/ops/difference_lag.py 33 14 12 1 49% 60->63, 70-79, 84, 87, 92, 95, 98
nvtabular/ops/dropna.py 8 3 2 0 70% 39-41
nvtabular/ops/fill.py 91 49 40 4 40% 53-55, 63-67, 73, 75-80, 86-87, 91-94, 114-116, 120-127, 132-133, 137-140, 147, 150-159, 162-165
nvtabular/ops/filter.py 20 3 8 3 79% 49, 56, 60
nvtabular/ops/groupby.py 128 84 82 6 28% 72, 82-83, 92, 93->95, 95->84, 105-123, 129-134, 137, 140-179, 184-189, 195-209, 216-219, 224-227, 233, 240, 246-253, 259-266
nvtabular/ops/hash_bucket.py 41 22 22 2 37% 68, 72, 81-92, 97-102, 105-112, 115, 118
nvtabular/ops/hashed_cross.py 36 19 17 1 38% 53, 59-70, 75, 78, 81, 86-91
nvtabular/ops/internal/init.py 5 0 0 0 100%
nvtabular/ops/internal/concat_columns.py 11 1 2 0 92% 50
nvtabular/ops/internal/identity.py 6 1 2 0 88% 42
nvtabular/ops/internal/selection.py 17 0 2 0 100%
nvtabular/ops/internal/subset_columns.py 13 4 2 0 73% 25, 46, 49, 53
nvtabular/ops/internal/subtraction.py 10 3 2 0 75% 25, 28-29
nvtabular/ops/join_external.py 92 75 38 0 15% 20-21, 101-118, 123-179, 182-186, 189-197, 202-207, 211-212, 216-227
nvtabular/ops/join_groupby.py 101 17 38 7 78% 106, 108, 115, 121-124, 131-134, 139-141, 205, 215-216, 219-220
nvtabular/ops/lambdaop.py 39 28 20 0 22% 57-65, 69-78, 83, 88-103
nvtabular/ops/list_slice.py 66 43 28 2 29% 21-22, 53-54, 56->exit, 61-92, 95, 104-118, 126-137
nvtabular/ops/logop.py 19 2 6 1 88% 48-49
nvtabular/ops/moments.py 69 1 24 1 98% 71
nvtabular/ops/normalize.py 88 30 22 3 63% 70, 75, 80, 87, 95-96, 102, 122-124, 130-138, 144, 151-155, 158-159, 163, 171, 174
nvtabular/ops/operator.py 66 8 18 2 88% 79-84, 111, 184, 189, 196
nvtabular/ops/rename.py 41 28 24 1 25% 47, 54-56, 61-77, 80-90
nvtabular/ops/stat_operator.py 8 0 2 0 100%
nvtabular/ops/target_encoding.py 154 110 68 0 21% 166-207, 210-213, 216, 219-230, 233-234, 237-238, 241-245, 249-328, 332-360, 363, 373-382
nvtabular/ops/value_counts.py 30 17 6 0 42% 37-53, 56, 59, 62
nvtabular/tags.py 16 0 2 0 100%
nvtabular/tools/init.py 0 0 0 0 100%
nvtabular/tools/data_gen.py 251 207 86 0 17% 25-26, 47-49, 52, 57, 60-70, 73, 83-85, 92-101, 111-150, 153-166, 171-179, 182-189, 197-213, 222-252, 256-282, 286-304, 307-316, 320-326, 332-343, 346-349, 357-359, 375-381, 399-407, 412-414, 452-466
nvtabular/tools/dataset_inspector.py 50 39 22 0 21% 30-39, 48, 68-109
nvtabular/tools/inspector_script.py 46 46 0 0 0% 17-168
nvtabular/utils.py 106 43 48 8 54% 31-32, 36-37, 50, 61-62, 64-66, 69, 72, 78, 84, 90-126, 145, 149->153
nvtabular/worker.py 80 5 38 7 90% 24-25, 81->97, 89, 90->97, 97->100, 106, 108, 109->111
nvtabular/workflow/init.py 2 0 0 0 100%
nvtabular/workflow/node.py 279 117 155 19 55% 48, 65, 75-83, 88->91, 106, 115-123, 126-128, 146, 164, 167, 176->179, 214, 235-251, 254-273, 288-292, 316, 330, 339, 347-354, 358-369, 373, 381, 390-398, 406-409, 421-425, 431, 432->415, 438-441, 445, 450-475, 479-484, 504
nvtabular/workflow/workflow.py 224 40 114 12 82% 28-29, 47, 139, 179->182, 198, 204->218, 230-232, 247, 332-371, 378-379, 417, 455->454, 502, 512-514, 516-518, 531
TOTAL 7954 3701 3388 318 49%
Coverage XML written to file coverage.xml
FAIL Required test coverage of 70% not reached. Total coverage: 49.26%
=========================== short test summary info ============================
SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:16: could not import 's3fs': No module named 's3fs'
SKIPPED [8] tests/unit/test_io.py:583: could not import 'uavro': No module named 'uavro'
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
====== 1 failed, 309 passed, 9 skipped, 235 warnings in 268.56s (0:04:28) ======
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[nvtabular_tests] $ /bin/bash /tmp/jenkins8251516016487246759.sh
@bschifferer Just tried testing the Training-with-TF-multiGPU notebook and I think in line 29 of tf_trainer.py the path for schema is wrong which results in
schema = nvt.Schema().load(os.path.join(BASE_DIR + "train/schema.pbtxt"))
<stderr>:FileNotFoundError: [Errno 2] No such file or directory: '/raid/data/criteo/test_dask/outputtrain/schema.pbtxt'
I believe it should be schema = nvt.Schema().load(os.path.join(BASE_DIR, "train/schema.pbtxt"))
Furthermore, if we fix the schema path and try to run again we get different error:
[1,1]<stderr>:Traceback (most recent call last):
[1,1]<stderr>: File "tf_trainer.py", line 81, in <module>
[1,1]<stderr>: cont_names=NUMERIC_COLUMNS,
[1,1]<stderr>:NameError: name 'NUMERIC_COLUMNS' is not defined
[1,0]<stderr>:Traceback (most recent call last):
[1,0]<stderr>: File "tf_trainer.py", line 81, in <module>
[1,0]<stderr>: cont_names=NUMERIC_COLUMNS,
[1,0]<stderr>:NameError: name 'NUMERIC_COLUMNS' is not defined
--------------------------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[17396,1],1]
Exit code: 1
--------------------------------------------------------------------------
@bschifferer Also tried 03-Training-with-TF.ipynb but the single-GPU TF model training in model.fit hangs for some reason. I see 0% GPU utilization in nvidia-smi
I'm testing your pull request inside the Merlin-TF 21.12 NGC container on 16GB V100 DGX Station. Here's my docker run
docker run --gpus='"device=0,1"' -v /raid/kshitizg:/raid -w /nvtabular --rm -it -p 8321:8321 --ipc=host nvcr.io/nvidia/merlin/merlin-tensorflow-training:21.12 /bin/bash
Click to view CI Results
GitHub pull request #1278 of commit 08e8abd54b9bd90947dcd865ab6d5a7a57d69d97, no merge conflicts.
Running as SYSTEM
Setting status of 08e8abd54b9bd90947dcd865ab6d5a7a57d69d97 to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/3961/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
> git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1278/*:refs/remotes/origin/pr/1278/* # timeout=10
> git rev-parse 08e8abd54b9bd90947dcd865ab6d5a7a57d69d97^{commit} # timeout=10
Checking out Revision 08e8abd54b9bd90947dcd865ab6d5a7a57d69d97 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 08e8abd54b9bd90947dcd865ab6d5a7a57d69d97 # timeout=10
Commit message: "fix multi-GPU"
> git rev-list --no-walk 20d8fc11fea662e9adf11fc57edd70ecfe818833 # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins8269390701844410600.sh
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in /var/jenkins_home/.local/lib/python3.8/site-packages (21.3.1)
Requirement already satisfied: setuptools in /var/jenkins_home/.local/lib/python3.8/site-packages (59.6.0)
Requirement already satisfied: wheel in /var/jenkins_home/.local/lib/python3.8/site-packages (0.37.0)
Requirement already satisfied: pybind11 in /var/jenkins_home/.local/lib/python3.8/site-packages (2.8.1)
Requirement already satisfied: numpy==1.20.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (1.20.3)
running develop
running egg_info
creating nvtabular.egg-info
writing nvtabular.egg-info/PKG-INFO
writing dependency_links to nvtabular.egg-info/dependency_links.txt
writing requirements to nvtabular.egg-info/requires.txt
writing top-level names to nvtabular.egg-info/top_level.txt
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
warning: no files found matching '*.h' under directory 'cpp'
warning: no files found matching '*.cu' under directory 'cpp'
warning: no files found matching '*.cuh' under directory 'cpp'
adding license file 'LICENSE'
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
running build_ext
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c flagcheck.cpp -o flagcheck.o -std=c++17
building 'nvtabular_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/cpp
creating build/temp.linux-x86_64-3.8/cpp/nvtabular
creating build/temp.linux-x86_64-3.8/cpp/nvtabular/inference
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+43.g08e8abd -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+43.g08e8abd -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+43.g08e8abd -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/categorify.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.7.1+43.g08e8abd -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/fill.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -std=c++17 -fvisibility=hidden -g0
creating build/lib.linux-x86_64-3.8
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -o build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so ->
Generating nvtabular/inference/triton/model_config_pb2.py from nvtabular/inference/triton/model_config.proto
Creating /var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link (link to .)
nvtabular 0.7.1+43.g08e8abd is already the active version in easy-install.pth
Installed /var/jenkins_home/workspace/nvtabular_tests/nvtabular
Processing dependencies for nvtabular==0.7.1+43.g08e8abd
Searching for protobuf==3.19.1
Best match: protobuf 3.19.1
Adding protobuf 3.19.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for tensorflow-metadata==1.5.0
Best match: tensorflow-metadata 1.5.0
Processing tensorflow_metadata-1.5.0-py3.8.egg
tensorflow-metadata 1.5.0 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/tensorflow_metadata-1.5.0-py3.8.egg
Searching for pyarrow==4.0.1
Best match: pyarrow 4.0.1
Adding pyarrow 4.0.1 to easy-install.pth file
Installing plasma_store script to /var/jenkins_home/.local/bin
Using /usr/local/lib/python3.8/dist-packages
Searching for tqdm==4.62.3
Best match: tqdm 4.62.3
Processing tqdm-4.62.3-py3.8.egg
tqdm 4.62.3 is already the active version in easy-install.pth
Installing tqdm script to /var/jenkins_home/.local/bin
Using /usr/local/lib/python3.8/dist-packages/tqdm-4.62.3-py3.8.egg
Searching for numba==0.54.1
Best match: numba 0.54.1
Adding numba 0.54.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for pandas==1.3.5
Best match: pandas 1.3.5
Processing pandas-1.3.5-py3.8-linux-x86_64.egg
pandas 1.3.5 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/pandas-1.3.5-py3.8-linux-x86_64.egg
Searching for distributed==2021.7.1
Best match: distributed 2021.7.1
Processing distributed-2021.7.1-py3.8.egg
distributed 2021.7.1 is already the active version in easy-install.pth
Installing dask-ssh script to /var/jenkins_home/.local/bin
Installing dask-scheduler script to /var/jenkins_home/.local/bin
Installing dask-worker script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages/distributed-2021.7.1-py3.8.egg
Searching for dask==2021.7.1
Best match: dask 2021.7.1
Processing dask-2021.7.1-py3.8.egg
dask 2021.7.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/dask-2021.7.1-py3.8.egg
Searching for googleapis-common-protos==1.54.0
Best match: googleapis-common-protos 1.54.0
Adding googleapis-common-protos 1.54.0 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for absl-py==0.12.0
Best match: absl-py 0.12.0
Processing absl_py-0.12.0-py3.8.egg
absl-py 0.12.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/absl_py-0.12.0-py3.8.egg
Searching for numpy==1.20.3
Best match: numpy 1.20.3
Adding numpy 1.20.3 to easy-install.pth file
Installing f2py script to /var/jenkins_home/.local/bin
Installing f2py3 script to /var/jenkins_home/.local/bin
Installing f2py3.8 script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for llvmlite==0.37.0
Best match: llvmlite 0.37.0
Processing llvmlite-0.37.0-py3.8-linux-x86_64.egg
llvmlite 0.37.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/llvmlite-0.37.0-py3.8-linux-x86_64.egg
Searching for setuptools==59.6.0
Best match: setuptools 59.6.0
Adding setuptools 59.6.0 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for pytz==2021.3
Best match: pytz 2021.3
Adding pytz 2021.3 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for python-dateutil==2.8.2
Best match: python-dateutil 2.8.2
Adding python-dateutil 2.8.2 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for zict==2.0.0
Best match: zict 2.0.0
Processing zict-2.0.0-py3.8.egg
zict 2.0.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg
Searching for tornado==6.1
Best match: tornado 6.1
Processing tornado-6.1-py3.8-linux-x86_64.egg
tornado 6.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg
Searching for toolz==0.11.2
Best match: toolz 0.11.2
Processing toolz-0.11.2-py3.8.egg
toolz 0.11.2 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/toolz-0.11.2-py3.8.egg
Searching for tblib==1.7.0
Best match: tblib 1.7.0
Processing tblib-1.7.0-py3.8.egg
tblib 1.7.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg
Searching for sortedcontainers==2.4.0
Best match: sortedcontainers 2.4.0
Processing sortedcontainers-2.4.0-py3.8.egg
sortedcontainers 2.4.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg
Searching for PyYAML==5.4.1
Best match: PyYAML 5.4.1
Processing PyYAML-5.4.1-py3.8-linux-x86_64.egg
PyYAML 5.4.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg
Searching for psutil==5.8.0
Best match: psutil 5.8.0
Processing psutil-5.8.0-py3.8-linux-x86_64.egg
psutil 5.8.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg
Searching for msgpack==1.0.3
Best match: msgpack 1.0.3
Processing msgpack-1.0.3-py3.8-linux-x86_64.egg
msgpack 1.0.3 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/msgpack-1.0.3-py3.8-linux-x86_64.egg
Searching for cloudpickle==2.0.0
Best match: cloudpickle 2.0.0
Processing cloudpickle-2.0.0-py3.8.egg
cloudpickle 2.0.0 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/cloudpickle-2.0.0-py3.8.egg
Searching for click==8.0.3
Best match: click 8.0.3
Processing click-8.0.3-py3.8.egg
click 8.0.3 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/click-8.0.3-py3.8.egg
Searching for partd==1.2.0
Best match: partd 1.2.0
Processing partd-1.2.0-py3.8.egg
partd 1.2.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg
Searching for packaging==21.3
Best match: packaging 21.3
Adding packaging 21.3 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for fsspec==2021.11.1
Best match: fsspec 2021.11.1
Adding fsspec 2021.11.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for six==1.15.0
Best match: six 1.15.0
Adding six 1.15.0 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for HeapDict==1.0.1
Best match: HeapDict 1.0.1
Processing HeapDict-1.0.1-py3.8.egg
HeapDict 1.0.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg
Searching for locket==0.2.1
Best match: locket 0.2.1
Processing locket-0.2.1-py3.8.egg
locket 0.2.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg
Searching for pyparsing==3.0.6
Best match: pyparsing 3.0.6
Adding pyparsing 3.0.6 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Finished processing dependencies for nvtabular==0.7.1+43.g08e8abd
Running black --check
All done! ✨ 🍰 ✨
149 files would be left unchanged.
Running flake8
Running isort
Skipped 2 files
Running bandit
Running pylint
************* Module nvtabular.dispatch
nvtabular/dispatch.py:605:11: I1101: Module 'numpy.random.mtrand' has no 'RandomState' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
************* Module nvtabular.ops.categorify
nvtabular/ops/categorify.py:496:15: I1101: Module 'nvtabular_cpp' has no 'inference' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
************* Module nvtabular.ops.fill
nvtabular/ops/fill.py:67:15: I1101: Module 'nvtabular_cpp' has no 'inference' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
Running flake8-nb
Building docs
make: Entering directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.7) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
make: Leaving directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 1570 items / 1 skipped / 1569 selected
tests/unit/test_dask_nvt.py ............................................ [ 2%]
....................................................................... [ 7%]
tests/unit/test_io.py .................................................. [ 10%]
........................................................................ [ 15%]
.................ssssssss............................................... [ 19%]
........ [ 20%]
tests/unit/test_notebooks.py F
=================================== FAILURES ===================================
___________________________ test_criteo_tf_notebook ____________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-0/test_criteo_tf_notebook0')
def test_criteo_tf_notebook(tmpdir):
tor = pytest.importorskip("tensorflow") # noqa
# create a toy dataset in tmpdir, and point environment variables so the notebook
# will read from it
os.system("mkdir -p " + os.path.join(tmpdir, "converted/criteo"))
for i in range(24):
df = _get_random_criteo_data(1000)
df.to_parquet(os.path.join(tmpdir, "converted/criteo", f"day_{i}.parquet"))
os.environ["BASE_DIR"] = str(tmpdir)
def _nb_modify(line):
# Disable LocalCUDACluster
line = line.replace("client.run(_rmm_pool)", "# client.run(_rmm_pool)")
line = line.replace("if cluster is None:", "if False:")
line = line.replace("client = Client(cluster)", "# client = Client(cluster)")
line = line.replace(
"workflow = nvt.Workflow(features, client=client)", "workflow = nvt.Workflow(features)"
)
line = line.replace("client", "# client")
line = line.replace("NUM_GPUS = [0, 1, 2, 3, 4, 5, 6, 7]", "NUM_GPUS = [0]")
line = line.replace("part_size = int(part_mem_frac * device_size)", "part_size = '128MB'")
return line
_run_notebook(
tmpdir,
os.path.join(
dirname(TEST_PATH),
"examples/scaling-criteo/",
"02-ETL-with-NVTabular.ipynb",
),
# disable rmm.reinitialize, seems to be causing issues
transform=_nb_modify,
)
tests/unit/test_notebooks.py:62:
tests/unit/test_notebooks.py:307: in _run_notebook
subprocess.check_output([sys.executable, script_path])
/usr/lib/python3.8/subprocess.py:415: in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
input = None, capture_output = False, timeout = None, check = True
popenargs = (['/usr/bin/python3', '/tmp/pytest-of-jenkins/pytest-0/test_criteo_tf_notebook0/notebook.py'],)
kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f0cf2ea5fa0>
stdout = b'feature {\n name: "C1"\n type: INT\n int_domain {\n name: "C1"\n min: 0\n max: 222\n is_categorical: ...eo/day_22.parquet']\n['/tmp/pytest-of-jenkins/pytest-0/test_criteo_tf_notebook0/converted/criteo/day_23.parquet']\n'
stderr = None, retcode = 1
def run(*popenargs,
input=None, capture_output=False, timeout=None, check=False, **kwargs):
"""Run command with arguments and return a CompletedProcess instance.
The returned instance will have attributes args, returncode, stdout and
stderr. By default, stdout and stderr are not captured, and those attributes
will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.
If check is True and the exit code was non-zero, it raises a
CalledProcessError. The CalledProcessError object will have the return code
in the returncode attribute, and output & stderr attributes if those streams
were captured.
If timeout is given, and the process takes too long, a TimeoutExpired
exception will be raised.
There is an optional argument "input", allowing you to
pass bytes or a string to the subprocess's stdin. If you use this argument
you may not also use the Popen constructor's "stdin" argument, as
it will be used internally.
By default, all communication is in bytes, and therefore any "input" should
be bytes, and the stdout and stderr will be bytes. If in text mode, any
"input" should be a string, and stdout and stderr will be strings decoded
according to locale encoding, or by "encoding" if set. Text mode is
triggered by setting any of text, encoding, errors or universal_newlines.
The other arguments are the same as for the Popen constructor.
"""
if input is not None:
if kwargs.get('stdin') is not None:
raise ValueError('stdin and input arguments may not both be used.')
kwargs['stdin'] = PIPE
if capture_output:
if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
raise ValueError('stdout and stderr arguments may not be used '
'with capture_output.')
kwargs['stdout'] = PIPE
kwargs['stderr'] = PIPE
with Popen(*popenargs, **kwargs) as process:
try:
stdout, stderr = process.communicate(input, timeout=timeout)
except TimeoutExpired as exc:
process.kill()
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
raise
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
raise
retcode = process.poll()
if check and retcode:
raise CalledProcessError(retcode, process.args,
output=stdout, stderr=stderr)
E subprocess.CalledProcessError: Command '['/usr/bin/python3', '/tmp/pytest-of-jenkins/pytest-0/test_criteo_tf_notebook0/notebook.py']' returned non-zero exit status 1.
/usr/lib/python3.8/subprocess.py:516: CalledProcessError
----------------------------- Captured stderr call -----------------------------
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (1) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
Traceback (most recent call last):
File "/tmp/pytest-of-jenkins/pytest-0/test_criteo_tf_notebook0/notebook.py", line 164, in
[x.name for x in schema.select_by_tag(nvt.graph.tags.Tags.CATEGORICAL)]
AttributeError: module 'nvtabular' has no attribute 'graph'
=============================== warnings summary ===============================
tests/unit/test_dask_nvt.py: 3 warnings
tests/unit/test_io.py: 24 warnings
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (1) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/test_dask_nvt.py: 2 warnings
tests/unit/test_io.py: 36 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/workflow/workflow.py:89: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler will be used for execution. Please use the client argument to initialize a Workflow object with distributed-execution enabled.
warnings.warn(
tests/unit/test_dask_nvt.py: 2 warnings
tests/unit/test_io.py: 52 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dask.py:375: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler will be used for this write operation. Please use the client argument to initialize a Dataset and/or Workflow object with distributed-execution enabled.
warnings.warn(
tests/unit/test_io.py: 96 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/init.py:38: DeprecationWarning: ColumnGroup is deprecated, use ColumnSelector instead
warnings.warn("ColumnGroup is deprecated, use ColumnSelector instead", DeprecationWarning)
tests/unit/test_io.py: 20 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:521: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler is being used for this shuffle operation. Please use the client argument to initialize a Dataset and/or Workflow object with distributed-execution enabled.
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
nvtabular/init.py 18 0 0 0 100%
nvtabular/columns/init.py 2 0 0 0 100%
nvtabular/columns/schema.py 120 24 59 9 75% 38, 51, 65, 107, 116->118, 119, 121, 124-126, 129->132, 164-166, 170, 172, 193, 196-208
nvtabular/columns/schema_io/schema_writer_base.py 8 0 2 0 100%
nvtabular/columns/schema_io/schema_writer_pbtxt.py 122 11 58 12 87% 45, 61->68, 75, 92->97, 95->97, 118->133, 121-122, 124-127, 129, 169->185, 177, 181
nvtabular/columns/selector.py 67 14 36 9 74% 50, 62-67, 90, 97, 103, 105, 114, 117, 121, 130
nvtabular/dispatch.py 339 188 164 18 40% 37-39, 42-46, 51-53, 59-69, 76-77, 101-112, 116-120, 128-130, 135-138, 142-147, 154, 173, 184, 190, 195->197, 200->202, 207-210, 223-226, 231-234, 239-250, 265-266, 274, 278-280, 285-288, 294, 303, 311, 318, 346-362, 371-377, 382, 393-403, 408-411, 415-418, 422-431, 435-437, 441-450, 455-481, 516, 521, 537-540, 545, 547, 553-555, 568-589, 593-599, 603-605
nvtabular/framework_utils/init.py 0 0 0 0 100%
nvtabular/framework_utils/tensorflow/init.py 1 0 0 0 100%
nvtabular/framework_utils/tensorflow/feature_column_utils.py 134 125 90 0 4% 28-32, 69-286
nvtabular/framework_utils/tensorflow/layers/init.py 4 0 0 0 100%
nvtabular/framework_utils/tensorflow/layers/embedding.py 153 130 89 0 11% 27, 31-32, 38-39, 48-75, 79-93, 103-115, 119-124, 173-185, 188-211, 214-228, 231-239, 242, 249-265, 307-311, 314-344, 347-360, 363-364, 367
nvtabular/framework_utils/tensorflow/layers/interaction.py 47 39 22 0 14% 48-52, 55-71, 74-103, 106-110, 113
nvtabular/framework_utils/tensorflow/layers/outer_product.py 30 24 12 0 19% 37-38, 41-60, 71-84, 87
nvtabular/framework_utils/tensorflow/tfrecords_to_parquet.py 58 58 30 0 0% 16-111
nvtabular/framework_utils/torch/init.py 0 0 0 0 100%
nvtabular/framework_utils/torch/layers/init.py 2 0 0 0 100%
nvtabular/framework_utils/torch/layers/embeddings.py 32 25 18 0 22% 37-46, 49-54, 74-82, 85-95
nvtabular/framework_utils/torch/models.py 45 40 30 0 9% 52-82, 85-110
nvtabular/framework_utils/torch/utils.py 75 66 34 0 12% 22, 25-33, 42-44, 47-55, 58-76, 106-140
nvtabular/inference/init.py 0 0 0 0 100%
nvtabular/inference/triton/init.py 36 23 14 0 26% 37-56, 60-63, 67, 71, 75-81
nvtabular/inference/triton/benchmarking_tools.py 52 52 10 0 0% 2-103
nvtabular/inference/triton/data_conversions.py 87 73 58 0 10% 32-33, 52-84, 88-94, 98-105, 109-115, 119-136, 140-150
nvtabular/inference/triton/ensemble.py 303 271 112 0 8% 66-97, 137-170, 191-214, 259-303, 307-338, 355-382, 395-403, 419-465, 480-517, 545-585, 591-607, 611-678, 682-708, 712-723, 727-737, 741-742, 748-773, 779-798, 804-828, 833-836
nvtabular/inference/triton/model.py 24 24 6 0 0% 27-66
nvtabular/inference/triton/model_config_pb2.py 299 0 2 0 100%
nvtabular/inference/triton/model_pt.py 101 101 42 0 0% 27-220
nvtabular/inference/triton/runners/init.py 0 0 0 0 100%
nvtabular/inference/triton/runners/base.py 104 104 64 0 0% 27-210
nvtabular/inference/triton/runners/hugectr.py 40 40 16 0 0% 27-90
nvtabular/inference/triton/runners/pytorch.py 15 15 6 0 0% 27-54
nvtabular/inference/triton/runners/tensorflow.py 31 31 10 0 0% 27-73
nvtabular/io/init.py 4 0 0 0 100%
nvtabular/io/avro.py 88 88 32 0 0% 16-189
nvtabular/io/csv.py 57 6 22 5 86% 22-23, 99, 103->107, 108, 110, 124
nvtabular/io/dask.py 183 9 74 12 92% 111, 114, 150, 226, 401, 411, 428->431, 439, 443->445, 445->441, 450, 452
nvtabular/io/dataframe_engine.py 61 5 30 6 88% 19-20, 50, 69, 88->92, 92->97, 94->97, 97->116, 125
nvtabular/io/dataset.py 364 45 178 29 85% 48-49, 268, 270, 283, 308-322, 446->520, 451-454, 459->469, 476->474, 477->481, 494->498, 509, 520->529, 580-581, 582->586, 634, 762, 764, 766, 772, 776-778, 780, 840-841, 868, 875-876, 882, 888, 985-986, 1012, 1104-1109, 1115, 1127-1128, 1216
nvtabular/io/dataset_engine.py 31 2 6 1 92% 48, 74
nvtabular/io/fsspec_utils.py 115 101 64 0 8% 26-27, 42-98, 103-114, 151-198, 220-270, 275-291, 295-297, 311-322
nvtabular/io/hugectr.py 45 2 26 2 92% 34, 74->97, 101
nvtabular/io/parquet.py 590 52 218 34 87% 34-35, 58, 80->156, 87, 101, 113-127, 140-153, 176, 205-206, 223->248, 234->248, 242->248, 279->295, 285-293, 313, 319, 337->339, 343->345, 351-353, 371->381, 374, 427, 435, 549-554, 592-597, 713->720, 781->786, 787-788, 908, 912, 916, 922, 954, 971, 975, 982->984, 1092->exit, 1096->1093, 1103->1108, 1113->1123, 1128, 1150, 1177, 1181
nvtabular/io/shuffle.py 31 7 18 4 73% 42, 44-45, 49, 62-64
nvtabular/io/writer.py 184 13 78 5 92% 24-25, 51, 79, 125, 128, 212, 221, 224, 267, 299-301
nvtabular/io/writer_factory.py 18 2 8 2 85% 35, 60
nvtabular/loader/init.py 0 0 0 0 100%
nvtabular/loader/backend.py 365 297 152 0 14% 27-28, 45, 68-75, 78, 82, 86, 89, 93-100, 108-120, 124-149, 153-161, 165-169, 172, 176-183, 209-242, 246-251, 255-258, 261-265, 268-271, 274-277, 281-283, 288-296, 300-309, 313-318, 321-338, 341, 344-348, 363-390, 394-474, 482-485, 492-501, 532-538, 547-586, 590-617
nvtabular/loader/tensorflow.py 169 129 58 0 19% 41-72, 76-98, 234-266, 274-275, 283, 291-293, 305-309, 312, 319, 323, 327, 330-339, 342-344, 347-379, 388-398, 402, 406-414, 417-420, 423-427, 430-435, 443-444, 447-460
nvtabular/loader/tf_utils.py 57 27 22 6 46% 32->35, 35->37, 42->44, 46, 47->68, 53-54, 62-64, 70-74, 89-94, 104-117
nvtabular/loader/torch.py 87 56 26 0 33% 28-30, 33-39, 90, 110, 113-115, 118-120, 123-131, 134-136, 139, 142, 146, 150, 154-166, 169, 174-179, 182-187, 190-191, 202, 205
nvtabular/ops/init.py 23 0 0 0 100%
nvtabular/ops/add_metadata.py 17 1 4 1 90% 45
nvtabular/ops/bucketize.py 37 19 20 2 39% 53-55, 59->exit, 60-65, 72-88, 91, 94
nvtabular/ops/categorify.py 638 175 344 78 68% 244, 246, 263, 267, 271, 275, 279, 283, 285, 289, 297, 312, 315-320, 331-332, 352-356, 359, 362-387, 412, 422, 443, 446, 466-467, 473-476, 491-496, 563-593, 598->600, 601, 602->604, 608, 610, 619, 695, 697->700, 703, 720, 729-734, 765, 799, 830->833, 834-836, 843-844, 859-863, 864->828, 882, 890, 897-898, 915-916, 921, 924->927, 953, 972->974, 977, 979-982, 987, 990, 992->995, 1000->969, 1008-1015, 1016->1018, 1020-1023, 1035, 1039, 1043, 1050, 1055-1058, 1136, 1138, 1200, 1208->1230, 1214->1230, 1231-1236, 1250, 1253-1261, 1264, 1275-1283, 1288, 1293->1298, 1297, 1306-1311, 1312->1304, 1319, 1322, 1327-1341, 1351->1356, 1362-1370
nvtabular/ops/clip.py 18 2 8 3 81% 44, 52->54, 55
nvtabular/ops/column_similarity.py 121 86 40 0 23% 19-20, 29-30, 73-79, 82-89, 93-115, 120-124, 127-129, 132, 135, 138, 164-193, 202-203, 212-214, 222-238, 247-272, 276-279, 283-284
nvtabular/ops/data_stats.py 56 40 24 0 22% 28-32, 35, 39-77, 80-99, 102
nvtabular/ops/difference_lag.py 33 14 12 1 49% 60->63, 70-79, 84, 87, 92, 95, 98
nvtabular/ops/dropna.py 8 3 2 0 70% 39-41
nvtabular/ops/fill.py 91 49 40 4 40% 53-55, 63-67, 73, 75-80, 86-87, 91-94, 114-116, 120-127, 132-133, 137-140, 147, 150-159, 162-165
nvtabular/ops/filter.py 20 3 8 3 79% 49, 56, 60
nvtabular/ops/groupby.py 128 84 82 6 28% 72, 82-83, 92, 93->95, 95->84, 105-123, 129-134, 137, 140-179, 184-189, 195-209, 216-219, 224-227, 233, 240, 246-253, 259-266
nvtabular/ops/hash_bucket.py 41 22 22 2 37% 68, 72, 81-92, 97-102, 105-112, 115, 118
nvtabular/ops/hashed_cross.py 36 19 17 1 38% 53, 59-70, 75, 78, 81, 86-91
nvtabular/ops/internal/init.py 5 0 0 0 100%
nvtabular/ops/internal/concat_columns.py 11 1 2 0 92% 50
nvtabular/ops/internal/identity.py 6 1 2 0 88% 42
nvtabular/ops/internal/selection.py 17 0 2 0 100%
nvtabular/ops/internal/subset_columns.py 13 4 2 0 73% 25, 46, 49, 53
nvtabular/ops/internal/subtraction.py 10 3 2 0 75% 25, 28-29
nvtabular/ops/join_external.py 92 75 38 0 15% 20-21, 101-118, 123-179, 182-186, 189-197, 202-207, 211-212, 216-227
nvtabular/ops/join_groupby.py 101 17 38 7 78% 106, 108, 115, 121-124, 131-134, 139-141, 205, 215-216, 219-220
nvtabular/ops/lambdaop.py 39 28 20 0 22% 57-65, 69-78, 83, 88-103
nvtabular/ops/list_slice.py 66 43 28 2 29% 21-22, 53-54, 56->exit, 61-92, 95, 104-118, 126-137
nvtabular/ops/logop.py 19 2 6 1 88% 48-49
nvtabular/ops/moments.py 69 1 24 1 98% 71
nvtabular/ops/normalize.py 88 30 22 3 63% 70, 75, 80, 87, 95-96, 102, 122-124, 130-138, 144, 151-155, 158-159, 163, 171, 174
nvtabular/ops/operator.py 66 8 18 2 88% 79-84, 111, 184, 189, 196
nvtabular/ops/rename.py 41 28 24 1 25% 47, 54-56, 61-77, 80-90
nvtabular/ops/stat_operator.py 8 0 2 0 100%
nvtabular/ops/target_encoding.py 154 110 68 0 21% 166-207, 210-213, 216, 219-230, 233-234, 237-238, 241-245, 249-328, 332-360, 363, 373-382
nvtabular/ops/value_counts.py 30 17 6 0 42% 37-53, 56, 59, 62
nvtabular/tags.py 16 0 2 0 100%
nvtabular/tools/init.py 0 0 0 0 100%
nvtabular/tools/data_gen.py 251 207 86 0 17% 25-26, 47-49, 52, 57, 60-70, 73, 83-85, 92-101, 111-150, 153-166, 171-179, 182-189, 197-213, 222-252, 256-282, 286-304, 307-316, 320-326, 332-343, 346-349, 357-359, 375-381, 399-407, 412-414, 452-466
nvtabular/tools/dataset_inspector.py 50 39 22 0 21% 30-39, 48, 68-109
nvtabular/tools/inspector_script.py 46 46 0 0 0% 17-168
nvtabular/utils.py 106 43 48 8 54% 31-32, 36-37, 50, 61-62, 64-66, 69, 72, 78, 84, 90-126, 145, 149->153
nvtabular/worker.py 80 5 38 7 90% 24-25, 81->97, 89, 90->97, 97->100, 106, 108, 109->111
nvtabular/workflow/init.py 2 0 0 0 100%
nvtabular/workflow/node.py 279 117 155 19 55% 48, 65, 75-83, 88->91, 106, 115-123, 126-128, 146, 164, 167, 176->179, 214, 235-251, 254-273, 288-292, 316, 330, 339, 347-354, 358-369, 373, 381, 390-398, 406-409, 421-425, 431, 432->415, 438-441, 445, 450-475, 479-484, 504
nvtabular/workflow/workflow.py 224 40 114 12 82% 28-29, 47, 139, 179->182, 198, 204->218, 230-232, 247, 332-371, 378-379, 417, 455->454, 502, 512-514, 516-518, 531
TOTAL 7954 3701 3388 318 49%
Coverage XML written to file coverage.xml
FAIL Required test coverage of 70% not reached. Total coverage: 49.26%
=========================== short test summary info ============================
SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:16: could not import 's3fs': No module named 's3fs'
SKIPPED [8] tests/unit/test_io.py:583: could not import 'uavro': No module named 'uavro'
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
====== 1 failed, 309 passed, 9 skipped, 235 warnings in 265.30s (0:04:25) ======
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[nvtabular_tests] $ /bin/bash /tmp/jenkins2698687815494441116.sh
Click to view CI Results
GitHub pull request #1278 of commit a1a6d9eb1c782c5310ec2d1acff9743bdaec62bf, no merge conflicts.
Running as SYSTEM
Setting status of a1a6d9eb1c782c5310ec2d1acff9743bdaec62bf to PENDING with url http://10.20.13.93:8080/job/nvtabular_tests/4035/ and message: 'Pending'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/nvtabular_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/NVTabular.git
> git init /var/jenkins_home/workspace/nvtabular_tests/nvtabular # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config remote.origin.url https://github.com/NVIDIA-Merlin/NVTabular.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/NVTabular.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/NVTabular.git +refs/pull/1278/*:refs/remotes/origin/pr/1278/* # timeout=10
> git rev-parse a1a6d9eb1c782c5310ec2d1acff9743bdaec62bf^{commit} # timeout=10
Checking out Revision a1a6d9eb1c782c5310ec2d1acff9743bdaec62bf (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f a1a6d9eb1c782c5310ec2d1acff9743bdaec62bf # timeout=10
Commit message: "Merge branch 'main' into criteo_updates"
> git rev-list --no-walk 9420c14e5855012fff67f0e120d6ad492ca9b60b # timeout=10
First time build. Skipping changelog.
[nvtabular_tests] $ /bin/bash /tmp/jenkins6381840624308461014.sh
Installing NVTabular
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in /var/jenkins_home/.local/lib/python3.8/site-packages (21.3.1)
Requirement already satisfied: setuptools in /var/jenkins_home/.local/lib/python3.8/site-packages (59.4.0)
Collecting setuptools
Downloading setuptools-60.5.0-py3-none-any.whl (958 kB)
Requirement already satisfied: wheel in /var/jenkins_home/.local/lib/python3.8/site-packages (0.37.1)
Requirement already satisfied: pybind11 in /var/jenkins_home/.local/lib/python3.8/site-packages (2.9.0)
Requirement already satisfied: numpy==1.20.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (1.20.3)
Found existing installation: nvtabular 0.8.0+7.gb459467
Can't uninstall 'nvtabular'. No files were found to uninstall.
running develop
running egg_info
creating nvtabular.egg-info
writing nvtabular.egg-info/PKG-INFO
writing dependency_links to nvtabular.egg-info/dependency_links.txt
writing requirements to nvtabular.egg-info/requires.txt
writing top-level names to nvtabular.egg-info/top_level.txt
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/var/jenkins_home/.local/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
warning: no files found matching '*.h' under directory 'cpp'
warning: no files found matching '*.cu' under directory 'cpp'
warning: no files found matching '*.cuh' under directory 'cpp'
adding license file 'LICENSE'
writing manifest file 'nvtabular.egg-info/SOURCES.txt'
running build_ext
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c flagcheck.cpp -o flagcheck.o -std=c++17
building 'nvtabular_cpp' extension
creating build
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/cpp
creating build/temp.linux-x86_64-3.8/cpp/nvtabular
creating build/temp.linux-x86_64-3.8/cpp/nvtabular/inference
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.8.0+60.ga1a6d9e -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.8.0+60.ga1a6d9e -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/__init__.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.8.0+60.ga1a6d9e -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/categorify.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o -std=c++17 -fvisibility=hidden -g0
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DVERSION_INFO=0.8.0+60.ga1a6d9e -I./cpp/ -I/var/jenkins_home/.local/lib/python3.8/site-packages/pybind11/include -I/usr/include/python3.8 -c cpp/nvtabular/inference/fill.cc -o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -std=c++17 -fvisibility=hidden -g0
creating build/lib.linux-x86_64-3.8
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/cpp/nvtabular/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/__init__.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/categorify.o build/temp.linux-x86_64-3.8/cpp/nvtabular/inference/fill.o -o build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.8/nvtabular_cpp.cpython-38-x86_64-linux-gnu.so ->
Generating nvtabular/inference/triton/model_config_pb2.py from nvtabular/inference/triton/model_config.proto
Creating /var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular.egg-link (link to .)
nvtabular 0.8.0+60.ga1a6d9e is already the active version in easy-install.pth
Installed /var/jenkins_home/workspace/nvtabular_tests/nvtabular
Processing dependencies for nvtabular==0.8.0+60.ga1a6d9e
Searching for packaging==21.3
Best match: packaging 21.3
Adding packaging 21.3 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for protobuf==3.19.1
Best match: protobuf 3.19.1
Adding protobuf 3.19.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for tensorflow-metadata==1.5.0
Best match: tensorflow-metadata 1.5.0
Processing tensorflow_metadata-1.5.0-py3.8.egg
tensorflow-metadata 1.5.0 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/tensorflow_metadata-1.5.0-py3.8.egg
Searching for pyarrow==4.0.1
Best match: pyarrow 4.0.1
Adding pyarrow 4.0.1 to easy-install.pth file
Installing plasma_store script to /var/jenkins_home/.local/bin
Using /usr/local/lib/python3.8/dist-packages
Searching for tqdm==4.62.3
Best match: tqdm 4.62.3
Processing tqdm-4.62.3-py3.8.egg
tqdm 4.62.3 is already the active version in easy-install.pth
Installing tqdm script to /var/jenkins_home/.local/bin
Using /usr/local/lib/python3.8/dist-packages/tqdm-4.62.3-py3.8.egg
Searching for numba==0.54.1
Best match: numba 0.54.1
Adding numba 0.54.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for pandas==1.3.5
Best match: pandas 1.3.5
Processing pandas-1.3.5-py3.8-linux-x86_64.egg
pandas 1.3.5 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/pandas-1.3.5-py3.8-linux-x86_64.egg
Searching for distributed==2021.7.1
Best match: distributed 2021.7.1
Processing distributed-2021.7.1-py3.8.egg
Adding distributed 2021.7.1 to easy-install.pth file
Installing dask-ssh script to /var/jenkins_home/.local/bin
Installing dask-scheduler script to /var/jenkins_home/.local/bin
Installing dask-worker script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages/distributed-2021.7.1-py3.8.egg
Searching for dask==2021.7.1
Best match: dask 2021.7.1
Processing dask-2021.7.1-py3.8.egg
Adding dask 2021.7.1 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages/dask-2021.7.1-py3.8.egg
Searching for pyparsing==3.0.6
Best match: pyparsing 3.0.6
Adding pyparsing 3.0.6 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for googleapis-common-protos==1.54.0
Best match: googleapis-common-protos 1.54.0
Adding googleapis-common-protos 1.54.0 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for absl-py==0.12.0
Best match: absl-py 0.12.0
Processing absl_py-0.12.0-py3.8.egg
absl-py 0.12.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/absl_py-0.12.0-py3.8.egg
Searching for numpy==1.20.3
Best match: numpy 1.20.3
Adding numpy 1.20.3 to easy-install.pth file
Installing f2py script to /var/jenkins_home/.local/bin
Installing f2py3 script to /var/jenkins_home/.local/bin
Installing f2py3.8 script to /var/jenkins_home/.local/bin
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for llvmlite==0.37.0
Best match: llvmlite 0.37.0
Processing llvmlite-0.37.0-py3.8-linux-x86_64.egg
llvmlite 0.37.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/llvmlite-0.37.0-py3.8-linux-x86_64.egg
Searching for setuptools==59.7.0
Best match: setuptools 59.7.0
Adding setuptools 59.7.0 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for pytz==2021.3
Best match: pytz 2021.3
Adding pytz 2021.3 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for python-dateutil==2.8.2
Best match: python-dateutil 2.8.2
Adding python-dateutil 2.8.2 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for zict==2.0.0
Best match: zict 2.0.0
Processing zict-2.0.0-py3.8.egg
zict 2.0.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg
Searching for tornado==6.1
Best match: tornado 6.1
Processing tornado-6.1-py3.8-linux-x86_64.egg
tornado 6.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg
Searching for toolz==0.11.2
Best match: toolz 0.11.2
Processing toolz-0.11.2-py3.8.egg
toolz 0.11.2 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/toolz-0.11.2-py3.8.egg
Searching for tblib==1.7.0
Best match: tblib 1.7.0
Processing tblib-1.7.0-py3.8.egg
tblib 1.7.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg
Searching for sortedcontainers==2.4.0
Best match: sortedcontainers 2.4.0
Processing sortedcontainers-2.4.0-py3.8.egg
sortedcontainers 2.4.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg
Searching for PyYAML==5.4.1
Best match: PyYAML 5.4.1
Processing PyYAML-5.4.1-py3.8-linux-x86_64.egg
PyYAML 5.4.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg
Searching for psutil==5.8.0
Best match: psutil 5.8.0
Processing psutil-5.8.0-py3.8-linux-x86_64.egg
psutil 5.8.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg
Searching for msgpack==1.0.3
Best match: msgpack 1.0.3
Processing msgpack-1.0.3-py3.8-linux-x86_64.egg
msgpack 1.0.3 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/msgpack-1.0.3-py3.8-linux-x86_64.egg
Searching for cloudpickle==2.0.0
Best match: cloudpickle 2.0.0
Processing cloudpickle-2.0.0-py3.8.egg
cloudpickle 2.0.0 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/cloudpickle-2.0.0-py3.8.egg
Searching for click==8.0.3
Best match: click 8.0.3
Processing click-8.0.3-py3.8.egg
click 8.0.3 is already the active version in easy-install.pth
Using /usr/local/lib/python3.8/dist-packages/click-8.0.3-py3.8.egg
Searching for partd==1.2.0
Best match: partd 1.2.0
Processing partd-1.2.0-py3.8.egg
partd 1.2.0 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg
Searching for fsspec==2021.11.1
Best match: fsspec 2021.11.1
Adding fsspec 2021.11.1 to easy-install.pth file
Using /usr/local/lib/python3.8/dist-packages
Searching for six==1.15.0
Best match: six 1.15.0
Adding six 1.15.0 to easy-install.pth file
Using /var/jenkins_home/.local/lib/python3.8/site-packages
Searching for HeapDict==1.0.1
Best match: HeapDict 1.0.1
Processing HeapDict-1.0.1-py3.8.egg
HeapDict 1.0.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg
Searching for locket==0.2.1
Best match: locket 0.2.1
Processing locket-0.2.1-py3.8.egg
locket 0.2.1 is already the active version in easy-install.pth
Using /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg
Finished processing dependencies for nvtabular==0.8.0+60.ga1a6d9e
Running black --check
All done! ✨ 🍰 ✨
174 files would be left unchanged.
Running flake8
Running isort
Skipped 2 files
Running bandit
Running pylint
************* Module nvtabular.dispatch
nvtabular/dispatch.py:607:11: I1101: Module 'numpy.random.mtrand' has no 'RandomState' member, but source is unavailable. Consider adding this module to extension-pkg-allow-list if you want to perform analysis based on run-time introspection of living objects. (c-extension-no-member)
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
Running flake8-nb
Building docs
make: Entering directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.7) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
/usr/local/lib/python3.8/dist-packages/recommonmark/parser.py:75: UserWarning: Container node skipped: type=document
warn("Container node skipped: type={0}".format(mdnode.t))
make: Leaving directory '/var/jenkins_home/workspace/nvtabular_tests/nvtabular/docs'
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/nvtabular_tests/nvtabular, configfile: pyproject.toml
plugins: xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 1641 items / 3 skipped / 1638 selected
tests/unit/test_dask_nvt.py ............................................ [ 2%]
....................................................................... [ 7%]
tests/unit/test_io.py .................................................. [ 10%]
........................................................................ [ 14%]
..................ssssssss.............................................. [ 18%]
......... [ 19%]
tests/unit/test_notebooks.py ...... [ 19%]
tests/unit/test_tf4rec.py . [ 19%]
tests/unit/test_tools.py ...................... [ 21%]
tests/unit/test_triton_inference.py ................................ [ 23%]
tests/unit/framework_utils/test_tf_feature_columns.py . [ 23%]
tests/unit/framework_utils/test_tf_layers.py ........................... [ 24%]
................................................... [ 27%]
tests/unit/framework_utils/test_torch_layers.py . [ 27%]
tests/unit/graph/test_base_operator.py .... [ 28%]
tests/unit/graph/test_column_schemas.py ................................ [ 30%]
.................................................. [ 33%]
tests/unit/graph/test_column_selector.py ..................... [ 34%]
tests/unit/graph/ops/test_selection.py ... [ 34%]
tests/unit/inference/test_graph.py . [ 34%]
tests/unit/inference/test_inference_ops.py .. [ 34%]
tests/unit/inference/test_op_runner.py ... [ 35%]
tests/unit/inference/test_tensorflow_inf_op.py ... [ 35%]
tests/unit/loader/test_dataloader_backend.py ...... [ 35%]
tests/unit/loader/test_tf_dataloader.py ................................ [ 37%]
........................................s.. [ 40%]
tests/unit/loader/test_torch_dataloader.py ............................. [ 41%]
........................................................ [ 45%]
tests/unit/ops/test_categorify.py ...................................... [ 47%]
........................................................................ [ 52%]
............................... [ 53%]
tests/unit/ops/test_column_similarity.py ........................ [ 55%]
tests/unit/ops/test_fill.py ............................................ [ 58%]
........ [ 58%]
tests/unit/ops/test_groupyby.py ....... [ 58%]
tests/unit/ops/test_hash_bucket.py ......................... [ 60%]
tests/unit/ops/test_join.py ............................................ [ 63%]
........................................................................ [ 67%]
.................................. [ 69%]
tests/unit/ops/test_lambda.py .... [ 69%]
tests/unit/ops/test_normalize.py ....................................... [ 72%]
.. [ 72%]
tests/unit/ops/test_ops.py ............................................. [ 75%]
.......................... [ 76%]
tests/unit/ops/test_ops_schema.py ...................................... [ 79%]
........................................................................ [ 83%]
........................................................................ [ 87%]
....................................... [ 90%]
tests/unit/ops/test_target_encode.py ..................... [ 91%]
tests/unit/workflow/test_cpu_workflow.py ...... [ 91%]
tests/unit/workflow/test_workflow.py ................................... [ 93%]
............................................................ [ 97%]
tests/unit/workflow/test_workflow_node.py ........... [ 98%]
tests/unit/workflow/test_workflow_ops.py ... [ 98%]
tests/unit/workflow/test_workflow_schemas.py ......................... [100%]
=============================== warnings summary ===============================
tests/unit/test_dask_nvt.py: 3 warnings
tests/unit/test_io.py: 24 warnings
tests/unit/test_tf4rec.py: 1 warning
tests/unit/test_tools.py: 2 warnings
tests/unit/test_triton_inference.py: 7 warnings
tests/unit/loader/test_tf_dataloader.py: 54 warnings
tests/unit/loader/test_torch_dataloader.py: 54 warnings
tests/unit/ops/test_categorify.py: 1 warning
tests/unit/ops/test_column_similarity.py: 7 warnings
tests/unit/ops/test_fill.py: 24 warnings
tests/unit/ops/test_join.py: 1 warning
tests/unit/ops/test_normalize.py: 28 warnings
tests/unit/ops/test_ops.py: 4 warnings
tests/unit/ops/test_target_encode.py: 21 warnings
tests/unit/workflow/test_workflow.py: 30 warnings
tests/unit/workflow/test_workflow_node.py: 1 warning
tests/unit/workflow/test_workflow_schemas.py: 1 warning
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (1) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/test_dask_nvt.py: 12 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 2 files did not have enough
partitions to create 8 files.
warnings.warn(
tests/unit/test_dask_nvt.py: 2 warnings
tests/unit/test_io.py: 36 warnings
tests/unit/workflow/test_workflow.py: 44 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/workflow/workflow.py:86: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler will be used for execution. Please use the client argument to initialize a Workflow object with distributed-execution enabled.
warnings.warn(
tests/unit/test_dask_nvt.py: 2 warnings
tests/unit/test_io.py: 52 warnings
tests/unit/workflow/test_workflow.py: 35 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dask.py:375: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler will be used for this write operation. Please use the client argument to initialize a Dataset and/or Workflow object with distributed-execution enabled.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 6 files did not have enough
partitions to create 7 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 8 files did not have enough
partitions to create 9 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 8 files did not have enough
partitions to create 10 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 8 files did not have enough
partitions to create 11 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 12 files did not have enough
partitions to create 13 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 12 files did not have enough
partitions to create 14 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 12 files did not have enough
partitions to create 15 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 12 files did not have enough
partitions to create 16 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 12 files did not have enough
partitions to create 17 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 12 files did not have enough
partitions to create 18 files.
warnings.warn(
tests/unit/test_io.py::test_io_partitions_push
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 12 files did not have enough
partitions to create 19 files.
warnings.warn(
tests/unit/test_io.py: 96 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/init.py:38: DeprecationWarning: ColumnGroup is deprecated, use ColumnSelector instead
warnings.warn("ColumnGroup is deprecated, use ColumnSelector instead", DeprecationWarning)
tests/unit/test_io.py: 12 warnings
tests/unit/workflow/test_workflow.py: 36 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 2 files did not have enough
partitions to create 20 files.
warnings.warn(
tests/unit/test_io.py::test_multifile_parquet[False-Shuffle.PER_WORKER-5-0-csv]
tests/unit/test_io.py::test_multifile_parquet[False-Shuffle.PER_WORKER-5-2-csv]
tests/unit/test_io.py::test_multifile_parquet[False-None-5-0-csv]
tests/unit/test_io.py::test_multifile_parquet[False-None-5-2-csv]
tests/unit/loader/test_torch_dataloader.py::test_horovod_multigpu
tests/unit/loader/test_torch_dataloader.py::test_distributed_multigpu
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 1 files did not have enough
partitions to create 5 files.
warnings.warn(
tests/unit/test_io.py::test_to_parquet_output_files[Shuffle.PER_WORKER-4-6]
tests/unit/test_io.py::test_to_parquet_output_files[False-4-6]
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 2 files did not have enough
partitions to create 6 files.
warnings.warn(
tests/unit/test_io.py: 6 warnings
tests/unit/loader/test_tf_dataloader.py: 2 warnings
tests/unit/loader/test_torch_dataloader.py: 12 warnings
tests/unit/workflow/test_workflow.py: 9 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 1 files did not have enough
partitions to create 2 files.
warnings.warn(
tests/unit/test_io.py: 20 warnings
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:521: UserWarning: A global dask.distributed client has been detected, but the single-threaded scheduler is being used for this shuffle operation. Please use the client argument to initialize a Dataset and/or Workflow object with distributed-execution enabled.
warnings.warn(
tests/unit/test_tools.py::test_cat_rep[None-1000]
tests/unit/test_tools.py::test_cat_rep[distro1-1000]
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (3) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/test_tools.py::test_cat_rep[None-10000]
tests/unit/test_tools.py::test_cat_rep[distro1-10000]
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (30) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/loader/test_tf_dataloader.py::test_nested_list
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (2) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/loader/test_tf_dataloader.py::test_sparse_tensors[False]
tests/unit/loader/test_tf_dataloader.py::test_sparse_tensors[True]
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (25) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/loader/test_tf_dataloader.py::test_sparse_tensors[False]
tests/unit/loader/test_tf_dataloader.py::test_sparse_tensors[True]
/usr/local/lib/python3.8/dist-packages/numba/cuda/compiler.py:865: NumbaPerformanceWarning: [1mGrid size (35) < 2 * SM count (112) will likely result in GPU under utilization due to low occupancy.[0m
warn(NumbaPerformanceWarning(msg))
tests/unit/ops/test_fill.py::test_fill_missing[True-True-parquet]
tests/unit/ops/test_fill.py::test_fill_missing[True-False-parquet]
tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]
/usr/local/lib/python3.8/dist-packages/pandas-1.3.5-py3.8-linux-x86_64.egg/pandas/core/indexing.py:1732: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
self._setitem_single_block(indexer, value, name)
tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]
tests/unit/ops/test_ops.py::test_filter[parquet-0.1-True]
tests/unit/ops/test_ops.py::test_filter[parquet-0.1-False]
tests/unit/ops/test_ops.py::test_filter[parquet-0.1-False]
/var/jenkins_home/.local/lib/python3.8/site-packages/dask-2021.7.1-py3.8.egg/dask/dataframe/core.py:6778: UserWarning: Insufficient elements for head. 1 elements requested, only 0 elements available. Try passing larger npartitions to head.
warnings.warn(msg.format(n, len(r)))
tests/unit/workflow/test_cpu_workflow.py: 6 warnings
tests/unit/workflow/test_workflow.py: 24 warnings
tests/unit/workflow/test_workflow_schemas.py: 1 warning
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 1 files did not have enough
partitions to create 10 files.
warnings.warn(
tests/unit/workflow/test_workflow.py::test_parquet_output[True-Shuffle.PER_WORKER]
tests/unit/workflow/test_workflow.py::test_parquet_output[True-Shuffle.PER_PARTITION]
tests/unit/workflow/test_workflow.py::test_parquet_output[True-None]
tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-Shuffle.PER_WORKER]
tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-Shuffle.PER_PARTITION]
tests/unit/workflow/test_workflow.py::test_workflow_apply[True-True-None]
tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-Shuffle.PER_WORKER]
tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-Shuffle.PER_PARTITION]
tests/unit/workflow/test_workflow.py::test_workflow_apply[False-True-None]
/var/jenkins_home/workspace/nvtabular_tests/nvtabular/nvtabular/io/dataset.py:868: UserWarning: Only created 2 files did not have enough
partitions to create 4 files.
warnings.warn(
-- Docs: https://docs.pytest.org/en/stable/warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
examples/multi-gpu-movielens/torch_trainer.py 65 0 6 1 99% 32->36
examples/multi-gpu-movielens/torch_trainer_dist.py 63 0 2 0 100%
nvtabular/init.py 18 0 0 0 100%
nvtabular/dispatch.py 341 79 166 29 75% 37-39, 42-46, 51-53, 59-69, 76-77, 118-120, 128-130, 135-138, 142-147, 154, 173, 184, 190, 195->197, 208, 231-234, 265->267, 274, 278-280, 286, 311, 318, 349->354, 352, 355, 358->362, 397, 408-411, 416, 438, 445-448, 478, 482, 523, 547, 549, 556, 571-585, 600, 607
nvtabular/framework_utils/init.py 0 0 0 0 100%
nvtabular/framework_utils/tensorflow/init.py 1 0 0 0 100%
nvtabular/framework_utils/tensorflow/feature_column_utils.py 134 78 90 15 39% 30, 99, 103, 114-130, 140, 143-158, 162, 166-167, 173-198, 207-217, 220-227, 229->233, 234, 239-279, 282
nvtabular/framework_utils/tensorflow/layers/init.py 4 0 0 0 100%
nvtabular/framework_utils/tensorflow/layers/embedding.py 153 12 89 6 91% 60, 68->49, 122, 179, 231-239, 335->343, 357->360, 363-364, 367
nvtabular/framework_utils/tensorflow/layers/interaction.py 47 25 22 1 45% 49, 74-103, 106-110, 113
nvtabular/framework_utils/tensorflow/layers/outer_product.py 30 24 12 0 19% 37-38, 41-60, 71-84, 87
nvtabular/framework_utils/tensorflow/tfrecords_to_parquet.py 58 58 30 0 0% 16-111
nvtabular/framework_utils/torch/init.py 0 0 0 0 100%
nvtabular/framework_utils/torch/layers/init.py 2 0 0 0 100%
nvtabular/framework_utils/torch/layers/embeddings.py 32 2 18 2 92% 50, 91
nvtabular/framework_utils/torch/models.py 45 1 30 4 93% 57->61, 87->89, 93->96, 103
nvtabular/framework_utils/torch/utils.py 75 5 34 5 91% 51->53, 64, 71->76, 75, 118-120
nvtabular/graph/init.py 4 0 0 0 100%
nvtabular/graph/base_operator.py 95 2 36 4 94% 143->148, 157->161, 169->173, 227, 231
nvtabular/graph/graph.py 44 5 24 0 87% 84-89
nvtabular/graph/node.py 282 54 151 19 77% 49, 73-81, 136, 225, 235-236, 283, 301, 316-321, 326, 328, 334, 348, 358-369, 374->377, 388-396, 405, 406->401, 420-421, 429, 430->413, 436-439, 443, 470, 477-482, 501
nvtabular/graph/ops/init.py 5 0 0 0 100%
nvtabular/graph/ops/concat_columns.py 18 0 2 0 100%
nvtabular/graph/ops/identity.py 6 1 2 0 88% 41
nvtabular/graph/ops/selection.py 20 0 2 0 100%
nvtabular/graph/ops/subset_columns.py 15 1 2 0 94% 60
nvtabular/graph/ops/subtraction.py 21 2 4 0 92% 54-55
nvtabular/graph/schema.py 129 8 65 6 93% 38, 64, 156, 161->exit, 165, 178, 185, 210, 213
nvtabular/graph/schema_io/init.py 0 0 0 0 100%
nvtabular/graph/schema_io/schema_writer_base.py 8 0 2 0 100%
nvtabular/graph/schema_io/schema_writer_pbtxt.py 122 8 58 10 90% 45, 61->68, 75, 92->97, 95->97, 118->133, 124-127, 169->185, 177, 181
nvtabular/graph/selector.py 88 1 48 1 99% 158
nvtabular/graph/tags.py 16 0 2 0 100%
nvtabular/inference/init.py 0 0 0 0 100%
nvtabular/inference/graph/init.py 3 0 0 0 100%
nvtabular/inference/graph/ensemble.py 57 42 26 0 20% 39-103, 107-118
nvtabular/inference/graph/graph.py 27 4 14 2 80% 42, 50-57
nvtabular/inference/graph/node.py 15 9 4 0 42% 22-23, 26-27, 31-36
nvtabular/inference/graph/op_runner.py 21 0 8 0 100%
nvtabular/inference/graph/ops/init.py 0 0 0 0 100%
nvtabular/inference/graph/ops/operator.py 32 6 12 1 80% 13-14, 19, 36, 40, 49
nvtabular/inference/graph/ops/tensorflow.py 50 18 16 2 64% 34-47, 79-83, 92-95
nvtabular/inference/graph/ops/workflow.py 30 1 4 1 94% 49
nvtabular/inference/triton/init.py 36 12 14 1 58% 42-49, 68, 72, 76-82
nvtabular/inference/triton/benchmarking_tools.py 52 52 10 0 0% 2-103
nvtabular/inference/triton/data_conversions.py 87 3 58 4 95% 32-33, 84
nvtabular/inference/triton/ensemble.py 285 147 100 9 51% 89-93, 156-192, 236-284, 301-305, 377-385, 414-430, 483-493, 542-582, 588-604, 608-675, 682->685, 685->681, 702->701, 751, 757-776, 782-806, 813
nvtabular/inference/triton/model/init.py 0 0 0 0 100%
nvtabular/inference/triton/model/model_pt.py 101 101 42 0 0% 27-220
nvtabular/inference/triton/model_config_pb2.py 299 0 2 0 100%
nvtabular/inference/triton/workflow_model.py 52 52 22 0 0% 27-124
nvtabular/inference/workflow/init.py 0 0 0 0 100%
nvtabular/inference/workflow/base.py 114 114 62 0 0% 27-210
nvtabular/inference/workflow/hugectr.py 37 37 16 0 0% 27-87
nvtabular/inference/workflow/pytorch.py 10 10 6 0 0% 27-46
nvtabular/inference/workflow/tensorflow.py 32 32 10 0 0% 26-68
nvtabular/io/init.py 5 0 0 0 100%
nvtabular/io/avro.py 88 88 32 0 0% 16-189
nvtabular/io/csv.py 57 6 22 5 86% 22-23, 99, 103->107, 108, 110, 124
nvtabular/io/dask.py 183 8 74 11 93% 111, 114, 150, 401, 411, 428->431, 439, 443->445, 445->441, 450, 452
nvtabular/io/dataframe_engine.py 61 5 30 6 88% 19-20, 50, 69, 88->92, 92->97, 94->97, 97->116, 125
nvtabular/io/dataframe_iter.py 21 1 14 1 94% 42
nvtabular/io/dataset.py 346 43 168 28 85% 48-49, 268, 270, 283, 308-322, 446->520, 451-454, 459->469, 476->474, 477->481, 494->498, 509, 520->529, 580-581, 582->586, 634, 762, 764, 766, 772, 776-778, 780, 840-841, 875, 882-883, 889, 895, 992-993, 1111-1116, 1122, 1134-1135
nvtabular/io/dataset_engine.py 31 1 6 0 97% 48
nvtabular/io/fsspec_utils.py 115 101 64 0 8% 26-27, 42-98, 103-114, 151-198, 220-270, 275-291, 295-297, 311-322
nvtabular/io/hugectr.py 45 2 26 2 92% 34, 74->97, 101
nvtabular/io/parquet.py 591 48 218 30 88% 35-36, 59, 81->161, 92, 106, 118-132, 145-158, 181, 210-211, 228->253, 239->253, 290-298, 318, 324, 342->344, 358, 376->386, 379, 428->440, 432, 554-559, 597-602, 718->725, 786->791, 792-793, 913, 917, 921, 927, 959, 976, 980, 987->989, 1097->exit, 1101->1098, 1108->1113, 1118->1128, 1133, 1155, 1182
nvtabular/io/shuffle.py 31 7 18 4 73% 42, 44-45, 49, 62-64
nvtabular/io/writer.py 184 13 78 5 92% 24-25, 51, 79, 125, 128, 212, 221, 224, 267, 299-301
nvtabular/io/writer_factory.py 18 2 8 2 85% 35, 60
nvtabular/loader/init.py 0 0 0 0 100%
nvtabular/loader/backend.py 372 16 154 11 95% 27-28, 159-160, 300->302, 312-316, 363-364, 403->407, 404->403, 479, 483-484, 513, 589-590, 625, 633
nvtabular/loader/tensorflow.py 168 20 58 7 88% 66, 83, 97, 311, 339, 350, 365-367, 396-398, 408-416, 419-422
nvtabular/loader/tf_utils.py 57 10 22 6 80% 32->35, 35->37, 42->44, 46, 47->68, 53-54, 62-64, 70-74
nvtabular/loader/torch.py 87 14 26 3 80% 28-30, 33-39, 114, 158-159, 164
nvtabular/ops/init.py 23 0 0 0 100%
nvtabular/ops/add_metadata.py 15 0 2 0 100%
nvtabular/ops/bucketize.py 38 9 20 3 72% 52-54, 58->exit, 61-64, 83-86
nvtabular/ops/categorify.py 658 70 350 47 87% 252, 254, 272, 276, 284, 292, 294, 321, 342-343, 390->394, 398-405, 451, 459, 482-483, 560-565, 636, 732, 749, 794, 872-873, 888-892, 893->857, 911, 919, 926->exit, 950, 953->956, 1005->1003, 1065, 1070, 1091->1095, 1097->1052, 1103-1106, 1118, 1122, 1126, 1133, 1138-1141, 1219, 1221, 1291->1314, 1297->1314, 1315-1320, 1365, 1385->1390, 1389, 1395, 1398, 1406-1416
nvtabular/ops/clip.py 18 2 8 3 81% 44, 52->54, 55
nvtabular/ops/column_similarity.py 122 26 38 5 74% 19-20, 29-30, 82->exit, 112, 207-208, 217-219, 227-243, 260->263, 264, 274
nvtabular/ops/data_stats.py 56 1 24 3 95% 91->93, 95, 97->87
nvtabular/ops/difference_lag.py 39 0 14 1 98% 73->75
nvtabular/ops/dropna.py 8 0 2 0 100%
nvtabular/ops/fill.py 65 5 26 1 91% 62-66, 102
nvtabular/ops/filter.py 20 1 8 1 93% 49
nvtabular/ops/groupby.py 110 3 72 5 96% 72, 83, 93->95, 105->110, 129
nvtabular/ops/hash_bucket.py 43 1 22 2 95% 73, 110->116
nvtabular/ops/hashed_cross.py 37 4 17 3 87% 52, 65, 82, 92
nvtabular/ops/join_external.py 96 8 34 7 88% 20-21, 114, 116, 118, 150->152, 205-206, 216->227, 221
nvtabular/ops/join_groupby.py 112 5 47 6 93% 108, 115, 124, 131->130, 173->170, 176->170, 235-236
nvtabular/ops/lambdaop.py 46 6 22 6 82% 59, 63, 77, 89, 94, 103
nvtabular/ops/list_slice.py 86 29 42 0 63% 21-22, 142-156, 164-186
nvtabular/ops/logop.py 21 0 6 0 100%
nvtabular/ops/moments.py 69 0 24 0 100%
nvtabular/ops/normalize.py 93 4 22 1 94% 89, 139-140, 167
nvtabular/ops/operator.py 12 1 2 0 93% 53
nvtabular/ops/rename.py 29 3 14 3 86% 45, 70-72
nvtabular/ops/stat_operator.py 8 0 2 0 100%
nvtabular/ops/target_encoding.py 175 8 76 4 94% 167->171, 175->184, 270-271, 287-293, 384->387
nvtabular/ops/value_counts.py 34 0 6 1 98% 40->38
nvtabular/tools/init.py 0 0 0 0 100%
nvtabular/tools/data_gen.py 251 12 86 6 95% 25-26, 124-127, 137-139, 161-162, 313, 323, 349
nvtabular/tools/dataset_inspector.py 50 7 22 1 81% 32-39
nvtabular/tools/inspector_script.py 46 46 0 0 0% 17-168
nvtabular/utils.py 106 43 48 8 54% 31-32, 36-37, 50, 61-62, 64-66, 69, 72, 78, 84, 90-126, 145, 149->153
nvtabular/worker.py 80 5 38 7 90% 24-25, 81->97, 89, 90->97, 97->100, 106, 108, 109->111
nvtabular/workflow/init.py 2 0 0 0 100%
nvtabular/workflow/node.py 7 0 4 0 100%
nvtabular/workflow/workflow.py 201 15 84 10 91% 28-29, 47, 177, 183->197, 209-211, 324, 339-340, 375, 451, 467-469, 482
TOTAL 8517 1705 3523 378 78%
Coverage XML written to file coverage.xml
Required test coverage of 70% reached. Total coverage: 77.62%
=========================== short test summary info ============================
SKIPPED [1] ../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/io/tests/test_s3.py:16: could not import 's3fs': No module named 's3fs'
SKIPPED [1] tests/unit/inference/test_ensemble.py:32: could not import 'nvtabular.loader.tf_utils.configure_tensorflow': No module named 'nvtabular.loader.tf_utils.configure_tensorflow'; 'nvtabular.loader.tf_utils' is not a package
SKIPPED [1] tests/unit/inference/test_export.py:8: could not import 'nvtabular.loader.tf_utils.configure_tensorflow': No module named 'nvtabular.loader.tf_utils.configure_tensorflow'; 'nvtabular.loader.tf_utils' is not a package
SKIPPED [8] tests/unit/test_io.py:613: could not import 'uavro': No module named 'uavro'
SKIPPED [1] tests/unit/loader/test_tf_dataloader.py:531: not working correctly in ci environment
========= 1632 passed, 12 skipped, 714 warnings in 1666.10s (0:27:46) ==========
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/NVTabular/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[nvtabular_tests] $ /bin/bash /tmp/jenkins1639770589002316166.sh