NVIDIA-Merlin
Change `BatchedDataset` from Keras Sequence to Generator
Relates to #722 (issue) Alternative to #724
Goals :soccer:
- Support Tensorflow/Keras 2.10
- Improve performance of dataloader by removing redundant calls to dataloader
stopmethod
Implementation Details :construction:
- The
BatchedDatasetis currently implementing the interface of tf.keras.utils.Sequence- However this class is not a good match for the current generator implementation of BatchedDataset.
- This is because the Sequence expects
__getitem__to be able to return batches by index to support shuffling. - If we can't support shuffling like this then it seems natural to remove
Sequencefrom the implementation. This will then be supported with theGeneratorDataAdapter
Testing Details :mag:
- Adds a test to check that we can call
.fiton a model twice with the same dataset.
Click to view CI Results
GitHub pull request #727 of commit fbd2083e16db79c010aa95ff7a372c102773a00c, no merge conflicts.
Running as SYSTEM
Setting status of fbd2083e16db79c010aa95ff7a372c102773a00c to PENDING with url https://10.20.13.93:8080/job/merlin_models/1172/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/727/*:refs/remotes/origin/pr/727/* # timeout=10
> git rev-parse fbd2083e16db79c010aa95ff7a372c102773a00c^{commit} # timeout=10
Checking out Revision fbd2083e16db79c010aa95ff7a372c102773a00c (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f fbd2083e16db79c010aa95ff7a372c102773a00c # timeout=10
Commit message: "Remove Keras Sequence from `BatchedDataset`"
> git rev-list --no-walk efe460a0943f07915b2baa855cac3d57ee938128 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins12315690478309828320.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.6)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.9.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version =2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version =2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.1)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version =1.4.0; python_version jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 683 items
tests/unit/config/test_schema.py .... [ 0%]
tests/unit/datasets/test_advertising.py .s [ 0%]
tests/unit/datasets/test_ecommerce.py ..sss [ 1%]
tests/unit/datasets/test_entertainment.py ....sss. [ 2%]
tests/unit/datasets/test_social.py . [ 2%]
tests/unit/datasets/test_synthetic.py ...... [ 3%]
tests/unit/implicit/test_implicit.py . [ 3%]
tests/unit/lightfm/test_lightfm.py . [ 4%]
tests/unit/tf/test_core.py ...... [ 4%]
tests/unit/tf/test_dataset.py ................ [ 7%]
tests/unit/tf/test_public_api.py . [ 7%]
tests/unit/tf/blocks/test_cross.py ........... [ 9%]
tests/unit/tf/blocks/test_dlrm.py .......... [ 10%]
tests/unit/tf/blocks/test_interactions.py . [ 10%]
tests/unit/tf/blocks/test_mlp.py ................................. [ 15%]
tests/unit/tf/blocks/test_optimizer.py sFF.FF........................... [ 20%]
..................... [ 23%]
tests/unit/tf/blocks/retrieval/test_base.py . [ 23%]
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py .. [ 23%]
tests/unit/tf/blocks/retrieval/test_two_tower.py ........... [ 25%]
tests/unit/tf/blocks/sampling/test_cross_batch.py . [ 25%]
tests/unit/tf/blocks/sampling/test_in_batch.py . [ 25%]
tests/unit/tf/core/test_aggregation.py ......... [ 27%]
tests/unit/tf/core/test_base.py .. [ 27%]
tests/unit/tf/core/test_combinators.py s................... [ 30%]
tests/unit/tf/core/test_encoder.py . [ 30%]
tests/unit/tf/core/test_index.py ..F [ 30%]
tests/unit/tf/core/test_prediction.py .. [ 31%]
tests/unit/tf/core/test_tabular.py .... [ 31%]
tests/unit/tf/core/test_transformations.py s............................ [ 36%]
.................. [ 38%]
tests/unit/tf/data_augmentation/test_misc.py . [ 38%]
tests/unit/tf/data_augmentation/test_negative_sampling.py .......... [ 40%]
tests/unit/tf/data_augmentation/test_noise.py ..... [ 40%]
tests/unit/tf/examples/test_01_getting_started.py . [ 41%]
tests/unit/tf/examples/test_02_dataschema.py . [ 41%]
tests/unit/tf/examples/test_03_exploring_different_models.py . [ 41%]
tests/unit/tf/examples/test_04_export_ranking_models.py . [ 41%]
tests/unit/tf/examples/test_05_export_retrieval_model.py . [ 41%]
tests/unit/tf/examples/test_06_advanced_own_architecture.py . [ 41%]
tests/unit/tf/examples/test_07_train_traditional_models.py . [ 42%]
tests/unit/tf/examples/test_usecase_ecommerce_session_based.py . [ 42%]
tests/unit/tf/examples/test_usecase_pretrained_embeddings.py . [ 42%]
tests/unit/tf/inputs/test_continuous.py ..... [ 43%]
tests/unit/tf/inputs/test_embedding.py ................................. [ 47%]
..... [ 48%]
tests/unit/tf/inputs/test_tabular.py .................. [ 51%]
tests/unit/tf/layers/test_queue.py .............. [ 53%]
tests/unit/tf/losses/test_losses.py ....................... [ 56%]
tests/unit/tf/metrics/test_metrics_popularity.py ..... [ 57%]
tests/unit/tf/metrics/test_metrics_topk.py ....................... [ 60%]
tests/unit/tf/models/test_base.py s.....FFFF........ [ 63%]
tests/unit/tf/models/test_benchmark.py .. [ 63%]
tests/unit/tf/models/test_ranking.py .............................. [ 68%]
tests/unit/tf/models/test_retrieval.py ................................ [ 72%]
tests/unit/tf/prediction_tasks/test_classification.py .. [ 73%]
tests/unit/tf/prediction_tasks/test_multi_task.py ................ [ 75%]
tests/unit/tf/prediction_tasks/test_next_item.py ..... [ 76%]
tests/unit/tf/prediction_tasks/test_regression.py .. [ 76%]
tests/unit/tf/prediction_tasks/test_retrieval.py . [ 76%]
tests/unit/tf/prediction_tasks/test_sampling.py ...... [ 77%]
tests/unit/tf/predictions/test_base.py ..... [ 78%]
tests/unit/tf/predictions/test_classification.py ....... [ 79%]
tests/unit/tf/predictions/test_dot_product.py ........ [ 80%]
tests/unit/tf/predictions/test_regression.py .. [ 80%]
tests/unit/tf/predictions/test_sampling.py .... [ 81%]
tests/unit/tf/utils/test_batch.py .... [ 81%]
tests/unit/tf/utils/test_tf_utils.py ..... [ 82%]
tests/unit/torch/test_dataset.py ......... [ 83%]
tests/unit/torch/test_public_api.py . [ 84%]
tests/unit/torch/block/test_base.py .... [ 84%]
tests/unit/torch/block/test_mlp.py . [ 84%]
tests/unit/torch/features/test_continuous.py .. [ 85%]
tests/unit/torch/features/test_embedding.py .............. [ 87%]
tests/unit/torch/features/test_tabular.py .... [ 87%]
tests/unit/torch/model/test_head.py ............ [ 89%]
tests/unit/torch/model/test_model.py .. [ 89%]
tests/unit/torch/tabular/test_aggregation.py ........ [ 90%]
tests/unit/torch/tabular/test_tabular.py ... [ 91%]
tests/unit/torch/tabular/test_transformations.py ....... [ 92%]
tests/unit/utils/test_schema_utils.py ................................ [ 97%]
tests/unit/xgb/test_xgboost.py .................... [100%]
=================================== FAILURES ===================================
_________________________ test_optimizers[optimizers0] _________________________
optimizers = ('sgd', 'adam')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99799997],
[0.99799997],
[0.99799997],
[0.99799997]], dtype=float32)
b = array([[0.9983299],
[0.9983299],
[0.9983299],
[0.9983299]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99799997 0.99799997 0.99799997 0.99799997]
E not close rhs = [0.9983299 0.9983299 0.9983299 0.9983299]
E not close dif = [0.00032991 0.00032991 0.00032991 0.00032991]
E not close tol = [1.99833e-06 1.99833e-06 1.99833e-06 1.99833e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00032991
E Max relative difference: 0.00033046
E x: array([[0.998],
E [0.998],
E [0.998],
E [0.998]], dtype=float32)
E y: array([[0.99833],
E [0.99833],
E [0.99833],
E [0.99833]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 941ms/step - loss: 1729.8229 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.4000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 865.2577 - precision: 0.6000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1729.8229 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 24ms/step - loss: 1727.3494 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1729.8229 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.4000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 33ms/step - loss: 865.2577 - precision_2: 0.6000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7f99b6f47550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7f99b6f47550> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers1] _________________________
optimizers = ('rmsprop', 'sgd')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
tests/unit/tf/blocks/test_optimizer.py:95:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
b = array([[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]), array([0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455
E 0.99455 0.99455 0.99455]
E not close rhs = [0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774
E 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774]
E not close dif = [0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775
E 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775]
E not close tol = [1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06]
E dtype = float32, shape = (3, 4)
E Mismatched elements: 12 / 12 (100%)
E Max absolute difference: 0.00228775
E Max relative difference: 0.002295
E x: array([[0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
E y: array([[0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 33ms/step - loss: 1386.6677 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 936ms/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 696.8232 - precision_1: 0.4000 - recall_1: 0.5000 - binary_accuracy: 0.5000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 32ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7f99b46d51f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7f99b46d51f0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers3] _________________________
optimizers = ('adagrad', 'rmsprop')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455],
[0.99455],
[0.99455],
[0.99455]], dtype=float32)
b = array([[0.9945456],
[0.9945456],
[0.9945456],
[0.9945456]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455]
E not close rhs = [0.9945456 0.9945456 0.9945456 0.9945456]
E not close dif = [4.4107437e-06 4.4107437e-06 4.4107437e-06 4.4107437e-06]
E not close tol = [1.9945455e-06 1.9945455e-06 1.9945455e-06 1.9945455e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 4.4107437e-06
E Max relative difference: 4.4349335e-06
E x: array([[0.99455],
E [0.99455],
E [0.99455],
E [0.99455]], dtype=float32)
E y: array([[0.994546],
E [0.994546],
E [0.994546],
E [0.994546]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 956ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 32ms/step - loss: 1390.9613 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1386.6677 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1389.4580 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
_________________________ test_optimizers[optimizers4] _________________________
optimizers = (SGD(), Adagrad())
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.9982936],
[0.9982936],
[0.9982936],
[0.9982936]], dtype=float32)
b = array([[0.999],
[0.999],
[0.999],
[0.999]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.9982936 0.9982936 0.9982936 0.9982936]
E not close rhs = [0.999 0.999 0.999 0.999]
E not close dif = [0.00070643 0.00070643 0.00070643 0.00070643]
E not close tol = [1.999e-06 1.999e-06 1.999e-06 1.999e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00070643
E Max relative difference: 0.00070714
E x: array([[0.998294],
E [0.998294],
E [0.998294],
E [0.998294]], dtype=float32)
E y: array([[0.999],
E [0.999],
E [0.999],
E [0.999]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 926ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 29ms/step - loss: 696.8232 - precision: 0.4000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 31ms/step - loss: 1390.9613 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
________________________ test_topk_recommender_outputs _________________________
ecommerce_data = <merlin.io.dataset.Dataset object at 0x7f99a6b0b1f0>
batch_size = 100
def test_topk_recommender_outputs(ecommerce_data: Dataset, batch_size=100):
import numpy as np
import tensorflow as tf
import merlin.models.tf.dataset as tf_dataloader
from merlin.models.tf.core.index import IndexBlock
from merlin.models.utils.dataset import unique_rows_by_features
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
model = mm.TwoTowerModel(
ecommerce_data.schema,
query_tower=mm.MLPBlock([64]),
samplers=[mm.InBatchSampler()],
)
model.compile("adam", metrics=[mm.RecallAt(10)], run_eagerly=False)
model.fit(ecommerce_data, batch_size=batch_size, epochs=3)
eval_metrics = model.evaluate(
ecommerce_data, item_corpus=ecommerce_data, batch_size=batch_size, return_dict=True
)
# Manually compute top-k ids for a given batch
batch = mm.sample_batch(ecommerce_data, batch_size=batch_size, include_targets=False)
item_dataset = unique_rows_by_features(ecommerce_data, Tags.ITEM, Tags.ITEM_ID)
candidates_dataset_df = IndexBlock.get_candidates_dataset(
block=model.retrieval_block.item_block(), data=item_dataset, id_column="item_id"
)
item_tower_ids, item_tower_embeddings = IndexBlock.extract_ids_embeddings(
candidates_dataset_df, check_unique_ids=True
)
batch_query_tower_embeddings = model.retrieval_block.query_block()(batch)
batch_user_scores_all_items = tf.matmul(
batch_query_tower_embeddings, item_tower_embeddings, transpose_b=True
)
top_scores, top_indices = tf.math.top_k(batch_user_scores_all_items, k=10)
top_ids = tf.gather(item_tower_ids, top_indices)
# Get top-k ids from the topk_recommender_model
topk_recommender_model = model.to_top_k_recommender(ecommerce_data, k=10)
topk_predictions, topk_items = topk_recommender_model(batch)
# Assert top-k items from top-k recommender are the same as the manually computed top-k items
tf.debugging.assert_equal(top_ids, topk_items)
# Compute recall using top-k recommender
data_dl = tf_dataloader.BatchedDataset(
ecommerce_data,
batch_size=batch_size,
shuffle=False,
)
topk_output = topk_recommender_model.predict(data_dl)
topk_predictions, topk_items = topk_output
test_df = ecommerce_data.to_ddf()
positive_item_ids = np.array(test_df["item_id"].compute().values.tolist())
recall_at_10 = numpy_recall(positive_item_ids, topk_items, k=10)
tests/unit/tf/core/test_index.py:132:
labels = array([ 4, 14, 4, 5, 36, 26, 15, 16, 68, 37, 20, 7, 42,
40, 36, 26, 11, 85, 20, 33, 52,... 25, 75, 5, 7, 8, 33, 9, 13, 25, 27, 39, 17, 72,
5, 229, 11, 36, 57, 31, 25, 8, 18])
top_item_ids = array([[ 44, 85, 29, ..., 16, 31, 28],
[ 21, 11, 31, ..., 115, 29, 20],
[ 53, 16, 28, ..., 8... 28, 26, ..., 21, 24, 6],
[ 16, 11, 206, ..., 25, 67, 53],
[ 31, 29, 11, ..., 68, 206, 6]])
k = 10
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
E ValueError: operands could not be broadcast together with shapes (100,1) (200,10)
tests/unit/tf/core/test_index.py:85: ValueError
----------------------------- Captured stdout call -----------------------------
Epoch 1/3
1/Unknown - 3s 3s/step - loss: 4.5866 - recall_at_10: 0.1200 - regularization_loss: 0.0000e+00
2/2 [==============================] - 3s 31ms/step - loss: 4.5826 - recall_at_10: 0.1533 - regularization_loss: 0.0000e+00
Epoch 2/3
2/2 [==============================] - 0s 2ms/step - loss: 4.5826 - recall_at_10: 0.1700 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 21ms/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
1/Unknown - 1s 618ms/step
2/2 [==============================] - 1s 24ms/step
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
------------------------------ Captured log call -------------------------------
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING tensorflow:data_adapter.py:1232 Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
_____________________ test_train_metrics_steps[1-1-1-1-1] ______________________
num_rows = 1, batch_size = 1, train_metrics_steps = 1, expected_steps = 1
expected_metrics_steps = 1
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 2 == 1
E + where 2 = len([{'auc': 0.0, 'loss': 0.706480860710144, 'regularization_loss': 0.0}, {'auc': 0.0, 'loss': 0.6618980765342712, 'regularization_loss': 0.0}])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 399ms/step - loss: 0.7065 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/Unknown - 1s 177ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 180ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-2-6-3] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 2, expected_steps = 6
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.5, 'loss': 0.6951462626457214, 'regularization_loss': 0.0}, {'auc': 0.5, 'loss': 0.6837323904037476, 'regul... 'regularization_loss': 0.0}, {'auc': 0.5933013558387756, 'loss': 0.6777012944221497, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 372ms/step - loss: 0.6951 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/Unknown - 1s 165ms/step - loss: 0.6837 - auc: 0.5000 - regularization_loss: 0.0000e+00
3/Unknown - 1s 153ms/step - loss: 0.6914 - auc: 0.4890 - regularization_loss: 0.0000e+00
4/Unknown - 1s 151ms/step - loss: 0.6832 - auc: 0.4890 - regularization_loss: 0.0000e+00
5/Unknown - 1s 148ms/step - loss: 0.6818 - auc: 0.5933 - regularization_loss: 0.0000e+00
6/Unknown - 1s 146ms/step - loss: 0.6777 - auc: 0.5933 - regularization_loss: 0.0000e+00
7/Unknown - 1s 144ms/step - loss: 0.6838 - auc: 0.5067 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 145ms/step - loss: 0.6838 - auc: 0.5067 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-3-6-2] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 3, expected_steps = 6
expected_metrics_steps = 2
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.4523809552192688, 'loss': 0.6864265203475952, 'regularization_loss': 0.0}, {'auc': 0.4523809552192688, 'los... 'regularization_loss': 0.0}, {'auc': 0.5659340620040894, 'loss': 0.6726491451263428, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 386ms/step - loss: 0.6864 - auc: 0.4524 - regularization_loss: 0.0000e+00
2/Unknown - 1s 178ms/step - loss: 0.7028 - auc: 0.4524 - regularization_loss: 0.0000e+00
3/Unknown - 1s 161ms/step - loss: 0.6814 - auc: 0.4524 - regularization_loss: 0.0000e+00
4/Unknown - 1s 161ms/step - loss: 0.6785 - auc: 0.5659 - regularization_loss: 0.0000e+00
5/Unknown - 1s 156ms/step - loss: 0.6773 - auc: 0.5659 - regularization_loss: 0.0000e+00
6/Unknown - 1s 154ms/step - loss: 0.6726 - auc: 0.5659 - regularization_loss: 0.0000e+00
7/Unknown - 1s 155ms/step - loss: 0.6805 - auc: 0.4933 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 155ms/step - loss: 0.6805 - auc: 0.4933 - regularization_loss: 0.0000e+00
___________________ test_train_metrics_steps[120-10-4-12-3] ____________________
num_rows = 120, batch_size = 10, train_metrics_steps = 4, expected_steps = 12
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 13 == 12
E + where 13 = len([{'auc': 1.0, 'loss': 0.6796059608459473, 'regularization_loss': 0.0}, {'auc': 1.0, 'loss': 0.6844526529312134, 'regul...603546143, 'regularization_loss': 0.0}, {'auc': 0.4609375, 'loss': 0.686423659324646, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 384ms/step - loss: 0.6796 - auc: 1.0000 - regularization_loss: 0.0000e+00
2/Unknown - 1s 182ms/step - loss: 0.6845 - auc: 1.0000 - regularization_loss: 0.0000e+00
3/Unknown - 1s 162ms/step - loss: 0.6867 - auc: 1.0000 - regularization_loss: 0.0000e+00
4/Unknown - 1s 155ms/step - loss: 0.6807 - auc: 1.0000 - regularization_loss: 0.0000e+00
5/Unknown - 1s 151ms/step - loss: 0.6729 - auc: 0.4609 - regularization_loss: 0.0000e+00
6/Unknown - 1s 150ms/step - loss: 0.6864 - auc: 0.4609 - regularization_loss: 0.0000e+00
7/Unknown - 1s 151ms/step - loss: 0.6814 - auc: 0.4609 - regularization_loss: 0.0000e+00
8/Unknown - 1s 149ms/step - loss: 0.6799 - auc: 0.4609 - regularization_loss: 0.0000e+00
9/Unknown - 2s 150ms/step - loss: 0.6867 - auc: 0.3225 - regularization_loss: 0.0000e+00
10/Unknown - 2s 150ms/step - loss: 0.6841 - auc: 0.3225 - regularization_loss: 0.0000e+00
11/Unknown - 2s 150ms/step - loss: 0.6848 - auc: 0.3225 - regularization_loss: 0.0000e+00
12/Unknown - 2s 149ms/step - loss: 0.6802 - auc: 0.3225 - regularization_loss: 0.0000e+00
13/Unknown - 2s 149ms/step - loss: 0.6766 - auc: 0.3732 - regularization_loss: 0.0000e+00
13/13 [==============================] - 2s 150ms/step - loss: 0.6766 - auc: 0.3732 - regularization_loss: 0.0000e+00
=============================== warnings summary ===============================
../../../../../usr/lib/python3/dist-packages/requests/init.py:89
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 6 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 8 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/core/test_transformations.py: 13 warnings
tests/unit/tf/data_augmentation/test_negative_sampling.py: 10 warnings
tests/unit/tf/data_augmentation/test_noise.py: 1 warning
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 34 warnings
tests/unit/tf/models/test_retrieval.py: 60 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 2 warnings
tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 9 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 3 warnings
tests/unit/xgb/test_xgboost.py: 18 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 5 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 3 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/core/test_transformations.py: 10 warnings
tests/unit/tf/data_augmentation/test_negative_sampling.py: 10 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 32 warnings
tests/unit/tf/models/test_retrieval.py: 32 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 2 warnings
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 7 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 2 warnings
tests/unit/xgb/test_xgboost.py: 17 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]
tests/unit/tf/test_dataset.py::test_tf_catname_ordering
tests/unit/tf/test_dataset.py::test_tf_map
/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.
warnings.warn(
tests/unit/datasets/test_entertainment.py: 1 warning
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 1 warning
tests/unit/tf/core/test_prediction.py: 1 warning
tests/unit/tf/data_augmentation/test_negative_sampling.py: 9 warnings
tests/unit/tf/inputs/test_continuous.py: 2 warnings
tests/unit/tf/inputs/test_embedding.py: 9 warnings
tests/unit/tf/inputs/test_tabular.py: 8 warnings
tests/unit/tf/models/test_ranking.py: 16 warnings
tests/unit/tf/models/test_retrieval.py: 4 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/xgb/test_xgboost.py: 12 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:879: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack
embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/core/test_index.py: 4 warnings
tests/unit/tf/models/test_retrieval.py: 54 warnings
tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings
tests/unit/tf/predictions/test_classification.py: 12 warnings
tests/unit/tf/predictions/test_dot_product.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 2 warnings
/tmp/autograph_generated_filebu7wn5jt.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.1]
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.3]
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.5]
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.7]
tests/unit/tf/models/test_base.py::test_model_pre_post[True]
tests/unit/tf/models/test_base.py::test_model_pre_post[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.
return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]
tests/unit/tf/models/test_base.py::test_freeze_sequential_block
tests/unit/tf/models/test_base.py::test_freeze_unfreeze
tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks
/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.
super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/core/transformations.py:980: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block
/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:201.)
return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]
tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple
tests/unit/xgb/test_xgboost.py::TestEvals::test_default
tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid
tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data
/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:335: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].
warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective
/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first
self.make_current()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [4] ../../../../../usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:2746: Not a test.
===== 9 failed, 663 passed, 11 skipped, 1016 warnings in 952.78s (0:15:52) =====
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_models] $ /bin/bash /tmp/jenkins15854098892101329047.sh
Click to view CI Results
GitHub pull request #727 of commit 0a43b121a088bb81bcd7ff8937c60e87d2af0bfd, no merge conflicts.
Running as SYSTEM
Setting status of 0a43b121a088bb81bcd7ff8937c60e87d2af0bfd to PENDING with url https://10.20.13.93:8080/job/merlin_models/1186/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/727/*:refs/remotes/origin/pr/727/* # timeout=10
> git rev-parse 0a43b121a088bb81bcd7ff8937c60e87d2af0bfd^{commit} # timeout=10
Checking out Revision 0a43b121a088bb81bcd7ff8937c60e87d2af0bfd (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 0a43b121a088bb81bcd7ff8937c60e87d2af0bfd # timeout=10
Commit message: "Merge branch 'main' into batched-dataset-generator"
> git rev-list --no-walk c77e09fde46ecff065d37910dfe6f5f38758987c # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins17049972779594685435.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.6)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.9.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version =2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version =2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.1)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version =1.4.0; python_version jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 683 items
tests/unit/config/test_schema.py .... [ 0%]
tests/unit/datasets/test_advertising.py .s [ 0%]
tests/unit/datasets/test_ecommerce.py ..sss [ 1%]
tests/unit/datasets/test_entertainment.py ....sss. [ 2%]
tests/unit/datasets/test_social.py . [ 2%]
tests/unit/datasets/test_synthetic.py ...... [ 3%]
tests/unit/implicit/test_implicit.py . [ 3%]
tests/unit/lightfm/test_lightfm.py . [ 4%]
tests/unit/tf/test_core.py ...... [ 4%]
tests/unit/tf/test_dataset.py ................ [ 7%]
tests/unit/tf/test_public_api.py . [ 7%]
tests/unit/tf/blocks/test_cross.py ........... [ 9%]
tests/unit/tf/blocks/test_dlrm.py .......... [ 10%]
tests/unit/tf/blocks/test_interactions.py . [ 10%]
tests/unit/tf/blocks/test_mlp.py ................................. [ 15%]
tests/unit/tf/blocks/test_optimizer.py sFF.FF........................... [ 20%]
..................... [ 23%]
tests/unit/tf/blocks/retrieval/test_base.py . [ 23%]
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py .. [ 23%]
tests/unit/tf/blocks/retrieval/test_two_tower.py ........... [ 25%]
tests/unit/tf/blocks/sampling/test_cross_batch.py . [ 25%]
tests/unit/tf/blocks/sampling/test_in_batch.py . [ 25%]
tests/unit/tf/core/test_aggregation.py ......... [ 27%]
tests/unit/tf/core/test_base.py .. [ 27%]
tests/unit/tf/core/test_combinators.py s................... [ 30%]
tests/unit/tf/core/test_encoder.py . [ 30%]
tests/unit/tf/core/test_index.py ..F [ 30%]
tests/unit/tf/core/test_prediction.py .. [ 31%]
tests/unit/tf/core/test_tabular.py .... [ 31%]
tests/unit/tf/core/test_transformations.py s............................ [ 36%]
.................. [ 38%]
tests/unit/tf/data_augmentation/test_misc.py . [ 38%]
tests/unit/tf/data_augmentation/test_negative_sampling.py .......... [ 40%]
tests/unit/tf/data_augmentation/test_noise.py ..... [ 40%]
tests/unit/tf/examples/test_01_getting_started.py . [ 41%]
tests/unit/tf/examples/test_02_dataschema.py . [ 41%]
tests/unit/tf/examples/test_03_exploring_different_models.py . [ 41%]
tests/unit/tf/examples/test_04_export_ranking_models.py . [ 41%]
tests/unit/tf/examples/test_05_export_retrieval_model.py . [ 41%]
tests/unit/tf/examples/test_06_advanced_own_architecture.py . [ 41%]
tests/unit/tf/examples/test_07_train_traditional_models.py . [ 42%]
tests/unit/tf/examples/test_usecase_ecommerce_session_based.py . [ 42%]
tests/unit/tf/examples/test_usecase_pretrained_embeddings.py . [ 42%]
tests/unit/tf/inputs/test_continuous.py ..... [ 43%]
tests/unit/tf/inputs/test_embedding.py ................................. [ 47%]
..... [ 48%]
tests/unit/tf/inputs/test_tabular.py .................. [ 51%]
tests/unit/tf/layers/test_queue.py .............. [ 53%]
tests/unit/tf/losses/test_losses.py ....................... [ 56%]
tests/unit/tf/metrics/test_metrics_popularity.py ..... [ 57%]
tests/unit/tf/metrics/test_metrics_topk.py ....................... [ 60%]
tests/unit/tf/models/test_base.py s.....FFFF........ [ 63%]
tests/unit/tf/models/test_benchmark.py .. [ 63%]
tests/unit/tf/models/test_ranking.py .............................. [ 68%]
tests/unit/tf/models/test_retrieval.py ................................ [ 72%]
tests/unit/tf/prediction_tasks/test_classification.py .. [ 73%]
tests/unit/tf/prediction_tasks/test_multi_task.py ................ [ 75%]
tests/unit/tf/prediction_tasks/test_next_item.py ..... [ 76%]
tests/unit/tf/prediction_tasks/test_regression.py .. [ 76%]
tests/unit/tf/prediction_tasks/test_retrieval.py . [ 76%]
tests/unit/tf/prediction_tasks/test_sampling.py ...... [ 77%]
tests/unit/tf/predictions/test_base.py ..... [ 78%]
tests/unit/tf/predictions/test_classification.py ....... [ 79%]
tests/unit/tf/predictions/test_dot_product.py ........ [ 80%]
tests/unit/tf/predictions/test_regression.py .. [ 80%]
tests/unit/tf/predictions/test_sampling.py .... [ 81%]
tests/unit/tf/utils/test_batch.py .... [ 81%]
tests/unit/tf/utils/test_tf_utils.py ..... [ 82%]
tests/unit/torch/test_dataset.py ......... [ 83%]
tests/unit/torch/test_public_api.py . [ 84%]
tests/unit/torch/block/test_base.py .... [ 84%]
tests/unit/torch/block/test_mlp.py . [ 84%]
tests/unit/torch/features/test_continuous.py .. [ 85%]
tests/unit/torch/features/test_embedding.py .............. [ 87%]
tests/unit/torch/features/test_tabular.py .... [ 87%]
tests/unit/torch/model/test_head.py ............ [ 89%]
tests/unit/torch/model/test_model.py .. [ 89%]
tests/unit/torch/tabular/test_aggregation.py ........ [ 90%]
tests/unit/torch/tabular/test_tabular.py ... [ 91%]
tests/unit/torch/tabular/test_transformations.py ....... [ 92%]
tests/unit/utils/test_schema_utils.py ................................ [ 97%]
tests/unit/xgb/test_xgboost.py .................... [100%]
=================================== FAILURES ===================================
_________________________ test_optimizers[optimizers0] _________________________
optimizers = ('sgd', 'adam')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99799997],
[0.99799997],
[0.99799997],
[0.99799997]], dtype=float32)
b = array([[0.9983299],
[0.9983299],
[0.9983299],
[0.9983299]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99799997 0.99799997 0.99799997 0.99799997]
E not close rhs = [0.9983299 0.9983299 0.9983299 0.9983299]
E not close dif = [0.00032991 0.00032991 0.00032991 0.00032991]
E not close tol = [1.99833e-06 1.99833e-06 1.99833e-06 1.99833e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00032991
E Max relative difference: 0.00033046
E x: array([[0.998],
E [0.998],
E [0.998],
E [0.998]], dtype=float32)
E y: array([[0.99833],
E [0.99833],
E [0.99833],
E [0.99833]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 964ms/step - loss: 1729.8229 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.4000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 31ms/step - loss: 865.2577 - precision: 0.6000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1729.8229 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 39ms/step - loss: 1727.3494 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1729.8229 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.4000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 865.2577 - precision_2: 0.6000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7fc1b9e0c0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7fc1b9e0c0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers1] _________________________
optimizers = ('rmsprop', 'sgd')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
tests/unit/tf/blocks/test_optimizer.py:95:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
b = array([[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]), array([0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455
E 0.99455 0.99455 0.99455]
E not close rhs = [0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774
E 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774]
E not close dif = [0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775
E 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775]
E not close tol = [1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06]
E dtype = float32, shape = (3, 4)
E Mismatched elements: 12 / 12 (100%)
E Max absolute difference: 0.00228775
E Max relative difference: 0.002295
E x: array([[0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
E y: array([[0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 32ms/step - loss: 1386.6677 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 964ms/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 34ms/step - loss: 696.8232 - precision_1: 0.4000 - recall_1: 0.5000 - binary_accuracy: 0.5000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 31ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7fc1abbc94c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7fc1abbc94c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers3] _________________________
optimizers = ('adagrad', 'rmsprop')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455],
[0.99455],
[0.99455],
[0.99455]], dtype=float32)
b = array([[0.9945456],
[0.9945456],
[0.9945456],
[0.9945456]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455]
E not close rhs = [0.9945456 0.9945456 0.9945456 0.9945456]
E not close dif = [4.4107437e-06 4.4107437e-06 4.4107437e-06 4.4107437e-06]
E not close tol = [1.9945455e-06 1.9945455e-06 1.9945455e-06 1.9945455e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 4.4107437e-06
E Max relative difference: 4.4349335e-06
E x: array([[0.99455],
E [0.99455],
E [0.99455],
E [0.99455]], dtype=float32)
E y: array([[0.994546],
E [0.994546],
E [0.994546],
E [0.994546]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 958ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1390.9613 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 29ms/step - loss: 1386.6677 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 27ms/step - loss: 1389.4580 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
_________________________ test_optimizers[optimizers4] _________________________
optimizers = (SGD(), Adagrad())
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.9982936],
[0.9982936],
[0.9982936],
[0.9982936]], dtype=float32)
b = array([[0.999],
[0.999],
[0.999],
[0.999]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.9982936 0.9982936 0.9982936 0.9982936]
E not close rhs = [0.999 0.999 0.999 0.999]
E not close dif = [0.00070643 0.00070643 0.00070643 0.00070643]
E not close tol = [1.999e-06 1.999e-06 1.999e-06 1.999e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00070643
E Max relative difference: 0.00070714
E x: array([[0.998294],
E [0.998294],
E [0.998294],
E [0.998294]], dtype=float32)
E y: array([[0.999],
E [0.999],
E [0.999],
E [0.999]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 955ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 31ms/step - loss: 696.8232 - precision: 0.4000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 41ms/step - loss: 1390.9613 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 995ms/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 33ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
________________________ test_topk_recommender_outputs _________________________
ecommerce_data = <merlin.io.dataset.Dataset object at 0x7fc1aa964430>
batch_size = 100
def test_topk_recommender_outputs(ecommerce_data: Dataset, batch_size=100):
import numpy as np
import tensorflow as tf
import merlin.models.tf.dataset as tf_dataloader
from merlin.models.tf.core.index import IndexBlock
from merlin.models.utils.dataset import unique_rows_by_features
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
model = mm.TwoTowerModel(
ecommerce_data.schema,
query_tower=mm.MLPBlock([64]),
samplers=[mm.InBatchSampler()],
)
model.compile("adam", metrics=[mm.RecallAt(10)], run_eagerly=False)
model.fit(ecommerce_data, batch_size=batch_size, epochs=3)
eval_metrics = model.evaluate(
ecommerce_data, item_corpus=ecommerce_data, batch_size=batch_size, return_dict=True
)
# Manually compute top-k ids for a given batch
batch = mm.sample_batch(ecommerce_data, batch_size=batch_size, include_targets=False)
item_dataset = unique_rows_by_features(ecommerce_data, Tags.ITEM, Tags.ITEM_ID)
candidates_dataset_df = IndexBlock.get_candidates_dataset(
block=model.retrieval_block.item_block(), data=item_dataset, id_column="item_id"
)
item_tower_ids, item_tower_embeddings = IndexBlock.extract_ids_embeddings(
candidates_dataset_df, check_unique_ids=True
)
batch_query_tower_embeddings = model.retrieval_block.query_block()(batch)
batch_user_scores_all_items = tf.matmul(
batch_query_tower_embeddings, item_tower_embeddings, transpose_b=True
)
top_scores, top_indices = tf.math.top_k(batch_user_scores_all_items, k=10)
top_ids = tf.gather(item_tower_ids, top_indices)
# Get top-k ids from the topk_recommender_model
topk_recommender_model = model.to_top_k_recommender(ecommerce_data, k=10)
topk_predictions, topk_items = topk_recommender_model(batch)
# Assert top-k items from top-k recommender are the same as the manually computed top-k items
tf.debugging.assert_equal(top_ids, topk_items)
# Compute recall using top-k recommender
data_dl = tf_dataloader.BatchedDataset(
ecommerce_data,
batch_size=batch_size,
shuffle=False,
)
topk_output = topk_recommender_model.predict(data_dl)
topk_predictions, topk_items = topk_output
test_df = ecommerce_data.to_ddf()
positive_item_ids = np.array(test_df["item_id"].compute().values.tolist())
recall_at_10 = numpy_recall(positive_item_ids, topk_items, k=10)
tests/unit/tf/core/test_index.py:132:
labels = array([ 4, 14, 4, 5, 36, 26, 15, 16, 68, 37, 20, 7, 42,
40, 36, 26, 11, 85, 20, 33, 52,... 25, 75, 5, 7, 8, 33, 9, 13, 25, 27, 39, 17, 72,
5, 229, 11, 36, 57, 31, 25, 8, 18])
top_item_ids = array([[ 44, 85, 29, ..., 16, 31, 28],
[ 21, 11, 31, ..., 115, 29, 20],
[ 53, 16, 28, ..., 8... 28, 26, ..., 21, 24, 6],
[ 16, 11, 206, ..., 25, 67, 53],
[ 31, 29, 11, ..., 68, 206, 6]])
k = 10
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
E ValueError: operands could not be broadcast together with shapes (100,1) (200,10)
tests/unit/tf/core/test_index.py:85: ValueError
----------------------------- Captured stdout call -----------------------------
Epoch 1/3
1/Unknown - 3s 3s/step - loss: 4.5866 - recall_at_10: 0.1200 - regularization_loss: 0.0000e+00
2/2 [==============================] - 3s 33ms/step - loss: 4.5826 - recall_at_10: 0.1533 - regularization_loss: 0.0000e+00
Epoch 2/3
2/2 [==============================] - 0s 3ms/step - loss: 4.5826 - recall_at_10: 0.1700 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 24ms/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
1/Unknown - 0s 244ms/step
2/2 [==============================] - 0s 28ms/step
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
------------------------------ Captured log call -------------------------------
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING tensorflow:data_adapter.py:1232 Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
_____________________ test_train_metrics_steps[1-1-1-1-1] ______________________
num_rows = 1, batch_size = 1, train_metrics_steps = 1, expected_steps = 1
expected_metrics_steps = 1
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 2 == 1
E + where 2 = len([{'auc': 0.0, 'loss': 0.706480860710144, 'regularization_loss': 0.0}, {'auc': 0.0, 'loss': 0.6618980765342712, 'regularization_loss': 0.0}])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 372ms/step - loss: 0.7065 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/Unknown - 1s 166ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 168ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-2-6-3] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 2, expected_steps = 6
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.4047619104385376, 'loss': 0.6930419206619263, 'regularization_loss': 0.0}, {'auc': 0.4047619104385376, 'los... 'regularization_loss': 0.0}, {'auc': 0.4107142686843872, 'loss': 0.6909902095794678, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 373ms/step - loss: 0.6930 - auc: 0.4048 - regularization_loss: 0.0000e+00
2/Unknown - 1s 163ms/step - loss: 0.6897 - auc: 0.4048 - regularization_loss: 0.0000e+00
3/Unknown - 1s 150ms/step - loss: 0.6938 - auc: 0.3021 - regularization_loss: 0.0000e+00
4/Unknown - 1s 145ms/step - loss: 0.6937 - auc: 0.3021 - regularization_loss: 0.0000e+00
5/Unknown - 1s 142ms/step - loss: 0.6931 - auc: 0.4107 - regularization_loss: 0.0000e+00
6/Unknown - 1s 140ms/step - loss: 0.6910 - auc: 0.4107 - regularization_loss: 0.0000e+00
7/Unknown - 1s 140ms/step - loss: 0.6853 - auc: 0.5120 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 140ms/step - loss: 0.6853 - auc: 0.5120 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-3-6-2] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 3, expected_steps = 6
expected_metrics_steps = 2
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.28125, 'loss': 0.7101359367370605, 'regularization_loss': 0.0}, {'auc': 0.28125, 'loss': 0.702970027923584,... 'regularization_loss': 0.0}, {'auc': 0.3461538255214691, 'loss': 0.6953961849212646, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 370ms/step - loss: 0.7101 - auc: 0.2812 - regularization_loss: 0.0000e+00
2/Unknown - 1s 167ms/step - loss: 0.7030 - auc: 0.2812 - regularization_loss: 0.0000e+00
3/Unknown - 1s 156ms/step - loss: 0.7054 - auc: 0.2812 - regularization_loss: 0.0000e+00
4/Unknown - 1s 149ms/step - loss: 0.6996 - auc: 0.3462 - regularization_loss: 0.0000e+00
5/Unknown - 1s 145ms/step - loss: 0.6989 - auc: 0.3462 - regularization_loss: 0.0000e+00
6/Unknown - 1s 145ms/step - loss: 0.6954 - auc: 0.3462 - regularization_loss: 0.0000e+00
7/Unknown - 1s 145ms/step - loss: 0.6934 - auc: 0.4661 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 145ms/step - loss: 0.6934 - auc: 0.4661 - regularization_loss: 0.0000e+00
___________________ test_train_metrics_steps[120-10-4-12-3] ____________________
num_rows = 120, batch_size = 10, train_metrics_steps = 4, expected_steps = 12
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 13 == 12
E + where 13 = len([{'auc': 0.1666666716337204, 'loss': 0.7004226446151733, 'regularization_loss': 0.0}, {'auc': 0.1666666716337204, 'los...'regularization_loss': 0.0}, {'auc': 0.28431370854377747, 'loss': 0.6743507981300354, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 370ms/step - loss: 0.7004 - auc: 0.1667 - regularization_loss: 0.0000e+00
2/Unknown - 1s 169ms/step - loss: 0.6863 - auc: 0.1667 - regularization_loss: 0.0000e+00
3/Unknown - 1s 152ms/step - loss: 0.6912 - auc: 0.1667 - regularization_loss: 0.0000e+00
4/Unknown - 1s 146ms/step - loss: 0.6993 - auc: 0.1667 - regularization_loss: 0.0000e+00
5/Unknown - 1s 144ms/step - loss: 0.6866 - auc: 0.2843 - regularization_loss: 0.0000e+00
6/Unknown - 1s 142ms/step - loss: 0.6744 - auc: 0.2843 - regularization_loss: 0.0000e+00
7/Unknown - 1s 141ms/step - loss: 0.6700 - auc: 0.2843 - regularization_loss: 0.0000e+00
8/Unknown - 1s 140ms/step - loss: 0.6615 - auc: 0.2843 - regularization_loss: 0.0000e+00
9/Unknown - 1s 139ms/step - loss: 0.6665 - auc: 0.2050 - regularization_loss: 0.0000e+00
10/Unknown - 2s 139ms/step - loss: 0.6707 - auc: 0.2050 - regularization_loss: 0.0000e+00
11/Unknown - 2s 138ms/step - loss: 0.6785 - auc: 0.2050 - regularization_loss: 0.0000e+00
12/Unknown - 2s 138ms/step - loss: 0.6794 - auc: 0.2050 - regularization_loss: 0.0000e+00
13/Unknown - 2s 137ms/step - loss: 0.6856 - auc: 0.1607 - regularization_loss: 0.0000e+00
13/13 [==============================] - 2s 137ms/step - loss: 0.6856 - auc: 0.1607 - regularization_loss: 0.0000e+00
=============================== warnings summary ===============================
../../../../../usr/lib/python3/dist-packages/requests/init.py:89
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 6 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 8 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/core/test_transformations.py: 13 warnings
tests/unit/tf/data_augmentation/test_negative_sampling.py: 10 warnings
tests/unit/tf/data_augmentation/test_noise.py: 1 warning
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 34 warnings
tests/unit/tf/models/test_retrieval.py: 60 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 2 warnings
tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 9 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 3 warnings
tests/unit/xgb/test_xgboost.py: 18 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 5 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 3 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/core/test_transformations.py: 10 warnings
tests/unit/tf/data_augmentation/test_negative_sampling.py: 10 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 32 warnings
tests/unit/tf/models/test_retrieval.py: 32 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 2 warnings
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 7 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 2 warnings
tests/unit/xgb/test_xgboost.py: 17 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]
tests/unit/tf/test_dataset.py::test_tf_catname_ordering
tests/unit/tf/test_dataset.py::test_tf_map
/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.
warnings.warn(
tests/unit/datasets/test_entertainment.py: 1 warning
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 1 warning
tests/unit/tf/core/test_prediction.py: 1 warning
tests/unit/tf/data_augmentation/test_negative_sampling.py: 9 warnings
tests/unit/tf/inputs/test_continuous.py: 2 warnings
tests/unit/tf/inputs/test_embedding.py: 9 warnings
tests/unit/tf/inputs/test_tabular.py: 8 warnings
tests/unit/tf/models/test_ranking.py: 16 warnings
tests/unit/tf/models/test_retrieval.py: 4 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/xgb/test_xgboost.py: 12 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:879: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack
embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/core/test_index.py: 4 warnings
tests/unit/tf/models/test_retrieval.py: 54 warnings
tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings
tests/unit/tf/predictions/test_classification.py: 12 warnings
tests/unit/tf/predictions/test_dot_product.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 2 warnings
/tmp/autograph_generated_filep2_luhsw.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.1]
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.3]
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.5]
tests/unit/tf/data_augmentation/test_noise.py::test_stochastic_swap_noise[0.7]
tests/unit/tf/models/test_base.py::test_model_pre_post[True]
tests/unit/tf/models/test_base.py::test_model_pre_post[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.
return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]
tests/unit/tf/models/test_base.py::test_freeze_sequential_block
tests/unit/tf/models/test_base.py::test_freeze_unfreeze
tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks
/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.
super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/core/transformations.py:980: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block
/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:201.)
return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]
tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple
tests/unit/xgb/test_xgboost.py::TestEvals::test_default
tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid
tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data
/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:335: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].
warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective
/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first
self.make_current()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [4] ../../../../../usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:2746: Not a test.
===== 9 failed, 663 passed, 11 skipped, 1016 warnings in 952.68s (0:15:52) =====
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_models] $ /bin/bash /tmp/jenkins18159804229374584082.sh
Click to view CI Results
GitHub pull request #727 of commit e71eed262579c7d89d6b760248fde2c8e2363e6c, no merge conflicts.
Running as SYSTEM
Setting status of e71eed262579c7d89d6b760248fde2c8e2363e6c to PENDING with url https://10.20.13.93:8080/job/merlin_models/1233/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/727/*:refs/remotes/origin/pr/727/* # timeout=10
> git rev-parse e71eed262579c7d89d6b760248fde2c8e2363e6c^{commit} # timeout=10
Checking out Revision e71eed262579c7d89d6b760248fde2c8e2363e6c (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f e71eed262579c7d89d6b760248fde2c8e2363e6c # timeout=10
Commit message: "Merge branch 'main' into batched-dataset-generator"
> git rev-list --no-walk ac62a1493006f73609c9730091a18b1f4e86a088 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins12663846900951396151.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.6)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.9.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version =2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version =2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.1)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version =1.4.0; python_version jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 685 items
tests/unit/config/test_schema.py .... [ 0%]
tests/unit/datasets/test_advertising.py .s [ 0%]
tests/unit/datasets/test_ecommerce.py ..sss [ 1%]
tests/unit/datasets/test_entertainment.py ....sss. [ 2%]
tests/unit/datasets/test_social.py . [ 2%]
tests/unit/datasets/test_synthetic.py ...... [ 3%]
tests/unit/implicit/test_implicit.py . [ 3%]
tests/unit/lightfm/test_lightfm.py . [ 4%]
tests/unit/tf/test_core.py ...... [ 4%]
tests/unit/tf/test_dataset.py ................ [ 7%]
tests/unit/tf/test_public_api.py . [ 7%]
tests/unit/tf/blocks/test_cross.py ........... [ 9%]
tests/unit/tf/blocks/test_dlrm.py .......... [ 10%]
tests/unit/tf/blocks/test_interactions.py . [ 10%]
tests/unit/tf/blocks/test_mlp.py ................................. [ 15%]
tests/unit/tf/blocks/test_optimizer.py sFF.FF........................... [ 20%]
..................... [ 23%]
tests/unit/tf/blocks/retrieval/test_base.py . [ 23%]
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py .. [ 23%]
tests/unit/tf/blocks/retrieval/test_two_tower.py ........... [ 25%]
tests/unit/tf/blocks/sampling/test_cross_batch.py . [ 25%]
tests/unit/tf/blocks/sampling/test_in_batch.py . [ 25%]
tests/unit/tf/core/test_aggregation.py ......... [ 27%]
tests/unit/tf/core/test_base.py .. [ 27%]
tests/unit/tf/core/test_combinators.py s................... [ 30%]
tests/unit/tf/core/test_encoder.py . [ 30%]
tests/unit/tf/core/test_index.py ..F [ 30%]
tests/unit/tf/core/test_prediction.py .. [ 31%]
tests/unit/tf/core/test_tabular.py .... [ 31%]
tests/unit/tf/examples/test_01_getting_started.py . [ 31%]
tests/unit/tf/examples/test_02_dataschema.py . [ 31%]
tests/unit/tf/examples/test_03_exploring_different_models.py . [ 32%]
tests/unit/tf/examples/test_04_export_ranking_models.py . [ 32%]
tests/unit/tf/examples/test_05_export_retrieval_model.py . [ 32%]
tests/unit/tf/examples/test_06_advanced_own_architecture.py . [ 32%]
tests/unit/tf/examples/test_07_train_traditional_models.py . [ 32%]
tests/unit/tf/examples/test_usecase_ecommerce_session_based.py . [ 32%]
tests/unit/tf/examples/test_usecase_pretrained_embeddings.py . [ 32%]
tests/unit/tf/inputs/test_continuous.py ..... [ 33%]
tests/unit/tf/inputs/test_embedding.py ................................. [ 38%]
..... [ 39%]
tests/unit/tf/inputs/test_tabular.py .................. [ 41%]
tests/unit/tf/layers/test_queue.py .............. [ 43%]
tests/unit/tf/losses/test_losses.py ....................... [ 47%]
tests/unit/tf/metrics/test_metrics_popularity.py ..... [ 48%]
tests/unit/tf/metrics/test_metrics_topk.py ....................... [ 51%]
tests/unit/tf/models/test_base.py s.....FFFF........ [ 54%]
tests/unit/tf/models/test_benchmark.py .. [ 54%]
tests/unit/tf/models/test_ranking.py .............................. [ 58%]
tests/unit/tf/models/test_retrieval.py ................................ [ 63%]
tests/unit/tf/prediction_tasks/test_classification.py .. [ 63%]
tests/unit/tf/prediction_tasks/test_multi_task.py ................ [ 65%]
tests/unit/tf/prediction_tasks/test_next_item.py ..... [ 66%]
tests/unit/tf/prediction_tasks/test_regression.py ..... [ 67%]
tests/unit/tf/prediction_tasks/test_retrieval.py . [ 67%]
tests/unit/tf/prediction_tasks/test_sampling.py ...... [ 68%]
tests/unit/tf/predictions/test_base.py ..... [ 69%]
tests/unit/tf/predictions/test_classification.py ....... [ 70%]
tests/unit/tf/predictions/test_dot_product.py ........ [ 71%]
tests/unit/tf/predictions/test_regression.py .. [ 71%]
tests/unit/tf/predictions/test_sampling.py .... [ 72%]
tests/unit/tf/transforms/test_bias.py .. [ 72%]
tests/unit/tf/transforms/test_features.py s............................. [ 76%]
............. [ 78%]
tests/unit/tf/transforms/test_negative_sampling.py .......... [ 80%]
tests/unit/tf/transforms/test_noise.py ..... [ 81%]
tests/unit/tf/transforms/test_tensor.py .. [ 81%]
tests/unit/tf/utils/test_batch.py .... [ 81%]
tests/unit/tf/utils/test_tf_utils.py ..... [ 82%]
tests/unit/torch/test_dataset.py ......... [ 83%]
tests/unit/torch/test_public_api.py . [ 84%]
tests/unit/torch/block/test_base.py .... [ 84%]
tests/unit/torch/block/test_mlp.py . [ 84%]
tests/unit/torch/features/test_continuous.py .. [ 85%]
tests/unit/torch/features/test_embedding.py .............. [ 87%]
tests/unit/torch/features/test_tabular.py .... [ 87%]
tests/unit/torch/model/test_head.py ............ [ 89%]
tests/unit/torch/model/test_model.py .. [ 89%]
tests/unit/torch/tabular/test_aggregation.py ........ [ 90%]
tests/unit/torch/tabular/test_tabular.py ... [ 91%]
tests/unit/torch/tabular/test_transformations.py ....... [ 92%]
tests/unit/utils/test_schema_utils.py ................................ [ 97%]
tests/unit/xgb/test_xgboost.py .................... [100%]
=================================== FAILURES ===================================
_________________________ test_optimizers[optimizers0] _________________________
optimizers = ('sgd', 'adam')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99799997],
[0.99799997],
[0.99799997],
[0.99799997]], dtype=float32)
b = array([[0.9983299],
[0.9983299],
[0.9983299],
[0.9983299]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99799997 0.99799997 0.99799997 0.99799997]
E not close rhs = [0.9983299 0.9983299 0.9983299 0.9983299]
E not close dif = [0.00032991 0.00032991 0.00032991 0.00032991]
E not close tol = [1.99833e-06 1.99833e-06 1.99833e-06 1.99833e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00032991
E Max relative difference: 0.00033046
E x: array([[0.998],
E [0.998],
E [0.998],
E [0.998]], dtype=float32)
E y: array([[0.99833],
E [0.99833],
E [0.99833],
E [0.99833]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 952ms/step - loss: 1729.8229 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.4000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 28ms/step - loss: 865.2577 - precision: 0.6000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1000ms/step - loss: 1729.8229 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1727.3494 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1729.8229 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.4000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 32ms/step - loss: 865.2577 - precision_2: 0.6000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7fe4937cdaf0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7fe4937cdaf0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers1] _________________________
optimizers = ('rmsprop', 'sgd')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
tests/unit/tf/blocks/test_optimizer.py:95:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
b = array([[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]), array([0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455
E 0.99455 0.99455 0.99455]
E not close rhs = [0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774
E 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774]
E not close dif = [0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775
E 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775]
E not close tol = [1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06]
E dtype = float32, shape = (3, 4)
E Mismatched elements: 12 / 12 (100%)
E Max absolute difference: 0.00228775
E Max relative difference: 0.002295
E x: array([[0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
E y: array([[0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 31ms/step - loss: 1386.6677 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 950ms/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 29ms/step - loss: 696.8232 - precision_1: 0.4000 - recall_1: 0.5000 - binary_accuracy: 0.5000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 2s 2s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 2s 29ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7fe4b01734c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7fe4b01734c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers3] _________________________
optimizers = ('adagrad', 'rmsprop')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455],
[0.99455],
[0.99455],
[0.99455]], dtype=float32)
b = array([[0.9945456],
[0.9945456],
[0.9945456],
[0.9945456]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455]
E not close rhs = [0.9945456 0.9945456 0.9945456 0.9945456]
E not close dif = [4.4107437e-06 4.4107437e-06 4.4107437e-06 4.4107437e-06]
E not close tol = [1.9945455e-06 1.9945455e-06 1.9945455e-06 1.9945455e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 4.4107437e-06
E Max relative difference: 4.4349335e-06
E x: array([[0.99455],
E [0.99455],
E [0.99455],
E [0.99455]], dtype=float32)
E y: array([[0.994546],
E [0.994546],
E [0.994546],
E [0.994546]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 944ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1390.9613 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 31ms/step - loss: 1386.6677 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1389.4580 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
_________________________ test_optimizers[optimizers4] _________________________
optimizers = (SGD(), Adagrad())
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.9982936],
[0.9982936],
[0.9982936],
[0.9982936]], dtype=float32)
b = array([[0.999],
[0.999],
[0.999],
[0.999]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.9982936 0.9982936 0.9982936 0.9982936]
E not close rhs = [0.999 0.999 0.999 0.999]
E not close dif = [0.00070643 0.00070643 0.00070643 0.00070643]
E not close tol = [1.999e-06 1.999e-06 1.999e-06 1.999e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00070643
E Max relative difference: 0.00070714
E x: array([[0.998294],
E [0.998294],
E [0.998294],
E [0.998294]], dtype=float32)
E y: array([[0.999],
E [0.999],
E [0.999],
E [0.999]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 956ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 35ms/step - loss: 696.8232 - precision: 0.4000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 941ms/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 27ms/step - loss: 1390.9613 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 952ms/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
________________________ test_topk_recommender_outputs _________________________
ecommerce_data = <merlin.io.dataset.Dataset object at 0x7fe4a0ac7730>
batch_size = 100
def test_topk_recommender_outputs(ecommerce_data: Dataset, batch_size=100):
import numpy as np
import tensorflow as tf
import merlin.models.tf.dataset as tf_dataloader
from merlin.models.tf.core.index import IndexBlock
from merlin.models.utils.dataset import unique_rows_by_features
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
model = mm.TwoTowerModel(
ecommerce_data.schema,
query_tower=mm.MLPBlock([64]),
samplers=[mm.InBatchSampler()],
)
model.compile("adam", metrics=[mm.RecallAt(10)], run_eagerly=False)
model.fit(ecommerce_data, batch_size=batch_size, epochs=3)
eval_metrics = model.evaluate(
ecommerce_data, item_corpus=ecommerce_data, batch_size=batch_size, return_dict=True
)
# Manually compute top-k ids for a given batch
batch = mm.sample_batch(ecommerce_data, batch_size=batch_size, include_targets=False)
item_dataset = unique_rows_by_features(ecommerce_data, Tags.ITEM, Tags.ITEM_ID)
candidates_dataset_df = IndexBlock.get_candidates_dataset(
block=model.retrieval_block.item_block(), data=item_dataset, id_column="item_id"
)
item_tower_ids, item_tower_embeddings = IndexBlock.extract_ids_embeddings(
candidates_dataset_df, check_unique_ids=True
)
batch_query_tower_embeddings = model.retrieval_block.query_block()(batch)
batch_user_scores_all_items = tf.matmul(
batch_query_tower_embeddings, item_tower_embeddings, transpose_b=True
)
top_scores, top_indices = tf.math.top_k(batch_user_scores_all_items, k=10)
top_ids = tf.gather(item_tower_ids, top_indices)
# Get top-k ids from the topk_recommender_model
topk_recommender_model = model.to_top_k_recommender(ecommerce_data, k=10)
topk_predictions, topk_items = topk_recommender_model(batch)
# Assert top-k items from top-k recommender are the same as the manually computed top-k items
tf.debugging.assert_equal(top_ids, topk_items)
# Compute recall using top-k recommender
data_dl = tf_dataloader.BatchedDataset(
ecommerce_data,
batch_size=batch_size,
shuffle=False,
)
topk_output = topk_recommender_model.predict(data_dl)
topk_predictions, topk_items = topk_output
test_df = ecommerce_data.to_ddf()
positive_item_ids = np.array(test_df["item_id"].compute().values.tolist())
recall_at_10 = numpy_recall(positive_item_ids, topk_items, k=10)
tests/unit/tf/core/test_index.py:132:
labels = array([ 4, 14, 4, 5, 36, 26, 15, 16, 68, 37, 20, 7, 42,
40, 36, 26, 11, 85, 20, 33, 52,... 25, 75, 5, 7, 8, 33, 9, 13, 25, 27, 39, 17, 72,
5, 229, 11, 36, 57, 31, 25, 8, 18])
top_item_ids = array([[ 44, 85, 29, ..., 16, 31, 28],
[ 21, 11, 31, ..., 115, 29, 20],
[ 53, 16, 28, ..., 8... 28, 26, ..., 21, 24, 6],
[ 16, 11, 206, ..., 25, 67, 53],
[ 31, 29, 11, ..., 68, 206, 6]])
k = 10
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
E ValueError: operands could not be broadcast together with shapes (100,1) (200,10)
tests/unit/tf/core/test_index.py:85: ValueError
----------------------------- Captured stdout call -----------------------------
Epoch 1/3
1/Unknown - 3s 3s/step - loss: 4.5866 - recall_at_10: 0.1200 - regularization_loss: 0.0000e+00
2/2 [==============================] - 3s 39ms/step - loss: 4.5826 - recall_at_10: 0.1533 - regularization_loss: 0.0000e+00
Epoch 2/3
2/2 [==============================] - 0s 3ms/step - loss: 4.5826 - recall_at_10: 0.1700 - regularization_loss: 0.0000e+00
1/Unknown - 1s 895ms/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 20ms/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
1/Unknown - 0s 242ms/step
2/2 [==============================] - 0s 29ms/step
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
------------------------------ Captured log call -------------------------------
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING tensorflow:data_adapter.py:1232 Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
_____________________ test_train_metrics_steps[1-1-1-1-1] ______________________
num_rows = 1, batch_size = 1, train_metrics_steps = 1, expected_steps = 1
expected_metrics_steps = 1
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 2 == 1
E + where 2 = len([{'auc': 0.0, 'loss': 0.706480860710144, 'regularization_loss': 0.0}, {'auc': 0.0, 'loss': 0.6618980765342712, 'regularization_loss': 0.0}])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 401ms/step - loss: 0.7065 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/Unknown - 1s 132ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 133ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-2-6-3] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 2, expected_steps = 6
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.5, 'loss': 0.6983961462974548, 'regularization_loss': 0.0}, {'auc': 0.5, 'loss': 0.6882072687149048, 'regul... 'regularization_loss': 0.0}, {'auc': 0.5333333015441895, 'loss': 0.6838648319244385, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 428ms/step - loss: 0.6984 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/Unknown - 1s 132ms/step - loss: 0.6882 - auc: 0.5000 - regularization_loss: 0.0000e+00
3/Unknown - 1s 133ms/step - loss: 0.6903 - auc: 0.4583 - regularization_loss: 0.0000e+00
4/Unknown - 1s 133ms/step - loss: 0.6879 - auc: 0.4583 - regularization_loss: 0.0000e+00
5/Unknown - 1s 134ms/step - loss: 0.6846 - auc: 0.5333 - regularization_loss: 0.0000e+00
6/Unknown - 1s 133ms/step - loss: 0.6839 - auc: 0.5333 - regularization_loss: 0.0000e+00
7/Unknown - 1s 134ms/step - loss: 0.6854 - auc: 0.5378 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 134ms/step - loss: 0.6854 - auc: 0.5378 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-3-6-2] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 3, expected_steps = 6
expected_metrics_steps = 2
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.2142857164144516, 'loss': 0.6933244466781616, 'regularization_loss': 0.0}, {'auc': 0.2142857164144516, 'los...'regularization_loss': 0.0}, {'auc': 0.23000000417232513, 'loss': 0.6849275827407837, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 393ms/step - loss: 0.6933 - auc: 0.2143 - regularization_loss: 0.0000e+00
2/Unknown - 1s 134ms/step - loss: 0.6921 - auc: 0.2143 - regularization_loss: 0.0000e+00
3/Unknown - 1s 134ms/step - loss: 0.6783 - auc: 0.2143 - regularization_loss: 0.0000e+00
4/Unknown - 1s 133ms/step - loss: 0.6927 - auc: 0.2300 - regularization_loss: 0.0000e+00
5/Unknown - 1s 133ms/step - loss: 0.6861 - auc: 0.2300 - regularization_loss: 0.0000e+00
6/Unknown - 1s 133ms/step - loss: 0.6849 - auc: 0.2300 - regularization_loss: 0.0000e+00
7/Unknown - 1s 135ms/step - loss: 0.6809 - auc: 0.4353 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 135ms/step - loss: 0.6809 - auc: 0.4353 - regularization_loss: 0.0000e+00
___________________ test_train_metrics_steps[120-10-4-12-3] ____________________
num_rows = 120, batch_size = 10, train_metrics_steps = 4, expected_steps = 12
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 13 == 12
E + where 13 = len([{'auc': 0.5208333730697632, 'loss': 0.6890256404876709, 'regularization_loss': 0.0}, {'auc': 0.5208333730697632, 'los... 'regularization_loss': 0.0}, {'auc': 0.1607142984867096, 'loss': 0.6994596123695374, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 397ms/step - loss: 0.6890 - auc: 0.5208 - regularization_loss: 0.0000e+00
2/Unknown - 1s 134ms/step - loss: 0.6848 - auc: 0.5208 - regularization_loss: 0.0000e+00
3/Unknown - 1s 133ms/step - loss: 0.6822 - auc: 0.5208 - regularization_loss: 0.0000e+00
4/Unknown - 1s 132ms/step - loss: 0.6745 - auc: 0.5208 - regularization_loss: 0.0000e+00
5/Unknown - 1s 132ms/step - loss: 0.7060 - auc: 0.1607 - regularization_loss: 0.0000e+00
6/Unknown - 1s 133ms/step - loss: 0.6995 - auc: 0.1607 - regularization_loss: 0.0000e+00
7/Unknown - 1s 135ms/step - loss: 0.6880 - auc: 0.1607 - regularization_loss: 0.0000e+00
8/Unknown - 1s 135ms/step - loss: 0.6951 - auc: 0.1607 - regularization_loss: 0.0000e+00
9/Unknown - 1s 135ms/step - loss: 0.6931 - auc: 0.2828 - regularization_loss: 0.0000e+00
10/Unknown - 2s 134ms/step - loss: 0.6937 - auc: 0.2828 - regularization_loss: 0.0000e+00
11/Unknown - 2s 134ms/step - loss: 0.6894 - auc: 0.2828 - regularization_loss: 0.0000e+00
12/Unknown - 2s 134ms/step - loss: 0.6859 - auc: 0.2828 - regularization_loss: 0.0000e+00
13/Unknown - 2s 134ms/step - loss: 0.6840 - auc: 0.3762 - regularization_loss: 0.0000e+00
13/13 [==============================] - 2s 134ms/step - loss: 0.6840 - auc: 0.3762 - regularization_loss: 0.0000e+00
=============================== warnings summary ===============================
../../../../../usr/lib/python3/dist-packages/requests/init.py:89
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 6 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 8 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 34 warnings
tests/unit/tf/models/test_retrieval.py: 60 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings
tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/transforms/test_bias.py: 2 warnings
tests/unit/tf/transforms/test_features.py: 10 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings
tests/unit/tf/transforms/test_noise.py: 1 warning
tests/unit/tf/utils/test_batch.py: 9 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 3 warnings
tests/unit/xgb/test_xgboost.py: 18 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 5 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 3 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 32 warnings
tests/unit/tf/models/test_retrieval.py: 32 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/transforms/test_features.py: 10 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings
tests/unit/tf/utils/test_batch.py: 7 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 2 warnings
tests/unit/xgb/test_xgboost.py: 17 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]
tests/unit/tf/test_dataset.py::test_tf_catname_ordering
tests/unit/tf/test_dataset.py::test_tf_map
/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.
warnings.warn(
tests/unit/datasets/test_entertainment.py: 1 warning
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 1 warning
tests/unit/tf/core/test_prediction.py: 1 warning
tests/unit/tf/inputs/test_continuous.py: 2 warnings
tests/unit/tf/inputs/test_embedding.py: 9 warnings
tests/unit/tf/inputs/test_tabular.py: 8 warnings
tests/unit/tf/models/test_ranking.py: 16 warnings
tests/unit/tf/models/test_retrieval.py: 4 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 3 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 9 warnings
tests/unit/xgb/test_xgboost.py: 12 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:879: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack
embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/core/test_index.py: 4 warnings
tests/unit/tf/models/test_retrieval.py: 54 warnings
tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings
tests/unit/tf/predictions/test_classification.py: 12 warnings
tests/unit/tf/predictions/test_dot_product.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 2 warnings
/tmp/autograph_generated_filel2a960z9.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/models/test_base.py::test_model_pre_post[True]
tests/unit/tf/models/test_base.py::test_model_pre_post[False]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.1]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.3]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.5]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.7]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.
return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]
tests/unit/tf/models/test_base.py::test_freeze_sequential_block
tests/unit/tf/models/test_base.py::test_freeze_unfreeze
tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks
/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.
super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transforms/features.py:481: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block
/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:201.)
return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]
tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple
tests/unit/xgb/test_xgboost.py::TestEvals::test_default
tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid
tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data
/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:335: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].
warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective
/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first
self.make_current()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [4] ../../../../../usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:2746: Not a test.
===== 9 failed, 665 passed, 11 skipped, 1024 warnings in 965.63s (0:16:05) =====
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_models] $ /bin/bash /tmp/jenkins5217970521901280108.sh
Click to view CI Results
GitHub pull request #727 of commit 5761c1d5c8a36d41a4a1b2414970a2debeb21ddb, no merge conflicts.
Running as SYSTEM
Setting status of 5761c1d5c8a36d41a4a1b2414970a2debeb21ddb to PENDING with url https://10.20.13.93:8080/job/merlin_models/1236/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/727/*:refs/remotes/origin/pr/727/* # timeout=10
> git rev-parse 5761c1d5c8a36d41a4a1b2414970a2debeb21ddb^{commit} # timeout=10
Checking out Revision 5761c1d5c8a36d41a4a1b2414970a2debeb21ddb (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f 5761c1d5c8a36d41a4a1b2414970a2debeb21ddb # timeout=10
Commit message: "Merge branch 'main' into batched-dataset-generator"
> git rev-list --no-walk dd10b78cadea98443c107e2c28ffe096a41e12a7 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins15061209980181420151.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.4.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.6)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.3.0)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.9.1)
Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.4)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version =2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version =2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (23.2.1)
Requirement already satisfied: tornado>=6.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version =1.4.0; python_version jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 685 items
tests/unit/config/test_schema.py .... [ 0%]
tests/unit/datasets/test_advertising.py .s [ 0%]
tests/unit/datasets/test_ecommerce.py ..sss [ 1%]
tests/unit/datasets/test_entertainment.py ....sss. [ 2%]
tests/unit/datasets/test_social.py . [ 2%]
tests/unit/datasets/test_synthetic.py ...... [ 3%]
tests/unit/implicit/test_implicit.py . [ 3%]
tests/unit/lightfm/test_lightfm.py . [ 4%]
tests/unit/tf/test_core.py ...... [ 4%]
tests/unit/tf/test_dataset.py ................ [ 7%]
tests/unit/tf/test_public_api.py . [ 7%]
tests/unit/tf/blocks/test_cross.py ........... [ 9%]
tests/unit/tf/blocks/test_dlrm.py .......... [ 10%]
tests/unit/tf/blocks/test_interactions.py . [ 10%]
tests/unit/tf/blocks/test_mlp.py ................................. [ 15%]
tests/unit/tf/blocks/test_optimizer.py sFF.FF........................... [ 20%]
..................... [ 23%]
tests/unit/tf/blocks/retrieval/test_base.py . [ 23%]
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py .. [ 23%]
tests/unit/tf/blocks/retrieval/test_two_tower.py ........... [ 25%]
tests/unit/tf/blocks/sampling/test_cross_batch.py . [ 25%]
tests/unit/tf/blocks/sampling/test_in_batch.py . [ 25%]
tests/unit/tf/core/test_aggregation.py ......... [ 27%]
tests/unit/tf/core/test_base.py .. [ 27%]
tests/unit/tf/core/test_combinators.py s................... [ 30%]
tests/unit/tf/core/test_encoder.py . [ 30%]
tests/unit/tf/core/test_index.py ..F [ 30%]
tests/unit/tf/core/test_prediction.py .. [ 31%]
tests/unit/tf/core/test_tabular.py .... [ 31%]
tests/unit/tf/examples/test_01_getting_started.py . [ 31%]
tests/unit/tf/examples/test_02_dataschema.py . [ 31%]
tests/unit/tf/examples/test_03_exploring_different_models.py . [ 32%]
tests/unit/tf/examples/test_04_export_ranking_models.py . [ 32%]
tests/unit/tf/examples/test_05_export_retrieval_model.py . [ 32%]
tests/unit/tf/examples/test_06_advanced_own_architecture.py . [ 32%]
tests/unit/tf/examples/test_07_train_traditional_models.py . [ 32%]
tests/unit/tf/examples/test_usecase_ecommerce_session_based.py . [ 32%]
tests/unit/tf/examples/test_usecase_pretrained_embeddings.py . [ 32%]
tests/unit/tf/inputs/test_continuous.py ..... [ 33%]
tests/unit/tf/inputs/test_embedding.py ................................. [ 38%]
..... [ 39%]
tests/unit/tf/inputs/test_tabular.py .................. [ 41%]
tests/unit/tf/layers/test_queue.py .............. [ 43%]
tests/unit/tf/losses/test_losses.py ....................... [ 47%]
tests/unit/tf/metrics/test_metrics_popularity.py ..... [ 48%]
tests/unit/tf/metrics/test_metrics_topk.py ....................... [ 51%]
tests/unit/tf/models/test_base.py s.....FFFF........ [ 54%]
tests/unit/tf/models/test_benchmark.py .. [ 54%]
tests/unit/tf/models/test_ranking.py .............................. [ 58%]
tests/unit/tf/models/test_retrieval.py ................................ [ 63%]
tests/unit/tf/prediction_tasks/test_classification.py .. [ 63%]
tests/unit/tf/prediction_tasks/test_multi_task.py ................ [ 65%]
tests/unit/tf/prediction_tasks/test_next_item.py ..... [ 66%]
tests/unit/tf/prediction_tasks/test_regression.py ..... [ 67%]
tests/unit/tf/prediction_tasks/test_retrieval.py . [ 67%]
tests/unit/tf/prediction_tasks/test_sampling.py ...... [ 68%]
tests/unit/tf/predictions/test_base.py ..... [ 69%]
tests/unit/tf/predictions/test_classification.py ....... [ 70%]
tests/unit/tf/predictions/test_dot_product.py ........ [ 71%]
tests/unit/tf/predictions/test_regression.py .. [ 71%]
tests/unit/tf/predictions/test_sampling.py .... [ 72%]
tests/unit/tf/transforms/test_bias.py .. [ 72%]
tests/unit/tf/transforms/test_features.py s............................. [ 76%]
............. [ 78%]
tests/unit/tf/transforms/test_negative_sampling.py .......... [ 80%]
tests/unit/tf/transforms/test_noise.py ..... [ 81%]
tests/unit/tf/transforms/test_tensor.py .. [ 81%]
tests/unit/tf/utils/test_batch.py .... [ 81%]
tests/unit/tf/utils/test_tf_utils.py ..... [ 82%]
tests/unit/torch/test_dataset.py ......... [ 83%]
tests/unit/torch/test_public_api.py . [ 84%]
tests/unit/torch/block/test_base.py .... [ 84%]
tests/unit/torch/block/test_mlp.py . [ 84%]
tests/unit/torch/features/test_continuous.py .. [ 85%]
tests/unit/torch/features/test_embedding.py .............. [ 87%]
tests/unit/torch/features/test_tabular.py .... [ 87%]
tests/unit/torch/model/test_head.py ............ [ 89%]
tests/unit/torch/model/test_model.py .. [ 89%]
tests/unit/torch/tabular/test_aggregation.py ........ [ 90%]
tests/unit/torch/tabular/test_tabular.py ... [ 91%]
tests/unit/torch/tabular/test_transformations.py ....... [ 92%]
tests/unit/utils/test_schema_utils.py ................................ [ 97%]
tests/unit/xgb/test_xgboost.py .................... [100%]
=================================== FAILURES ===================================
_________________________ test_optimizers[optimizers0] _________________________
optimizers = ('sgd', 'adam')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99799997],
[0.99799997],
[0.99799997],
[0.99799997]], dtype=float32)
b = array([[0.9983299],
[0.9983299],
[0.9983299],
[0.9983299]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99799997 0.99799997 0.99799997 0.99799997]
E not close rhs = [0.9983299 0.9983299 0.9983299 0.9983299]
E not close dif = [0.00032991 0.00032991 0.00032991 0.00032991]
E not close tol = [1.99833e-06 1.99833e-06 1.99833e-06 1.99833e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00032991
E Max relative difference: 0.00033046
E x: array([[0.998],
E [0.998],
E [0.998],
E [0.998]], dtype=float32)
E y: array([[0.99833],
E [0.99833],
E [0.99833],
E [0.99833]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 957ms/step - loss: 1729.8229 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.4000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 32ms/step - loss: 865.2577 - precision: 0.6000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 976ms/step - loss: 1729.8229 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1727.3494 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.4000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1729.8229 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.4000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 33ms/step - loss: 865.2577 - precision_2: 0.6000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7fb79c1abb80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 15 calls to <function Model.make_train_function..train_function at 0x7fb79c1abb80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers1] _________________________
optimizers = ('rmsprop', 'sgd')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
tests/unit/tf/blocks/test_optimizer.py:95:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455],
[0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
b = array([[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774],
[0.99683774, 0.99683774, 0.99683774, 0.99683774]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]), array([0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455 0.99455
E 0.99455 0.99455 0.99455]
E not close rhs = [0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774
E 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774 0.99683774]
E not close dif = [0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775
E 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775 0.00228775]
E not close tol = [1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06 1.9968377e-06
E 1.9968377e-06 1.9968377e-06]
E dtype = float32, shape = (3, 4)
E Mismatched elements: 12 / 12 (100%)
E Max absolute difference: 0.00228775
E Max relative difference: 0.002295
E x: array([[0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455],
E [0.99455, 0.99455, 0.99455, 0.99455]], dtype=float32)
E y: array([[0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838],
E [0.996838, 0.996838, 0.996838, 0.996838]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 30ms/step - loss: 1386.6677 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 932ms/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 29ms/step - loss: 696.8232 - precision_1: 0.4000 - recall_1: 0.5000 - binary_accuracy: 0.5000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 36ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7fb78c0f3280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
------------------------------ Captured log call -------------------------------
WARNING tensorflow:def_function.py:148 5 out of the last 11 calls to <function Model.make_train_function..train_function at 0x7fb78c0f3280> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.
_________________________ test_optimizers[optimizers3] _________________________
optimizers = ('adagrad', 'rmsprop')
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.99455],
[0.99455],
[0.99455],
[0.99455]], dtype=float32)
b = array([[0.9945456],
[0.9945456],
[0.9945456],
[0.9945456]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.99455 0.99455 0.99455 0.99455]
E not close rhs = [0.9945456 0.9945456 0.9945456 0.9945456]
E not close dif = [4.4107437e-06 4.4107437e-06 4.4107437e-06 4.4107437e-06]
E not close tol = [1.9945455e-06 1.9945455e-06 1.9945455e-06 1.9945455e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 4.4107437e-06
E Max relative difference: 4.4349335e-06
E x: array([[0.99455],
E [0.99455],
E [0.99455],
E [0.99455]], dtype=float32)
E y: array([[0.994546],
E [0.994546],
E [0.994546],
E [0.994546]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 955ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 42ms/step - loss: 1390.9613 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 32ms/step - loss: 1386.6677 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 31ms/step - loss: 1389.4580 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
_________________________ test_optimizers[optimizers4] _________________________
optimizers = (SGD(), Adagrad())
@pytest.mark.parametrize(
"optimizers",
[
("sgd", "adam"),
("rmsprop", "sgd"),
("adam", "adagrad"),
("adagrad", "rmsprop"),
(tf.keras.optimizers.SGD(), tf.keras.optimizers.Adagrad()),
],
)
def test_optimizers(optimizers):
testing_data = generate_data("e-commerce", num_rows=5)
train_schema = testing_data.schema.select_by_name(
names=["user_categories", "user_brands", "user_shops", "user_intensions"]
)
# "first_opt" means set optimizer of the model with the first optimizers (optimizers[0])
# "multi_opt" means set optimizer with multi_optimizers, first layer with the first optimizer
# and the second layer with the second optimizer
test_cases = ["first_opt", "second_opt", "multi_opt"]
models, layers = {}, {}
for t in test_cases:
layers[t] = generate_two_layers()
input_block = ParallelBlock(TabularBlock.from_schema(schema=train_schema), is_input=True)
models[t] = ml.Model.from_block(
block=SequentialBlock([input_block, layers[t][0], layers[t][1]]),
schema=train_schema,
prediction_tasks=ml.BinaryClassificationTask("click"),
)
if t == "first_opt":
optimizer = optimizers[0]
if t == "second_opt":
optimizer = optimizers[1]
if t == "multi_opt":
multi_optimizer = ml.MultiOptimizer(
default_optimizer=optimizers[0],
optimizers_and_blocks=[
ml.OptimizerBlocks(optimizers[0], layers[t][0]),
ml.OptimizerBlocks(optimizers[1], layers[t][1]),
],
)
optimizer = multi_optimizer
# run only one batch
tf.keras.utils.set_random_seed(1)
models[t].compile(optimizer=optimizer)
models[t].fit(testing_data, batch_size=5, epochs=1)
# Compare trainable variables updated by the same optimizer
test_case = TestCase()
test_case.assertAllClose(
layers["first_opt"][0].trainable_variables[0], layers["multi_opt"][0].trainable_variables[0]
)
test_case.assertAllClose(
layers["second_opt"][1].trainable_variables[0],
layers["multi_opt"][1].trainable_variables[0],
)
tests/unit/tf/blocks/test_optimizer.py:98:
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:1533: in decorated
return f(*args, **kwds)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3108: in assertAllClose
self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3064: in _assertAllCloseRecursive
self._assertArrayLikeAllClose(
self = <tensorflow.python.framework.test_util.TensorFlowTestCase testMethod=runTest>
a = array([[0.9982936],
[0.9982936],
[0.9982936],
[0.9982936]], dtype=float32)
b = array([[0.999],
[0.999],
[0.999],
[0.999]], dtype=float32)
rtol = 1e-06, atol = 1e-06, msg = 'Mismatched value: a is different from b. '
def _assertArrayLikeAllClose(self, a, b, rtol=1e-6, atol=1e-6, msg=None):
(a, b) = self.evaluate_if_both_tensors(a, b)
a = self._GetNdArray(a)
b = self._GetNdArray(b)
# When the array rank is small, print its contents. Numpy array printing is
# implemented using inefficient recursion so prints can cause tests to
# time out.
if a.shape != b.shape and (b.ndim <= 3 or b.size < 500):
shape_mismatch_msg = ("Shape mismatch: expected %s, got %s with contents "
"%s.") % (a.shape, b.shape, b)
else:
shape_mismatch_msg = "Shape mismatch: expected %s, got %s." % (a.shape,
b.shape)
self.assertEqual(a.shape, b.shape, shape_mismatch_msg)
msgs = [msg]
# np.allclose does not always work for our custom bfloat16 extension type
# when type promotions are involved, so we first cast any bfloat16 arrays
# to float32.
a_dtype = a.dtype
a = a.astype(np.float32) if a.dtype == dtypes.bfloat16.as_numpy_dtype else a
b = b.astype(np.float32) if b.dtype == dtypes.bfloat16.as_numpy_dtype else b
if not np.allclose(a, b, rtol=rtol, atol=atol):
# Adds more details to np.testing.assert_allclose.
#
# NOTE: numpy.allclose (and numpy.testing.assert_allclose)
# checks whether two arrays are element-wise equal within a
# tolerance. The relative difference (rtol * abs(b)) and the
# absolute difference atol are added together to compare against
# the absolute difference between a and b. Here, we want to
# tell user which elements violate such conditions.
cond = np.logical_or(
np.abs(a - b) > atol + rtol * np.abs(b),
np.isnan(a) != np.isnan(b))
if a.ndim:
x = a[np.where(cond)]
y = b[np.where(cond)]
msgs.append("not close where = {}".format(np.where(cond)))
else:
# np.where is broken for scalars
x, y = a, b
msgs.append("not close lhs = {}".format(x))
msgs.append("not close rhs = {}".format(y))
msgs.append("not close dif = {}".format(np.abs(x - y)))
msgs.append("not close tol = {}".format(atol + rtol * np.abs(y)))
msgs.append("dtype = {}, shape = {}".format(a_dtype, a.shape))
# TODO(xpan): There seems to be a bug:
# tensorflow/compiler/tests:binary_ops_test pass with float32
# nan even though the equal_nan is False by default internally.
np.testing.assert_allclose(
a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
E AssertionError:
E Not equal to tolerance rtol=1e-06, atol=1e-06
E Mismatched value: a is different from b.
E not close where = (array([0, 1, 2, 3]), array([0, 0, 0, 0]))
E not close lhs = [0.9982936 0.9982936 0.9982936 0.9982936]
E not close rhs = [0.999 0.999 0.999 0.999]
E not close dif = [0.00070643 0.00070643 0.00070643 0.00070643]
E not close tol = [1.999e-06 1.999e-06 1.999e-06 1.999e-06]
E dtype = float32, shape = (4, 1)
E Mismatched elements: 4 / 4 (100%)
E Max absolute difference: 0.00070643
E Max relative difference: 0.00070714
E x: array([[0.998294],
E [0.998294],
E [0.998294],
E [0.998294]], dtype=float32)
E y: array([[0.999],
E [0.999],
E [0.999],
E [0.999]], dtype=float32)
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:3001: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 1s 933ms/step - loss: 1392.9529 - precision: 0.0000e+00 - recall: 0.0000e+00 - binary_accuracy: 0.6000 - auc: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 28ms/step - loss: 696.8232 - precision: 0.4000 - recall: 0.5000 - binary_accuracy: 0.5000 - auc: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 946ms/step - loss: 1392.9529 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 35ms/step - loss: 1390.9613 - precision_1: 0.0000e+00 - recall_1: 0.0000e+00 - binary_accuracy: 0.6000 - auc_1: 0.5000 - regularization_loss: 0.0000e+00
1/Unknown - 1s 962ms/step - loss: 1392.9529 - precision_2: 0.0000e+00 - recall_2: 0.0000e+00 - binary_accuracy: 0.6000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 35ms/step - loss: 696.8232 - precision_2: 0.4000 - recall_2: 0.5000 - binary_accuracy: 0.5000 - auc_2: 0.5000 - regularization_loss: 0.0000e+00
________________________ test_topk_recommender_outputs _________________________
ecommerce_data = <merlin.io.dataset.Dataset object at 0x7fb79c1a7790>
batch_size = 100
def test_topk_recommender_outputs(ecommerce_data: Dataset, batch_size=100):
import numpy as np
import tensorflow as tf
import merlin.models.tf.dataset as tf_dataloader
from merlin.models.tf.core.index import IndexBlock
from merlin.models.utils.dataset import unique_rows_by_features
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
model = mm.TwoTowerModel(
ecommerce_data.schema,
query_tower=mm.MLPBlock([64]),
samplers=[mm.InBatchSampler()],
)
model.compile("adam", metrics=[mm.RecallAt(10)], run_eagerly=False)
model.fit(ecommerce_data, batch_size=batch_size, epochs=3)
eval_metrics = model.evaluate(
ecommerce_data, item_corpus=ecommerce_data, batch_size=batch_size, return_dict=True
)
# Manually compute top-k ids for a given batch
batch = mm.sample_batch(ecommerce_data, batch_size=batch_size, include_targets=False)
item_dataset = unique_rows_by_features(ecommerce_data, Tags.ITEM, Tags.ITEM_ID)
candidates_dataset_df = IndexBlock.get_candidates_dataset(
block=model.retrieval_block.item_block(), data=item_dataset, id_column="item_id"
)
item_tower_ids, item_tower_embeddings = IndexBlock.extract_ids_embeddings(
candidates_dataset_df, check_unique_ids=True
)
batch_query_tower_embeddings = model.retrieval_block.query_block()(batch)
batch_user_scores_all_items = tf.matmul(
batch_query_tower_embeddings, item_tower_embeddings, transpose_b=True
)
top_scores, top_indices = tf.math.top_k(batch_user_scores_all_items, k=10)
top_ids = tf.gather(item_tower_ids, top_indices)
# Get top-k ids from the topk_recommender_model
topk_recommender_model = model.to_top_k_recommender(ecommerce_data, k=10)
topk_predictions, topk_items = topk_recommender_model(batch)
# Assert top-k items from top-k recommender are the same as the manually computed top-k items
tf.debugging.assert_equal(top_ids, topk_items)
# Compute recall using top-k recommender
data_dl = tf_dataloader.BatchedDataset(
ecommerce_data,
batch_size=batch_size,
shuffle=False,
)
topk_output = topk_recommender_model.predict(data_dl)
topk_predictions, topk_items = topk_output
test_df = ecommerce_data.to_ddf()
positive_item_ids = np.array(test_df["item_id"].compute().values.tolist())
recall_at_10 = numpy_recall(positive_item_ids, topk_items, k=10)
tests/unit/tf/core/test_index.py:132:
labels = array([ 4, 14, 4, 5, 36, 26, 15, 16, 68, 37, 20, 7, 42,
40, 36, 26, 11, 85, 20, 33, 52,... 25, 75, 5, 7, 8, 33, 9, 13, 25, 27, 39, 17, 72,
5, 229, 11, 36, 57, 31, 25, 8, 18])
top_item_ids = array([[ 44, 85, 29, ..., 16, 31, 28],
[ 21, 11, 31, ..., 115, 29, 20],
[ 53, 16, 28, ..., 8... 28, 26, ..., 21, 24, 6],
[ 16, 11, 206, ..., 25, 67, 53],
[ 31, 29, 11, ..., 68, 206, 6]])
k = 10
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
E ValueError: operands could not be broadcast together with shapes (100,1) (200,10)
tests/unit/tf/core/test_index.py:85: ValueError
----------------------------- Captured stdout call -----------------------------
Epoch 1/3
1/Unknown - 3s 3s/step - loss: 4.5866 - recall_at_10: 0.1200 - regularization_loss: 0.0000e+00
2/2 [==============================] - 3s 37ms/step - loss: 4.5826 - recall_at_10: 0.1533 - regularization_loss: 0.0000e+00
Epoch 2/3
2/2 [==============================] - 0s 2ms/step - loss: 4.5826 - recall_at_10: 0.1700 - regularization_loss: 0.0000e+00
1/Unknown - 1s 1s/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 29ms/step - loss: 1.3559 - recall_at_10: 0.5900 - regularization_loss: 0.0000e+00
1/Unknown - 1s 613ms/step
2/2 [==============================] - 1s 25ms/step
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
------------------------------ Captured log call -------------------------------
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING tensorflow:data_adapter.py:1232 Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 6 batches). You may need to use the repeat() function when building your dataset.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
_____________________ test_train_metrics_steps[1-1-1-1-1] ______________________
num_rows = 1, batch_size = 1, train_metrics_steps = 1, expected_steps = 1
expected_metrics_steps = 1
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 2 == 1
E + where 2 = len([{'auc': 0.0, 'loss': 0.706480860710144, 'regularization_loss': 0.0}, {'auc': 0.0, 'loss': 0.6618980765342712, 'regularization_loss': 0.0}])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 384ms/step - loss: 0.7065 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/Unknown - 1s 127ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
2/2 [==============================] - 1s 128ms/step - loss: 0.6619 - auc: 0.0000e+00 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-2-6-3] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 2, expected_steps = 6
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.6428571343421936, 'loss': 0.6961261034011841, 'regularization_loss': 0.0}, {'auc': 0.6428571343421936, 'los... 'regularization_loss': 0.0}, {'auc': 0.5092592239379883, 'loss': 0.6835638880729675, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 376ms/step - loss: 0.6961 - auc: 0.6429 - regularization_loss: 0.0000e+00
2/Unknown - 1s 129ms/step - loss: 0.6869 - auc: 0.6429 - regularization_loss: 0.0000e+00
3/Unknown - 1s 129ms/step - loss: 0.6797 - auc: 0.6548 - regularization_loss: 0.0000e+00
4/Unknown - 1s 128ms/step - loss: 0.6837 - auc: 0.6548 - regularization_loss: 0.0000e+00
5/Unknown - 1s 128ms/step - loss: 0.6882 - auc: 0.5093 - regularization_loss: 0.0000e+00
6/Unknown - 1s 129ms/step - loss: 0.6836 - auc: 0.5093 - regularization_loss: 0.0000e+00
7/Unknown - 1s 128ms/step - loss: 0.6840 - auc: 0.5783 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 129ms/step - loss: 0.6840 - auc: 0.5783 - regularization_loss: 0.0000e+00
____________________ test_train_metrics_steps[60-10-3-6-2] _____________________
num_rows = 60, batch_size = 10, train_metrics_steps = 3, expected_steps = 6
expected_metrics_steps = 2
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 7 == 6
E + where 7 = len([{'auc': 0.3333333432674408, 'loss': 0.696563720703125, 'regularization_loss': 0.0}, {'auc': 0.3333333432674408, 'loss... 'regularization_loss': 0.0}, {'auc': 0.4479166567325592, 'loss': 0.6811782717704773, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 403ms/step - loss: 0.6966 - auc: 0.3333 - regularization_loss: 0.0000e+00
2/Unknown - 1s 128ms/step - loss: 0.6831 - auc: 0.3333 - regularization_loss: 0.0000e+00
3/Unknown - 1s 129ms/step - loss: 0.6783 - auc: 0.3333 - regularization_loss: 0.0000e+00
4/Unknown - 1s 129ms/step - loss: 0.6807 - auc: 0.4479 - regularization_loss: 0.0000e+00
5/Unknown - 1s 129ms/step - loss: 0.6928 - auc: 0.4479 - regularization_loss: 0.0000e+00
6/Unknown - 1s 129ms/step - loss: 0.6812 - auc: 0.4479 - regularization_loss: 0.0000e+00
7/Unknown - 1s 129ms/step - loss: 0.6831 - auc: 0.4412 - regularization_loss: 0.0000e+00
7/7 [==============================] - 1s 129ms/step - loss: 0.6831 - auc: 0.4412 - regularization_loss: 0.0000e+00
___________________ test_train_metrics_steps[120-10-4-12-3] ____________________
num_rows = 120, batch_size = 10, train_metrics_steps = 4, expected_steps = 12
expected_metrics_steps = 3
@pytest.mark.parametrize(
["num_rows", "batch_size", "train_metrics_steps", "expected_steps", "expected_metrics_steps"],
[
(1, 1, 1, 1, 1),
(60, 10, 2, 6, 3),
(60, 10, 3, 6, 2),
(120, 10, 4, 12, 3),
],
)
def test_train_metrics_steps(
num_rows, batch_size, train_metrics_steps, expected_steps, expected_metrics_steps
):
dataset = generate_data("e-commerce", num_rows=num_rows)
model = ml.Model(
ml.InputBlock(dataset.schema),
ml.MLPBlock([64]),
ml.BinaryClassificationTask("click"),
)
model.compile(
run_eagerly=True,
optimizer="adam",
metrics=[tf.keras.metrics.AUC(from_logits=True, name="auc")],
)
metrics_callback = MetricsLogger()
callbacks = [metrics_callback]
_ = model.fit(
dataset,
callbacks=callbacks,
epochs=1,
batch_size=batch_size,
train_metrics_steps=train_metrics_steps,
)
epoch0_logs = metrics_callback.epoch_logs[0]
# number of times compute_metrics called (number of batches in epoch)
assert len(epoch0_logs) == expected_steps
E AssertionError: assert 13 == 12
E + where 13 = len([{'auc': 0.4166666567325592, 'loss': 0.6960674524307251, 'regularization_loss': 0.0}, {'auc': 0.4166666567325592, 'los... 'regularization_loss': 0.0}, {'auc': 0.3541666865348816, 'loss': 0.6945654153823853, 'regularization_loss': 0.0}, ...])
tests/unit/tf/models/test_base.py:158: AssertionError
----------------------------- Captured stdout call -----------------------------
1/Unknown - 0s 403ms/step - loss: 0.6961 - auc: 0.4167 - regularization_loss: 0.0000e+00
2/Unknown - 1s 128ms/step - loss: 0.6943 - auc: 0.4167 - regularization_loss: 0.0000e+00
3/Unknown - 1s 128ms/step - loss: 0.6959 - auc: 0.4167 - regularization_loss: 0.0000e+00
4/Unknown - 1s 129ms/step - loss: 0.6931 - auc: 0.4167 - regularization_loss: 0.0000e+00
5/Unknown - 1s 129ms/step - loss: 0.6958 - auc: 0.3542 - regularization_loss: 0.0000e+00
6/Unknown - 1s 129ms/step - loss: 0.6946 - auc: 0.3542 - regularization_loss: 0.0000e+00
7/Unknown - 1s 129ms/step - loss: 0.6949 - auc: 0.3542 - regularization_loss: 0.0000e+00
8/Unknown - 1s 129ms/step - loss: 0.6929 - auc: 0.3542 - regularization_loss: 0.0000e+00
9/Unknown - 1s 129ms/step - loss: 0.6926 - auc: 0.4163 - regularization_loss: 0.0000e+00
10/Unknown - 2s 129ms/step - loss: 0.6914 - auc: 0.4163 - regularization_loss: 0.0000e+00
11/Unknown - 2s 129ms/step - loss: 0.6876 - auc: 0.4163 - regularization_loss: 0.0000e+00
12/Unknown - 2s 129ms/step - loss: 0.6884 - auc: 0.4163 - regularization_loss: 0.0000e+00
13/Unknown - 2s 129ms/step - loss: 0.6854 - auc: 0.5041 - regularization_loss: 0.0000e+00
13/13 [==============================] - 2s 129ms/step - loss: 0.6854 - auc: 0.5041 - regularization_loss: 0.0000e+00
=============================== warnings summary ===============================
../../../../../usr/lib/python3/dist-packages/requests/init.py:89
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.11) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 6 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 8 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 34 warnings
tests/unit/tf/models/test_retrieval.py: 60 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings
tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/transforms/test_bias.py: 2 warnings
tests/unit/tf/transforms/test_features.py: 10 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings
tests/unit/tf/transforms/test_noise.py: 1 warning
tests/unit/tf/utils/test_batch.py: 9 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 3 warnings
tests/unit/xgb/test_xgboost.py: 18 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 5 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 10 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 3 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 32 warnings
tests/unit/tf/models/test_retrieval.py: 32 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings
tests/unit/tf/predictions/test_base.py: 5 warnings
tests/unit/tf/predictions/test_classification.py: 7 warnings
tests/unit/tf/predictions/test_dot_product.py: 8 warnings
tests/unit/tf/predictions/test_regression.py: 2 warnings
tests/unit/tf/transforms/test_features.py: 10 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings
tests/unit/tf/utils/test_batch.py: 7 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 2 warnings
tests/unit/xgb/test_xgboost.py: 17 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py::test_synthetic_aliccp_raw_data
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-True-8]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-10]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-9]
tests/unit/tf/test_dataset.py::test_tf_drp_reset[100-False-8]
tests/unit/tf/test_dataset.py::test_tf_catname_ordering
tests/unit/tf/test_dataset.py::test_tf_map
/usr/local/lib/python3.8/dist-packages/cudf/core/frame.py:384: UserWarning: The deep parameter is ignored and is only included for pandas compatibility.
warnings.warn(
tests/unit/datasets/test_entertainment.py: 1 warning
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_dataset.py: 1 warning
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 10 warnings
tests/unit/tf/core/test_encoder.py: 1 warning
tests/unit/tf/core/test_prediction.py: 1 warning
tests/unit/tf/inputs/test_continuous.py: 2 warnings
tests/unit/tf/inputs/test_embedding.py: 9 warnings
tests/unit/tf/inputs/test_tabular.py: 8 warnings
tests/unit/tf/models/test_ranking.py: 16 warnings
tests/unit/tf/models/test_retrieval.py: 4 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 3 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 9 warnings
tests/unit/xgb/test_xgboost.py: 12 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:879: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack
embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/core/test_index.py: 4 warnings
tests/unit/tf/models/test_retrieval.py: 54 warnings
tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings
tests/unit/tf/predictions/test_classification.py: 12 warnings
tests/unit/tf/predictions/test_dot_product.py: 2 warnings
tests/unit/tf/utils/test_batch.py: 2 warnings
/tmp/autograph_generated_file5n67fknq.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/models/test_base.py::test_model_pre_post[True]
tests/unit/tf/models/test_base.py::test_model_pre_post[False]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.1]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.3]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.5]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.7]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.
return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]
tests/unit/tf/models/test_base.py::test_freeze_sequential_block
tests/unit/tf/models/test_base.py::test_freeze_unfreeze
tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks
/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.
super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transforms/features.py:481: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block
/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:201.)
return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]
tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple
tests/unit/xgb/test_xgboost.py::TestEvals::test_default
tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid
tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data
/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:335: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].
warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective
/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first
self.make_current()
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [4] ../../../../../usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:2746: Not a test.
===== 9 failed, 665 passed, 11 skipped, 1024 warnings in 951.53s (0:15:51) =====
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_models] $ /bin/bash /tmp/jenkins4198900465695055060.sh
Click to view CI Results
GitHub pull request #727 of commit a4ea6e665daeb85a9cfd14b53784f85bfe6ef8d6, no merge conflicts.
Running as SYSTEM
Setting status of a4ea6e665daeb85a9cfd14b53784f85bfe6ef8d6 to PENDING with url https://10.20.13.93:8080/job/merlin_models/1388/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
> git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
> git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/727/*:refs/remotes/origin/pr/727/* # timeout=10
> git rev-parse a4ea6e665daeb85a9cfd14b53784f85bfe6ef8d6^{commit} # timeout=10
Checking out Revision a4ea6e665daeb85a9cfd14b53784f85bfe6ef8d6 (detached)
> git config core.sparsecheckout # timeout=10
> git checkout -f a4ea6e665daeb85a9cfd14b53784f85bfe6ef8d6 # timeout=10
Commit message: "Move order of schema check and return iterator in dataset conversion"
> git rev-list --no-walk dd7cfcd6e788b49d55fe49bb16be64a222fed86f # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins8484377487081349638.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.4.0)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version =2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version =2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version =1.4.0; python_version jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-3.0.0
collected 711 items
tests/unit/config/test_schema.py .... [ 0%]
tests/unit/datasets/test_advertising.py .s [ 0%]
tests/unit/datasets/test_ecommerce.py ..sss [ 1%]
tests/unit/datasets/test_entertainment.py ....sss. [ 2%]
tests/unit/datasets/test_social.py . [ 2%]
tests/unit/datasets/test_synthetic.py ...... [ 3%]
tests/unit/implicit/test_implicit.py . [ 3%]
tests/unit/lightfm/test_lightfm.py . [ 3%]
tests/unit/tf/test_core.py ...... [ 4%]
tests/unit/tf/test_loader.py ................ [ 7%]
tests/unit/tf/test_public_api.py . [ 7%]
tests/unit/tf/blocks/test_cross.py ........... [ 8%]
tests/unit/tf/blocks/test_dlrm.py .......... [ 10%]
tests/unit/tf/blocks/test_interactions.py ... [ 10%]
tests/unit/tf/blocks/test_mlp.py ................................. [ 15%]
tests/unit/tf/blocks/test_optimizer.py s................................ [ 19%]
..................... [ 22%]
tests/unit/tf/blocks/retrieval/test_base.py . [ 22%]
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py .. [ 23%]
tests/unit/tf/blocks/retrieval/test_two_tower.py ............ [ 24%]
tests/unit/tf/blocks/sampling/test_cross_batch.py . [ 25%]
tests/unit/tf/blocks/sampling/test_in_batch.py . [ 25%]
tests/unit/tf/core/test_aggregation.py ......... [ 26%]
tests/unit/tf/core/test_base.py .. [ 26%]
tests/unit/tf/core/test_combinators.py s.................... [ 29%]
tests/unit/tf/core/test_encoder.py . [ 29%]
tests/unit/tf/core/test_index.py ..F [ 30%]
tests/unit/tf/core/test_prediction.py .. [ 30%]
tests/unit/tf/core/test_tabular.py ...... [ 31%]
tests/unit/tf/examples/test_01_getting_started.py . [ 31%]
tests/unit/tf/examples/test_02_dataschema.py . [ 31%]
tests/unit/tf/examples/test_03_exploring_different_models.py . [ 31%]
tests/unit/tf/examples/test_04_export_ranking_models.py . [ 31%]
tests/unit/tf/examples/test_05_export_retrieval_model.py . [ 32%]
tests/unit/tf/examples/test_06_advanced_own_architecture.py . [ 32%]
tests/unit/tf/examples/test_07_train_traditional_models.py . [ 32%]
tests/unit/tf/examples/test_usecase_ecommerce_session_based.py . [ 32%]
tests/unit/tf/examples/test_usecase_pretrained_embeddings.py . [ 32%]
tests/unit/tf/inputs/test_continuous.py ..... [ 33%]
tests/unit/tf/inputs/test_embedding.py ................................. [ 37%]
...... [ 38%]
tests/unit/tf/inputs/test_tabular.py .................. [ 41%]
tests/unit/tf/layers/test_queue.py .............. [ 43%]
tests/unit/tf/losses/test_losses.py ....................... [ 46%]
tests/unit/tf/metrics/test_metrics_popularity.py ..... [ 47%]
tests/unit/tf/metrics/test_metrics_topk.py ....................... [ 50%]
tests/unit/tf/models/test_base.py s................. [ 53%]
tests/unit/tf/models/test_benchmark.py .. [ 53%]
tests/unit/tf/models/test_ranking.py .................................. [ 58%]
tests/unit/tf/models/test_retrieval.py ................................ [ 62%]
tests/unit/tf/outputs/test_base.py ..... [ 63%]
tests/unit/tf/outputs/test_classification.py ...... [ 64%]
tests/unit/tf/outputs/test_contrastive.py ........... [ 65%]
tests/unit/tf/outputs/test_regression.py .. [ 65%]
tests/unit/tf/outputs/test_sampling.py .... [ 66%]
tests/unit/tf/prediction_tasks/test_classification.py .. [ 66%]
tests/unit/tf/prediction_tasks/test_multi_task.py ................ [ 69%]
tests/unit/tf/prediction_tasks/test_next_item.py ..... [ 69%]
tests/unit/tf/prediction_tasks/test_regression.py ..... [ 70%]
tests/unit/tf/prediction_tasks/test_retrieval.py . [ 70%]
tests/unit/tf/prediction_tasks/test_sampling.py ...... [ 71%]
tests/unit/tf/transforms/test_bias.py .. [ 71%]
tests/unit/tf/transforms/test_features.py s............................. [ 75%]
....................s...... [ 79%]
tests/unit/tf/transforms/test_negative_sampling.py ......... [ 81%]
tests/unit/tf/transforms/test_noise.py ..... [ 81%]
tests/unit/tf/transforms/test_tensor.py .. [ 81%]
tests/unit/tf/utils/test_batch.py .... [ 82%]
tests/unit/tf/utils/test_tf_utils.py ..... [ 83%]
tests/unit/torch/test_dataset.py ......... [ 84%]
tests/unit/torch/test_public_api.py . [ 84%]
tests/unit/torch/block/test_base.py .... [ 85%]
tests/unit/torch/block/test_mlp.py . [ 85%]
tests/unit/torch/features/test_continuous.py .. [ 85%]
tests/unit/torch/features/test_embedding.py .............. [ 87%]
tests/unit/torch/features/test_tabular.py .... [ 88%]
tests/unit/torch/model/test_head.py ............ [ 89%]
tests/unit/torch/model/test_model.py .. [ 90%]
tests/unit/torch/tabular/test_aggregation.py ........ [ 91%]
tests/unit/torch/tabular/test_tabular.py ... [ 91%]
tests/unit/torch/tabular/test_transformations.py ....... [ 92%]
tests/unit/utils/test_schema_utils.py ................................ [ 97%]
tests/unit/xgb/test_xgboost.py .................... [100%]
=================================== FAILURES ===================================
________________________ test_topk_recommender_outputs _________________________
ecommerce_data = <merlin.io.dataset.Dataset object at 0x7f053113f400>
batch_size = 100
def test_topk_recommender_outputs(ecommerce_data: Dataset, batch_size=100):
import numpy as np
import tensorflow as tf
from merlin.models.tf.core.index import IndexBlock
from merlin.models.utils.dataset import unique_rows_by_features
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
model = mm.TwoTowerModel(
ecommerce_data.schema,
query_tower=mm.MLPBlock([64]),
samplers=[mm.InBatchSampler()],
)
model.compile("adam", metrics=[mm.RecallAt(10)], run_eagerly=False)
model.fit(ecommerce_data, batch_size=batch_size, epochs=3)
eval_metrics = model.evaluate(
ecommerce_data, item_corpus=ecommerce_data, batch_size=batch_size, return_dict=True
)
# Manually compute top-k ids for a given batch
batch = mm.sample_batch(ecommerce_data, batch_size=batch_size, include_targets=False)
item_dataset = unique_rows_by_features(ecommerce_data, Tags.ITEM, Tags.ITEM_ID)
candidates_dataset_df = IndexBlock.get_candidates_dataset(
block=model.retrieval_block.item_block(), data=item_dataset, id_column="item_id"
)
item_tower_ids, item_tower_embeddings = IndexBlock.extract_ids_embeddings(
candidates_dataset_df, check_unique_ids=True
)
batch_query_tower_embeddings = model.retrieval_block.query_block()(batch)
batch_user_scores_all_items = tf.matmul(
batch_query_tower_embeddings, item_tower_embeddings, transpose_b=True
)
top_scores, top_indices = tf.math.top_k(batch_user_scores_all_items, k=10)
top_ids = tf.gather(item_tower_ids, top_indices)
# Get top-k ids from the topk_recommender_model
topk_recommender_model = model.to_top_k_recommender(ecommerce_data, k=10)
topk_predictions, topk_items = topk_recommender_model(batch)
# Assert top-k items from top-k recommender are the same as the manually computed top-k items
tf.debugging.assert_equal(top_ids, topk_items)
# Compute recall using top-k recommender
data_dl = mm.Loader(
ecommerce_data,
batch_size=batch_size,
shuffle=False,
)
topk_output = topk_recommender_model.predict(data_dl)
topk_predictions, topk_items = topk_output
test_df = ecommerce_data.to_ddf()
positive_item_ids = np.array(test_df["item_id"].compute().values.tolist())
recall_at_10 = numpy_recall(positive_item_ids, topk_items, k=10)
tests/unit/tf/core/test_index.py:131:
labels = array([ 30, 23, 134, 22, 36, 24, 21, 4, 22, 43, 218, 19, 31,
30, 9, 15, 31, 14, 12, 15, 116,... 213, 61, 13, 49, 12, 27, 10, 20, 63, 22, 7, 2, 2,
124, 12, 34, 51, 7, 40, 33, 14, 68])
top_item_ids = array([[ 49, 218, 30, ..., 26, 21, 116],
[ 21, 26, 218, ..., 30, 124, 65],
[ 68, 21, 33, ..., 3... 37, 218, ..., 3, 26, 21],
[110, 116, 49, ..., 24, 22, 56],
[ 26, 110, 24, ..., 65, 218, 22]])
k = 10
def numpy_recall(labels, top_item_ids, k):
return np.equal(np.expand_dims(labels, -1), top_item_ids[:, :k]).max(axis=-1).mean()
E ValueError: operands could not be broadcast together with shapes (100,1) (200,10)
tests/unit/tf/core/test_index.py:84: ValueError
----------------------------- Captured stdout call -----------------------------
Epoch 1/3
1/Unknown - 3s 3s/step - loss: 4.5925 - recall_at_10: 0.0600 - regularization_loss: 0.0000e+00
1/1 [==============================] - 3s 3s/step - loss: 4.5925 - recall_at_10: 0.0600 - regularization_loss: 0.0000e+00
Epoch 2/3
1/1 [==============================] - 0s 6ms/step - loss: 4.5925 - recall_at_10: 0.0600 - regularization_loss: 0.0000e+00
1/Unknown - 1s 907ms/step - loss: 0.8059 - recall_at_10: 0.3500 - regularization_loss: 0.0000e+00
1/1 [==============================] - 1s 911ms/step - loss: 0.8059 - recall_at_10: 0.3500 - regularization_loss: 0.0000e+00
1/Unknown - 0s 246ms/step
2/2 [==============================] - 0s 29ms/step
----------------------------- Captured stderr call -----------------------------
WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 3 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING:tensorflow:No training configuration found in save file, so the model was not compiled. Compile it manually.
------------------------------ Captured log call -------------------------------
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING tensorflow:data_adapter.py:1232 Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 3 batches). You may need to use the repeat() function when building your dataset.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING merlin_models:api.py:446 The sampler InBatchSampler returned no samples for this batch.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
WARNING absl:save.py:233 Found untraced functions such as model_context_layer_call_fn, model_context_layer_call_and_return_conditional_losses, parallel_block_layer_call_fn, parallel_block_layer_call_and_return_conditional_losses, sequential_block_1_layer_call_fn while saving (showing 5 of 54). These functions will not be directly callable after loading.
WARNING tensorflow:load.py:167 No training configuration found in save file, so the model was not compiled. Compile it manually.
=============================== warnings summary ===============================
../../../../../usr/lib/python3/dist-packages/requests/init.py:89
/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41
/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 6 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_loader.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_interactions.py: 2 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 11 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 8 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 38 warnings
tests/unit/tf/models/test_retrieval.py: 60 warnings
tests/unit/tf/outputs/test_base.py: 5 warnings
tests/unit/tf/outputs/test_classification.py: 6 warnings
tests/unit/tf/outputs/test_contrastive.py: 15 warnings
tests/unit/tf/outputs/test_regression.py: 2 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings
tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning
tests/unit/tf/transforms/test_bias.py: 2 warnings
tests/unit/tf/transforms/test_features.py: 10 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings
tests/unit/tf/transforms/test_noise.py: 1 warning
tests/unit/tf/utils/test_batch.py: 9 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 3 warnings
tests/unit/xgb/test_xgboost.py: 18 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_ecommerce.py: 2 warnings
tests/unit/datasets/test_entertainment.py: 4 warnings
tests/unit/datasets/test_social.py: 1 warning
tests/unit/datasets/test_synthetic.py: 5 warnings
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_core.py: 6 warnings
tests/unit/tf/test_loader.py: 1 warning
tests/unit/tf/blocks/test_cross.py: 5 warnings
tests/unit/tf/blocks/test_dlrm.py: 9 warnings
tests/unit/tf/blocks/test_interactions.py: 2 warnings
tests/unit/tf/blocks/test_mlp.py: 26 warnings
tests/unit/tf/blocks/test_optimizer.py: 30 warnings
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings
tests/unit/tf/core/test_aggregation.py: 6 warnings
tests/unit/tf/core/test_base.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 11 warnings
tests/unit/tf/core/test_encoder.py: 2 warnings
tests/unit/tf/core/test_index.py: 3 warnings
tests/unit/tf/core/test_prediction.py: 2 warnings
tests/unit/tf/inputs/test_continuous.py: 4 warnings
tests/unit/tf/inputs/test_embedding.py: 19 warnings
tests/unit/tf/inputs/test_tabular.py: 18 warnings
tests/unit/tf/models/test_base.py: 17 warnings
tests/unit/tf/models/test_benchmark.py: 2 warnings
tests/unit/tf/models/test_ranking.py: 36 warnings
tests/unit/tf/models/test_retrieval.py: 32 warnings
tests/unit/tf/outputs/test_base.py: 5 warnings
tests/unit/tf/outputs/test_classification.py: 6 warnings
tests/unit/tf/outputs/test_contrastive.py: 15 warnings
tests/unit/tf/outputs/test_regression.py: 2 warnings
tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings
tests/unit/tf/transforms/test_features.py: 10 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings
tests/unit/tf/utils/test_batch.py: 7 warnings
tests/unit/torch/block/test_base.py: 4 warnings
tests/unit/torch/block/test_mlp.py: 1 warning
tests/unit/torch/features/test_continuous.py: 1 warning
tests/unit/torch/features/test_embedding.py: 4 warnings
tests/unit/torch/features/test_tabular.py: 4 warnings
tests/unit/torch/model/test_head.py: 12 warnings
tests/unit/torch/model/test_model.py: 2 warnings
tests/unit/torch/tabular/test_aggregation.py: 6 warnings
tests/unit/torch/tabular/test_transformations.py: 2 warnings
tests/unit/xgb/test_xgboost.py: 17 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/datasets/test_entertainment.py: 1 warning
tests/unit/implicit/test_implicit.py: 1 warning
tests/unit/lightfm/test_lightfm.py: 1 warning
tests/unit/tf/test_loader.py: 1 warning
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings
tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings
tests/unit/tf/core/test_combinators.py: 11 warnings
tests/unit/tf/core/test_encoder.py: 1 warning
tests/unit/tf/core/test_prediction.py: 1 warning
tests/unit/tf/inputs/test_continuous.py: 2 warnings
tests/unit/tf/inputs/test_embedding.py: 9 warnings
tests/unit/tf/inputs/test_tabular.py: 8 warnings
tests/unit/tf/models/test_ranking.py: 20 warnings
tests/unit/tf/models/test_retrieval.py: 4 warnings
tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings
tests/unit/tf/prediction_tasks/test_regression.py: 3 warnings
tests/unit/tf/transforms/test_negative_sampling.py: 9 warnings
tests/unit/xgb/test_xgboost.py: 12 warnings
/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].
warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export
tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:910: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack
embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/blocks/retrieval/test_two_tower.py: 1 warning
tests/unit/tf/core/test_index.py: 4 warnings
tests/unit/tf/models/test_retrieval.py: 54 warnings
tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings
tests/unit/tf/utils/test_batch.py: 2 warnings
/tmp/autograph_generated_fileg17gzbmg.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/core/test_combinators.py::test_parallel_block_select_by_tags
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/core/tabular.py:614: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
elif isinstance(self.feature_names, collections.Sequence):
tests/unit/tf/models/test_base.py::test_model_pre_post[True]
tests/unit/tf/models/test_base.py::test_model_pre_post[False]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.1]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.3]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.5]
tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.7]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.
return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]
tests/unit/tf/models/test_base.py::test_freeze_sequential_block
tests/unit/tf/models/test_base.py::test_freeze_unfreeze
tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks
/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.
super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_ranking.py::test_deepfm_model_only_categ_feats[False]
tests/unit/tf/models/test_ranking.py::test_deepfm_model_categ_and_continuous_feats[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transforms/features.py:569: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag
return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]
/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.
warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block
/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at ../torch/csrc/utils/tensor_new.cpp:201.)
return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]
tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]
tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple
tests/unit/xgb/test_xgboost.py::TestEvals::test_default
tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid
tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data
/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:335: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].
warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective
/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first
self.make_current()
tests/unit/xgb/test_xgboost.py: 14 warnings
/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited
client.wait_for_workers(n_workers)
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/xgb/test_xgboost.py: 11 warnings
/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1183: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.
mask = pd.Series(mask)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP
SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR
SKIPPED [5] ../../../../../usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:2746: Not a test.
==== 1 failed, 698 passed, 12 skipped, 1064 warnings in 1034.96s (0:17:14) =====
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"
[merlin_models] $ /bin/bash /tmp/jenkins9020630992804606638.sh
@jperez999 , please track this development and if there is another PR. that covers this work , please close this appropriately
closing since we have https://github.com/NVIDIA-Merlin/dataloader/pull/65 merged.
