loss is decreasing but val_loss is increasing

[zhifeng-huang]

I have followed the tutorial to have the code below. I confused about the result from the code. The loss is decreasing. However, the val_loss is increasing from the beginning. The val_metrics is also decreasing, compared the result from Epoch 5 and Epoch 10.

This is very simple model. Why is it overfitting from the beginning?

This is the result:

Epoch 5: val_factorized_top_k/top_1_categorical_accuracy: 5.0000e-05 - val_factorized_top_k/top_5_categorical_accuracy: 0.0016 - val_factorized_top_k/top_10_categorical_accuracy: 0.0054 - val_factorized_top_k/top_50_categorical_accuracy: 0.0772 - val_factorized_top_k/top_100_categorical_accuracy: 0.1771 - val_loss: 28846.7070 - val_regularization_loss: 0.0000e+00 - val_total_loss: 28846.7070

Epoch 10: val_factorized_top_k/top_1_categorical_accuracy: 5.0000e-05 - val_factorized_top_k/top_5_categorical_accuracy: 3.5000e-04 - val_factorized_top_k/top_10_categorical_accuracy: 0.0016 - val_factorized_top_k/top_50_categorical_accuracy: 0.0512 - val_factorized_top_k/top_100_categorical_accuracy: 0.1436 - val_loss: 29914.9844 - val_regularization_loss: 0.0000e+00 - val_total_loss: 29914.9844

Epoch 15: val_factorized_top_k/top_1_categorical_accuracy: 0.0000e+00 - val_factorized_top_k/top_5_categorical_accuracy: 5.0000e-05 - val_factorized_top_k/top_10_categorical_accuracy: 0.0011 - val_factorized_top_k/top_50_categorical_accuracy: 0.0456 - val_factorized_top_k/top_100_categorical_accuracy: 0.1371 - val_loss: 30586.1602 - val_regularization_loss: 0.0000e+00 - val_total_loss: 30586.1602

from typing import Dict, Text

import tensorflow as tf
import tensorflow_datasets as tfds
import tensorflow_recommenders as tfrs

ratings = tfds.load('movielens/100k-ratings', split="train")
movies = tfds.load('movielens/100k-movies', split="train")

ratings = ratings.map(lambda x: {
    "movie_id": tf.strings.to_number(x["movie_id"]),
    "user_id": tf.strings.to_number(x["user_id"])
})
movies = movies.map(lambda x: tf.strings.to_number(x["movie_id"]))

class Model(tfrs.Model):

  def __init__(self):
    super().__init__()

    # Set up user representation.
    self.user_model = tf.keras.layers.Embedding(
        input_dim=2000, output_dim=64)
    # Set up movie representation.
    self.item_model = tf.keras.layers.Embedding(
        input_dim=2000, output_dim=64)
    # Set up a retrieval task and evaluation metrics over the
    # entire dataset of candidates.
    self.task = tfrs.tasks.Retrieval(
        metrics=tfrs.metrics.FactorizedTopK(
            candidates=movies.batch(128).map(self.item_model)
        )
    )

  def compute_loss(self, features: Dict[Text, tf.Tensor], training=False) -> tf.Tensor:

    user_embeddings = self.user_model(features["user_id"])
    movie_embeddings = self.item_model(features["movie_id"])

    return self.task(user_embeddings, movie_embeddings)


model = Model()
model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))

# Randomly shuffle data and split between train and test.
tf.random.set_seed(42)
shuffled = ratings.shuffle(100_000, seed=42, reshuffle_each_iteration=False)

train = shuffled.take(80_000)
test = shuffled.skip(80_000).take(20_000)

# Train.
history = model.fit(
    train.batch(4096),
    validation_data=test.batch(4096),
    validation_freq=5,
    epochs=50)

# Evaluate.
model.evaluate(test.batch(4096), return_dict=True)

[leoniloris]

Hello @zhifeng-huang ! were you able to understand why this loss is increasing?

[OmarMAmin]

@leoniloris @zhifeng-huang I believe this is some sort of overfitting, the model tried to memorize training examples, but wasn't able to generalize well to the validation ones, I believe this might change if you added some features describing items and users