reczilla icon indicating copy to clipboard operation
reczilla copied to clipboard
verified publisher icon naszilla

memory errors with SpectralCF

Open duncanmcelfresh opened this issue 3 years ago • 0 comments

Below are the the traces. These occurred on a GCP n1-highmem-2 node with 1x tesla T4. Saw this when running experiments with the following datasets:

  • AmazonMoviesTV
  • Dating
  • GoogleLocalReviews
  • Gowalla
  • YahooMusic
SpectralCF_RecommenderWrapper: Instantiating model...
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_time = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 170, in fit
    self.data_generator = Data(self.URM_train, batch_size=self.batch_size)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 34, in __init__
    self.R = np.zeros((self.n_users, self.n_items), dtype=np.float32)
MemoryError

and the following memory error on

  • BookCrossing
  • Epinions
  • NetflixPrize
  • Recipes
SpectralCF_RecommenderWrapper: Instantiating model...
SpectralCF: Computing adjacient_matrix...
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_time = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 181, in fit
    self.model.compute_eigenvalues()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 25, in compute_eigenvalues
    self.A = self._adjacient_matrix(self_connection=True)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 118, in _adjacient_matrix
    A = np.zeros([self.n_users+self.n_items, self.n_users+self.n_items], dtype=np.float32)
MemoryError

and the following on:

  • CiaoDVD
  • LastFM
  • YahooMovies
SpectralCF_RecommenderWrapper: Training SpectralCF...
SpectralCF_RecommenderWrapper: Saving model in file './result_experiments/__Temp
_SpectralCF_RecommenderWrapper_1185/_best_model'
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_tim
e = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 203, in fit
    self._update_best_model()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 227, in _update_best_model
    self.save_model(self.temp_file_folder, file_name="_best_model")
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 287, in save_model
    saver.save(self.sess, folder_path + file_name + "_session")
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1196, in save
    meta_graph_filename, strip_default_attrs=strip_default_attrs)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1234, in export_meta_graph
    graph_def=ops.get_default_graph().as_graph_def(add_shapes=True),
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3150, in as_graph_def
    result, _ = self._as_graph_def(from_version, add_shapes)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3108, in _as_graph_def
    c_api.TF_GraphToGraphDef(self._c_graph, buf)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot serialize protocol buffer of type tensorflow.GraphDef as the serialized size (5223729696bytes) would be larger than the limit (2147483647 bytes)

and this one on MarketBiasAmazon

SpectralCF: Initialization complete in 1.49 hour
SpectralCF: Building Tensorflow graph...
/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/Spec
tralCF_our_interface/SpectralCF.py:74: ComplexWarning: Casting complex values to
 real discards the imaginary part
  A_hat = A_hat.astype(np.float32)
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_tim
e = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 187, in fit
    self.model.build_graph()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 80, in build_graph
    embeddings = tf.matmul(A_hat, embeddings)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2399, in matmul
    a = ops.convert_to_tensor(a, name="a")
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1039, in convert_to_tensor
    return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1097, in convert_to_tensor_v2
    as_ref=False)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1175, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 304, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 245, in constant
    allow_broadcast=True)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
    allow_broadcast=allow_broadcast))
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 537, in make_tensor_proto
    "Cannot create a tensor proto whose content is larger than 2GB.")
ValueError: Cannot create a tensor proto whose content is larger than 2GB.

and the following on MovieTweetings and Movielens10M

SpectralCF: Computing laplacian_matrix...
SpectralCF: Computing eigenvalues...
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_time = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 181, in fit
    self.model.compute_eigenvalues()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 34, in compute_eigenvalues
    self.lamda, self.U = np.linalg.eig(self.L)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 1311, in eig
    w, vt = _umath_linalg.eig(a, signature=signature, extobj=extobj)
MemoryError

duncanmcelfresh avatar Jul 29 '22 02:07 duncanmcelfresh