Rafay M.

icon indicating a website link [email protected]

icon company @foretheta icon location Code crunching, AI craftsmanship, turning bits into insights & magic for tomorrow!

Results 1 comments of Rafay M.

build_index using too much RAM during training and crashes

`from pyspark.sql import SparkSession spark = SparkSession.builder \ .appName("MyPySparkApp") \ .getOrCreate() from autofaiss import build_index _, index_path2_metric_infos = build_index( embeddings="output_parquet_embeddings/", distributed="pyspark", file_format="parquet", ids_path="ids_folder/", id_columns=["index"], embedding_column_name="vectors", metric_type="l2", temporary_indices_folder="TEST/", current_memory_available="24G", max_index_memory_usage="20G", nb_indices_to_keep=1,...