ValueError: Serialized size of Dataset (459161045 bytes) exceeds maximum allowed (104857600 bytes)

[karantai]

Hello. My initial data is about 3.8 gigabyte. I have used different batch_sizes but again, I have this serialization size issue. I am using a Tesla T4, 15 GB for that, but I face the same issue when I use CPU as well.

This is the complete error: Traceback (most recent call last): File "/home/gkarant/mobispaces/tensorflow_federated/tff.py", line 198, in <module> main() File "/home/gkarant/mobispaces/tensorflow_federated/tff.py", line 183, in main updated_server_state = training_process.next(server_state, federated_train_data) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/computation/computation_impl.py", line 148, in __call__ return self._context_stack.current.invoke(self, arg) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/sync_execution_context.py", line 65, in invoke return self._async_runner.run_coro_and_return_result( File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/async_utils.py", line 149, in run_coro_and_return_result return future.result() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/retrying.py", line 119, in retry_coro_fn raise e File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/retrying.py", line 109, in retry_coro_fn result = await fn(*args, **kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 236, in invoke arg = await tracing.wrap_coroutine_in_current_trace_context( File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 406, in _wrapped return await coro File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 120, in _ingest ingested = await asyncio.gather(*ingested) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/execution_contexts/async_execution_context.py", line 127, in _ingest return await executor.create_value(val, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 207, in async_trace result = await fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/remote_executor.py", line 236, in create_value value_proto, type_spec = serialize_value() File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/remote_executor.py", line 229, in serialize_value return value_serialization.serialize_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 374, in serialize_value return _serialize_federated_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 321, in _serialize_federated_value federated_value_proto, it_type = serialize_value(v, type_spec.member) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 370, in serialize_value return _serialize_sequence_value(value, type_spec) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/common_libs/tracing.py", line 236, in sync_trace result = fn(*fn_args, **fn_kwargs) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 272, in _serialize_sequence_value value_proto.sequence.serialized_graph_def = _serialize_dataset(value) File "/home/gkarant/anaconda3/envs/federated_learning/lib/python3.10/site-packages/tensorflow_federated/python/core/impl/executors/value_serialization.py", line 166, in _serialize_dataset raise ValueError( ValueError: Serialized size of Dataset (459161045 bytes) exceeds maximum allowed (104857600 bytes) Is there a dedicated solution for that?

Thank you