Top2Vec
Top2Vec copied to clipboard
ddangelov
(0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[114389,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator Simple allocator [[{{node EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/GatherV2}}]]
I got this error in self.document_vectors = self._embed_documents(documents)
with this example:
import numpy as np
import pandas as pd
import json
import os
import ipywidgets as widgets
from IPython.display import clear_output, display
from top2vec import Top2Vec
papers_prepared_df = pd.read_feather("/Users/davidlaxer/Downloads/archive/covid19_papers_processed.feather")
top2vec_trained = Top2Vec(documents=papers_prepared_df.text.tolist(), embedding_model="universal-sentence-encoder", use_embedding_model_tokenizer=True, embedding_model_path="/Users/davidlaxer/Downloads/universal-sentence-encoder_4/", workers=4)
The python 3.8 process grows to 100gb and then generates the OOM exception here:
def _embed_documents(self, train_corpus):
self._check_import_status()
self._check_model_status()
# embed documents
batch_size = 10
document_vectors = []
current = 0
batches = int(len(train_corpus) / batch_size)
extra = len(train_corpus) % batch_size
#shape = tf.shape(self.embed)
for ind in range(0, batches):
document_vectors.append(self.embed(train_corpus[current:current + batch_size]))
current += batch_size
if extra > 0:
document_vectors.append(self.embed(train_corpus[current:current + extra]))
document_vectors = self._l2_normalize(tf.experimental.numpy.vstack(document_vectors))
return document_vectors
Are you embedding full papers? It looks like a GPU memory problem. Switching to CPU might be a temporary solution.
I filtered the papers_feathered_df from 81,089 to 34,989 rows. The embedding process ran for a while … and used about 110GB of DRAM. Then I got the exception.
I am running on Tensorflow-macos and Tensorflow-metal. My AMD GPU has ~14 GB.
1,114,389 X 320 is ~350GB, yes?
On Dec 23, 2021, at 6:36 AM, Dimo Angelov @.***> wrote:
Are you embedding full papers? It looks like a GPU memory problem. Switching to CPU might be a temporary solution.
— Reply to this email directly, view it on GitHub https://github.com/ddangelov/Top2Vec/issues/232#issuecomment-1000345706, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXWFW25NUOJHTLTVOSN5BLUSMXVVANCNFSM5KOCNSGQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.
Here's my test code:
import sys
import faulthandler
import numpy as np
import pandas as pd
import json
import os
import ipywidgets as widgets
from IPython.display import clear_output, display
from top2vec import Top2Vec
faulthandler.enable(file=sys.stderr, all_threads=True)
def filter_short(papers_df):
papers_df["token_counts"] = papers_df["text"].str.split().map(len)
papers_df = papers_df[papers_df.token_counts > 500].reset_index(drop=True)
papers_df.drop('token_counts', axis=1, inplace=True)
return papers_df
papers_prepared_df = pd.read_feather("/Users/davidlaxer/Downloads/archive/covid19_papers_processed.feather")
papers_feathered_filtered_df = filter_short(papers_prepared_df)
top2vec_trained = Top2Vec(documents=papers_feathered_filtered_df.text.tolist(), embedding_model="universal-sentence-encoder", use_embedding_model_tokenizer=True, embedding_model_path="/Users/davidlaxer/Downloads/universal-sentence-encoder_4/", workers=4)
#
It runs on the CPU (but get's a socket exception):
/Users/davidlaxer/anaconda3/envs/ai/bin/python /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 61480 --file /Users/davidlaxer/Top2Vec/notebooks/test1.py
Connected to pydev debugger (build 213.5744.248)
2021-12-23 08:38:31,356 - top2vec - INFO - Pre-processing documents for training
2021-12-23 08:41:47,873 - top2vec - INFO - Loading universal-sentence-encoder model at /Users/davidlaxer/Downloads/universal-sentence-encoder_4/
2021-12-23 08:41:48.050777: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-23 08:41:51,638 - top2vec - INFO - Creating joint document/word embedding
2021-12-23 08:44:36,819 - top2vec - INFO - Creating lower dimension embedding of documents
2021-12-23 08:45:08,849 - top2vec - INFO - Finding dense areas of documents
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292, in _on_run
r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292, in _on_run
r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292, in _on_run
r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292, in _on_run
r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 292, in _on_run
r = self.sock.recv(1024)
OSError: [Errno 9] Bad file descriptor
2021-12-23 08:45:11,505 - top2vec - INFO - Finding topics
Process finished with exit code 0
The python 3.8.5 process grows to 78GB, gets an OOM exception on the GPU in a tensorflow-metal virtual environment:
/Users/davidlaxer/tensorflow-metal/bin/python /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 65429 --file /Users/davidlaxer/Top2Vec/notebooks/test1.py
Connected to pydev debugger (build 213.5744.248)
2021-12-23 08:54:49,163 - top2vec - INFO - Pre-processing documents for training
/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.
warnings.warn(msg, category=FutureWarning)
2021-12-23 08:58:06,171 - top2vec - INFO - Loading universal-sentence-encoder model at /Users/davidlaxer/Downloads/universal-sentence-encoder_4/
2021-12-23 08:58:06.333613: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-12-23 08:58:06.334684: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-12-23 08:58:06.334908: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
Metal device set to: AMD Radeon Pro 5700 XT
systemMemory: 128.00 GB
maxCacheSize: 7.99 GB
2021-12-23 08:58:09.017270: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-23 08:58:10,404 - top2vec - INFO - Creating joint document/word embedding
2021-12-23 08:58:10.536431: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
Some resource has been exhausted.
For example, this error might be raised if a per-user quota is
exhausted, or perhaps the entire file system is out of space.
@@__init__
2 root error(s) found.
(0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[477052,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator Simple allocator
[[{{node EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/GatherV2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[[StatefulPartitionedCall/StatefulPartitionedCall/EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/Reshape_1/_188]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
(1) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[477052,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator Simple allocator
[[{{node EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/GatherV2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
0 successful operations.
0 derived errors ignored. [Op:__inference_restored_function_body_4561]
Function call stack:
restored_function_body -> restored_function_body
Traceback (most recent call last):
File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/top2vec/Top2Vec.py", line 538, in _embed_documents
document_vectors.append(self.embed(train_corpus[current:current + batch_size]))
File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py", line 701, in _call_attribute
return instance.__call__(*args, **kwargs)
File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 58, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[477052,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator Simple allocator
[[{{node EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/GatherV2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[[StatefulPartitionedCall/StatefulPartitionedCall/EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/Reshape_1/_188]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
(1) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[477052,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator Simple allocator
[[{{node EncoderDNN/EmbeddingLookup/EmbeddingLookupUnique/GatherV2}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
0 successful operations.
0 derived errors ignored. [Op:__inference_restored_function_body_4561]
Function call stack:
restored_function_body -> restored_function_body
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/top2vec/Top2Vec.py", line 542, in _embed_documents
faulthandler.dump_traceback(file=sys.stderr, all_threads=True)
NameError: name 'faulthandler' is not defined
python-BaseException
```
ipython
Python 3.8.5 (default, Sep 4 2020, 02:22:02)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.28.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import tensorflow
In [2]: print(tensorflow.version) 2.7.0
The new top2vec version 1.0.27 has some memory bugfixes as well as an option for document chunking for long documents which could help solve some of these issue. There is also a embedding_batch_size parameter.
I am working with 1.0.27 with a version of the COVID-19 repository from 11/21 with 822279 rows × 19 columns. If I filter the number of documents from 822279 to 615449 it still takes a lot of time in: _embed_documents. Here's an example of building a model with the universal-sentence-encoder_4:
top2vec_trained = Top2Vec(documents=papers_filtered_df.text.tolist(), split_documents=True, embedding_batch_size=16, embedding_model="universal-sentence-encoder", use_embedding_model_tokenizer=True, embedding_model_path="/Users/davidlaxer/Downloads/universal-sentence-encoder_4", workers=8)
The code is taking quite a bit of time in Top2vec.py _embed_documents, here:
document_vectors.append(self.embed(train_corpus[current:current + batch_size]))
Here's a profile with the filter short size set at 5,000:
papers_df = papers_df[papers_df.token_counts > 5000].reset_index(drop=True)
import cProfile, pstats, io
from pstats import SortKey
pr = cProfile.Profile()
pr.enable()
for ind in range(0, batches):
document_vectors.append(self.embed(train_corpus[current:current + batch_size]))
current += batch_size
if extra > 0:
document_vectors.append(self.embed(train_corpus[current:current + extra]))
document_vectors = self._l2_normalize(np.array(np.vstack(document_vectors)))
pr.disable()
s = io.StringIO()
sortby = SortKey.CUMULATIVE
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
ps.print_stats()
3638257 function calls (3568425 primitive calls) in 154.779 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
6983 0.015 0.000 152.706 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py:743(_call_attribute)
6983 0.019 0.000 152.692 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py:138(error_handler)
6983 0.129 0.000 152.643 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:887(__call__)
6983 0.086 0.000 152.281 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:934(_call)
6983 0.068 0.000 152.188 0.022 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2951(__call__)
6983 0.081 0.000 149.721 0.021 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:1782(_call_flat)
6983 0.205 0.000 149.416 0.021 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:454(call)
6983 0.034 0.000 149.041 0.021 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/execute.py:29(quick_execute)
6983 148.998 0.021 148.998 0.021 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_Execute}
6983 0.048 0.000 1.928 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:3219(_maybe_define_function)
6/4 0.148 0.025 1.925 0.481 {built-in method numpy.core._multiarray_umath.implement_array_function}
1 0.000 0.000 1.882 1.882 <__array_function__ internals>:2(vstack)
1 0.000 0.000 1.877 1.877 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/shape_base.py:223(vstack)
6987 1.833 0.000 1.833 0.000 {built-in method numpy.array}
1 0.000 0.000 1.736 1.736 <__array_function__ internals>:2(atleast_2d)
1 0.015 0.015 1.734 1.734 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/shape_base.py:82(atleast_2d)
6984 0.004 0.000 1.718 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/_asarray.py:110(asanyarray)
6983 0.046 0.000 1.381 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2648(canonicalize_function_inputs)
6983 0.080 0.000 1.311 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2817(_convert_inputs_to_signature)
6983 0.009 0.000 0.652 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py:178(wrapped)
6983 0.076 0.000 0.643 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:1636(convert_to_tensor)
6983 0.015 0.000 0.472 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2051(captured_inputs)
6983 0.005 0.000 0.405 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:340(_constant_tensor_conversion_function)
6983 0.007 0.000 0.400 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:170(constant)
6983 0.016 0.000 0.393 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:271(_constant_impl)
6983 0.009 0.000 0.367 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:302(_constant_eager_impl)
6983 0.343 0.000 0.359 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py:74(convert_to_eager_tensor)
27932 0.024 0.000 0.322 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:353(flatten)
27932 0.298 0.000 0.298 0.000 {built-in method tensorflow.python.util._pywrap_utils.Flatten}
6983 0.027 0.000 0.275 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:250(make_cache_key)
6983 0.169 0.000 0.232 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2058(<listcomp>)
6983 0.014 0.000 0.219 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:1157(flatten_up_to)
6983 0.006 0.000 0.181 0.000 {built-in method builtins.any}
13966 0.012 0.000 0.175 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:689(pack_sequence_as)
13966 0.010 0.000 0.175 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2851(<genexpr>)
6983 0.051 0.000 0.172 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2196(_build_call_outputs)
6983 0.018 0.000 0.165 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:124(is_compatible_with)
6983 0.016 0.000 0.164 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:154(lookup)
13966 0.032 0.000 0.163 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:649(_pack_sequence_as)
13966/6983 0.058 0.000 0.158 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:1029(assert_shallow_structure)
13966 0.044 0.000 0.151 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/handle_data_util.py:25(copy_handle_data)
6983 0.031 0.000 0.146 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:70(is_compatible_with)
1 0.000 0.000 0.141 0.141 <__array_function__ internals>:2(concatenate)
6983 0.030 0.000 0.137 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:185(called_without_tracing)
13966 0.020 0.000 0.119 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:98(__eq__)
6983 0.051 0.000 0.116 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:265(_make_execution_context)
251402 0.060 0.000 0.112 0.000 {built-in method builtins.isinstance}
6983 0.014 0.000 0.106 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:314(make_function_signature)
1 0.000 0.000 0.099 0.099 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/top2vec/Top2Vec.py:812(_l2_normalize)
1 0.026 0.026 0.099 0.099 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/preprocessing/_data.py:1731(normalize)
6983 0.023 0.000 0.093 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:180(_get_detector)
6983 0.048 0.000 0.090 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_EncodeArg}
48881 0.089 0.000 0.089 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:175(__eq__)
13966 0.016 0.000 0.082 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:201(__eq__)
6983 0.006 0.000 0.082 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py:1534(_autopacking_conversion_function)
6983 0.011 0.000 0.075 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py:1524(_should_not_autopack)
6983 0.033 0.000 0.075 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:183(_sequence_like)
1256940 0.063 0.000 0.063 0.000 {built-in method builtins.callable}
69830/20949 0.045 0.000 0.062 0.000 {built-in method builtins.hash}
34915 0.062 0.000 0.062 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:927(executing_eagerly)
20949 0.007 0.000 0.061 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:95(__hash__)
6983 0.019 0.000 0.060 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:1320(shape)
1 0.000 0.000 0.059 0.059 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/validation.py:486(check_array)
6983 0.058 0.000 0.058 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/weakref.py:422(__contains__)
6983 0.015 0.000 0.053 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2087(executing_eagerly)
55866 0.016 0.000 0.052 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/abc.py:96(__instancecheck__)
41898 0.039 0.000 0.050 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:1193(dtype)
41898 0.026 0.000 0.049 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:253(_yield_value)
69830 0.049 0.000 0.049 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsNestedOrComposite}
6983 0.005 0.000 0.048 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:1242(<listcomp>)
13966 0.016 0.000 0.044 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:343(__eq__)
20949 0.006 0.000 0.043 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:210(__hash__)
27932/13966 0.027 0.000 0.042 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:999(_yield_flat_up_to)
6983 0.020 0.000 0.039 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:229(__tf_tracing_type__)
55866 0.035 0.000 0.035 0.000 {built-in method _abc._abc_instancecheck}
6983 0.011 0.000 0.035 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:761(__init__)
6983 0.013 0.000 0.034 0.000 {built-in method builtins.all}
69830 0.029 0.000 0.032 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:258(_yield_sorted_items)
13967 0.031 0.000 0.031 0.000 {built-in method builtins.getattr}
6983 0.006 0.000 0.030 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/traceback_utils.py:32(is_traceback_filtering_enabled)
20949 0.009 0.000 0.030 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:188(__ne__)
13966 0.016 0.000 0.029 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:6285(get_default_graph)
6983 0.013 0.000 0.029 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:609(_packed_nest_with_indices)
1 0.000 0.000 0.029 0.029 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/validation.py:90(_assert_all_finite)
1 0.000 0.000 0.028 0.028 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/extmath.py:869(_safe_accumulator_op)
1 0.000 0.000 0.028 0.028 <__array_function__ internals>:2(sum)
1 0.000 0.000 0.028 0.028 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/fromnumeric.py:2111(sum)
1 0.000 0.000 0.028 0.028 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/fromnumeric.py:70(_wrapreduction)
1 0.028 0.028 0.028 0.028 {method 'reduce' of 'numpy.ufunc' objects}
20949 0.008 0.000 0.028 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:340(__hash__)
6983 0.013 0.000 0.024 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:155(is_compatible_with)
13966 0.023 0.000 0.024 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:678(as_dtype)
20949 0.009 0.000 0.024 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:63(base_dtype)
6983 0.013 0.000 0.023 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:771(<listcomp>)
6983 0.005 0.000 0.022 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/util/nest.py:146(is_namedtuple)
118711 0.020 0.000 0.020 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/array_ops.py:1530(<genexpr>)
104747 0.019 0.000 0.019 0.000 {built-in method builtins.len}
6983 0.014 0.000 0.019 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:61(__init__)
6983 0.017 0.000 0.017 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsNamedtuple}
13966 0.010 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2968(input_signature)
6983 0.008 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5418(_distribution_strategy_stack)
13966 0.015 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:850(experimental_get_tracing_count)
20949 0.015 0.000 0.015 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/dtypes.py:50(_is_ref_dtype)
6983 0.008 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_conversion_registry.py:112(get)
6983 0.010 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/six.py:582(iterkeys)
6983 0.013 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2865(<listcomp>)
6983 0.014 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:162(called_without_tracing)
1 0.000 0.000 0.014 0.014 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/extmath.py:51(row_norms)
1 0.000 0.000 0.014 0.014 <__array_function__ internals>:2(einsum)
1 0.000 0.000 0.014 0.014 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/einsumfunc.py:997(einsum)
1 0.014 0.014 0.014 0.014 {built-in method numpy.core._multiarray_umath.c_einsum}
6983 0.006 0.000 0.014 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/tape.py:220(could_possibly_record)
13966 0.010 0.000 0.013 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:158(_has_same_structure)
6983 0.011 0.000 0.013 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:290(__init__)
13966 0.013 0.000 0.013 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:5814(get_default)
13966 0.012 0.000 0.012 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:949(device_name)
6983 0.011 0.000 0.011 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/weakref.py:382(__getitem__)
6983 0.011 0.000 0.011 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:1203(function_call_options)
6983 0.008 0.000 0.011 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/save_context.py:59(in_save_context)
41898 0.011 0.000 0.011 0.000 {method '_datatype_enum' of 'tensorflow.python.framework.ops.EagerTensor' objects}
6983 0.005 0.000 0.011 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py:1016(PossibleTapeGradientTypes)
6983 0.010 0.000 0.010 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:539(__exit__)
13976 0.010 0.000 0.010 0.000 {built-in method builtins.hasattr}
6983 0.007 0.000 0.009 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:1107(is_compatible_with)
6983 0.009 0.000 0.009 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:196(__init__)
6983 0.009 0.000 0.009 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsTensor}
6983 0.004 0.000 0.008 0.000 <string>:1(__new__)
6983 0.008 0.000 0.008 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_TapeSetIsEmpty}
13966 0.007 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:546(ensure_initialized)
20949 0.007 0.000 0.007 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsCompositeTensor}
6983 0.006 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2973(flat_input_signature)
13966 0.007 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:60(dtype)
6983 0.005 0.000 0.007 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2640(_validate_inputs)
6983 0.006 0.000 0.006 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py:50(__init__)
6983 0.006 0.000 0.006 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_cache.py:60(__init__)
6983 0.006 0.000 0.006 0.000 {method '_shape_tuple' of 'tensorflow.python.framework.ops.EagerTensor' objects}
6983 0.006 0.000 0.006 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsMutableMapping}
6983 0.006 0.000 0.006 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:38(__init__)
6983 0.006 0.000 0.006 0.000 {built-in method tensorflow.python._pywrap_tfe.TFE_Py_TapeSetPossibleGradientTypes}
27932 0.005 0.000 0.005 0.000 {method 'append' of 'list' objects}
6983 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/c_api_util.py:100(has_been_garbage_collected)
27932 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:2057(context_safe)
13966 0.005 0.000 0.005 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsTypeSpec}
6983 0.005 0.000 0.005 0.000 {method 'acquire' of '_thread.RLock' objects}
13966 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2552(input_signature)
6983 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:216(__exit__)
13966 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:885(_handle)
13966 0.005 0.000 0.005 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:176(config_proto_serialized)
6983 0.004 0.000 0.004 0.000 {built-in method __new__ of type object at 0x106304808}
6983 0.004 0.000 0.004 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:155(__init__)
6983 0.004 0.000 0.004 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:174(value_type)
6983 0.004 0.000 0.004 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:529(__init__)
6983 0.004 0.000 0.004 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:210(__init__)
6984 0.003 0.000 0.003 0.000 {method 'get' of 'dict' objects}
6983 0.003 0.000 0.003 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:1927(outputs)
6983 0.003 0.000 0.003 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/save_context.py:42(in_save_context)
6983 0.003 0.000 0.003 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsMapping}
6983 0.003 0.000 0.003 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py:87(set_metadata)
6986 0.003 0.000 0.003 0.000 {built-in method builtins.issubclass}
6983 0.003 0.000 0.003 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/ops.py:3652(building_function)
6983 0.003 0.000 0.003 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsMappingView}
6983 0.002 0.000 0.003 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:1306(as_shape)
6983 0.002 0.000 0.002 0.000 {built-in method builtins.iter}
6983 0.002 0.000 0.002 0.000 {built-in method tensorflow.python.util._pywrap_utils.IsAttrs}
6983 0.002 0.000 0.002 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/saved_model/function_deserialization.py:202(_run_functions_eagerly)
6983 0.002 0.000 0.002 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/context.py:168(executor_type)
6983 0.002 0.000 0.002 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:55(shape)
6983 0.002 0.000 0.002 0.000 {method 'keys' of 'dict' objects}
6983 0.002 0.000 0.002 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_spec.py:65(name)
6983 0.002 0.000 0.002 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/framework/tensor_shape.py:834(rank)
6983 0.001 0.000 0.001 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:76(include_tensor_ranks_only)
6983 0.001 0.000 0.001 0.000 {method 'release' of '_thread.RLock' objects}
6983 0.001 0.000 0.001 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py:121(__exit__)
6983 0.001 0.000 0.001 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:535(__enter__)
6983 0.001 0.000 0.001 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2556(flat_input_signature)
6983 0.001 0.000 0.001 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function_trace_type.py:80(deletion_observer)
6983 0.001 0.000 0.001 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py:83(__enter__)
6983 0.001 0.000 0.001 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/tensorflow/python/eager/function.py:213(__enter__)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/preprocessing/_data.py:84(_handle_zeros_in_scale)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/validation.py:253(_num_samples)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/numerictypes.py:359(issubdtype)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/numeric.py:1865(isscalar)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/warnings.py:165(simplefilter)
2 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/numerictypes.py:285(issubclass_)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/warnings.py:181(_add_filter)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/warnings.py:458(__enter__)
4 0.000 0.000 0.000 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/abc.py:100(__subclasscheck__)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/_config.py:24(get_config)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/getlimits.py:382(__new__)
3 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/scipy/sparse/base.py:1205(isspmatrix)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/warnings.py:437(__init__)
4 0.000 0.000 0.000 0.000 {built-in method _abc._abc_subclasscheck}
1 0.000 0.000 0.000 0.000 <__array_function__ internals>:2(may_share_memory)
1 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:389(parent)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/_config.py:16(_get_threadlocal_config)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/shape_base.py:219(_vhstack_dispatcher)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/_asarray.py:23(asarray)
5 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/einsumfunc.py:989(_einsum_dispatcher)
1 0.000 0.000 0.000 0.000 {method 'remove' of 'list' objects}
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/shape_base.py:208(_arrays_for_stack_dispatcher)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/anaconda3/lib/python3.8/warnings.py:477(__exit__)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/sklearn/utils/validation.py:476(_ensure_no_complex_data)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/fromnumeric.py:71(<dictcomp>)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1 0.000 0.000 0.000 0.000 {method 'insert' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'rpartition' of 'str' objects}
1 0.000 0.000 0.000 0.000 {method 'copy' of 'dict' objects}
3 0.000 0.000 0.000 0.000 {built-in method _warnings._filters_mutated}
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/multiarray.py:143(concatenate)
1 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/multiarray.py:1351(may_share_memory)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/fromnumeric.py:2106(_sum_dispatcher)
1 0.000 0.000 0.000 0.000 /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages/numpy/core/shape_base.py:78(_atleast_2d_dispatcher)
I would recommend training the model on a subset of data then adding the remainder of the documents with add_documents