ludwig
ludwig copied to clipboard
Published
20 hours ago •
ludwig-ai
ludwig-ai
Ludwig serve reserves gpu memory but doesn't utilize gpu (tf-inference pip install Ludwig 0.4)
Describe the bug Using ludwig serve from the cmd , Model reserves gpu memory but does not utilize gpu at all. Although unlike https://github.com/ludwig-ai/ludwig/issues/73 , training works well on gpu (speed difference for training on gpu and cpu is significantly different) on the same instance and same environment. Inference speed remains equal on gpu or cpu and even batch size (single inference or batch of 50) seems to have no effect for inference. Also I feel the overhead for tf and ludwig is still considerable as mentioned in #73 , have we made any new optimizations for that?
Logs
2021-12-27 08:25:23.778011: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-12-27 08:25:24.398823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13793 MB memory: -> device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5 /opt/conda/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/x509.py:15: CryptographyDeprecationWarning: This version of cryptography contains a temporary pyOpenSSL fallback path. Upgrade pyOpenSSL now. warnings.warn( Downloading: 100%|██████████| 570/570 [00:00<00:00, 595kB/s] Downloading: 100%|██████████| 511M/511M [00:09<00:00, 55.4MB/s]2021-12-27 08:25:38.408429: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 93763584 exceeds 10% of free system memory. Some layers from the model checkpoint at bert-base-uncased were not used when initializing TFBertModel: ['mlm___cls', 'nsp___cls']
- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). All the layers of TFBertModel were initialized from the model checkpoint at bert-base-uncased. If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training. 2021-12-27 08:25:39.003222: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 93763584 exceeds 10% of free system memory. Downloading: 100%|██████████| 28.0/28.0 [00:00<00:00, 21.7kB/s] Downloading: 100%|██████████| 226k/226k [00:00<00:00, 315kB/s] Downloading: 100%|██████████| 455k/455k [00:00<00:00, 506kB/s] Prediction: 100%|██████████| 1/1 [00:06<00:00, 6.26s/it]
Expected behavior Faster inference and gpu utilization.
Environment (please complete the following information):
- OS: Ubuntu 18.04 Version
- Python version - 3.8.10
- Ludwig version - 0.4 tf inference (installed using pip install ludwig)
Config:
Experiment name: BERT_128_unweighted_4jan_712
Model name: bert_128
Output directory: models/results/BERT_128_unweighted_4jan_712_bert_128_0
ludwig_version: '0.4'
command: ('/usr/local/lib/python3.8/dist-packages/ipykernel_launcher.py -f '
'/root/.local/share/jupyter/runtime/kernel-8a7019b2-4606-4285-b569-e4be5dc2edc8.json')
random_seed: 42
data_format: "<class 'pandas.core.frame.DataFrame'>"
config: { 'combiner': {'type': 'concat'},
'input_features': [ { 'column': 'clean_title',
'encoder': 'bert',
'level': 'word',
'name': 'clean_title',
'pretrained_model_name_or_path': 'bert-base-uncased',
'proc_column': 'clean_title_mZFLky',
'tied': None,
'type': 'text'}],
'output_features': [ { 'column': 'category',
'dependencies': [],
'loss': { 'class_similarities_temperature': 0,
'class_weights': 1,
'confidence_penalty': 0,
'labels_smoothing': 0,
'robust_lambda': 0,
'type': 'softmax_cross_entropy',
'weight': 1},
'name': 'category',
'proc_column': 'category_mZFLky',
'reduce_dependencies': 'sum',
'reduce_input': 'sum',
'top_k': 3,
'type': 'category'}],
'preprocessing': { 'audio': { 'audio_feature': {'type': 'raw'},
'audio_file_length_limit_in_s': 7.5,
'in_memory': True,
'missing_value_strategy': 'backfill',
'norm': None,
'padding_value': 0},
'bag': { 'fill_value': '<UNK>',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000,
'tokenizer': 'space'},
'binary': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const'},
'category': { 'fill_value': '<UNK>',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000},
'date': { 'datetime_format': None,
'fill_value': '',
'missing_value_strategy': 'fill_with_const'},
'force_split': True,
'h3': { 'fill_value': 576495936675512319,
'missing_value_strategy': 'fill_with_const'},
'image': { 'in_memory': True,
'missing_value_strategy': 'backfill',
'num_processes': 1,
'resize_method': 'interpolate',
'scaling': 'pixel_normalization'},
'numerical': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const',
'normalization': None},
'sequence': { 'fill_value': '<UNK>',
'lowercase': True,
'missing_value_strategy': 'fill_with_const',
'most_common': 20000,
'padding': 'right',
'padding_symbol': '<PAD>',
'sequence_length_limit': 128,
'tokenizer': 'space',
'unknown_symbol': '<UNK>',
'vocab_file': None},
'set': { 'fill_value': '<UNK>',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000,
'tokenizer': 'space'},
'split_probabilities': [0.8, 0.1, 0.1],
'stratify': 'category',
'text': { 'char_most_common': 70,
'char_sequence_length_limit': 1024,
'char_tokenizer': 'characters',
'char_vocab_file': None,
'fill_value': '<UNK>',
'lowercase': True,
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_symbol': '<PAD>',
'pretrained_model_name_or_path': None,
'unknown_symbol': '<UNK>',
'word_most_common': 20000,
'word_sequence_length_limit': 128,
'word_tokenizer': 'space_punct',
'word_vocab_file': None},
'timeseries': { 'fill_value': '',
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_value': 0,
'timeseries_length_limit': 256,
'tokenizer': 'space'},
'vector': { 'fill_value': '',
'missing_value_strategy': 'fill_with_const'}},
'training': { 'batch_size': 64,
'bucketing_field': None,
'decay': True,
'decay_rate': 0.96,
'decay_steps': 10000,
'early_stop': 5,
'epochs': 10,
'eval_batch_size': 0,
'gradient_clipping': None,
'increase_batch_size_on_plateau': 0,
'increase_batch_size_on_plateau_max': 512,
'increase_batch_size_on_plateau_patience': 5,
'increase_batch_size_on_plateau_rate': 2,
'learning_rate': 5e-05,
'learning_rate_warmup_epochs': 1,
'optimizer': { 'beta_1': 0.9,
'beta_2': 0.999,
'epsilon': 1e-08,
'type': 'adam'},
'reduce_learning_rate_on_plateau': 0,
'reduce_learning_rate_on_plateau_patience': 5,
'reduce_learning_rate_on_plateau_rate': 0.5,
'regularization_lambda': 0,
'staircase': False,
'trainable': True,
'validation_field': 'combined',
'validation_metric': 'loss'}}
tf_version: '2.7.0'
Using training dataframe
Building dataset (it may take a while)
Writing preprocessed training set cache
Writing preprocessed test set cache
Writing preprocessed validation set cache
Writing train set metadata
Training set: 83648
Validation set: 10525
Test set: 10274
Sure @tgaddair I am trying to deploy a fine tuned bert model to production on gpu trained with ludwig. I am facing the following issues:
- Ludwig serve is very slow and has an inference speed of 7-8 secs on cpu and gpu. I have tried to deploy this model in the same environment (docker container) in which this model was trained.
- Batch size has no effect on the speed of inference on gpu. As we can see in the logs , model consumes significant amount of gpu(t4) memory but the gpu utilization is found to be zero.
- I have already deployed hf transformers (bert) before in similiar gpu environment and have had an inference time of 300-400ms max
I also feel there is significant communication overhead between ludwig and tf when using ludwig for inference - this could be due to the variety of inputs that handled by ludwig (sound,text,image etc - just my speculation)
alternatives that i am currently considering but still having issues with :
- Exporting ludwig to Savedmodel and inferencing in tf - As we know that ludwig handles the pre-processing and post-processing of data , there are no clear steps or tutorial of how we can port and use the trained model for inference in tf (Here I can handle gpu inferencing myself). Even after deep diving I was able to use the original tokenizer to get output from savedmodel but the output has different mappings to what i expect and was unable to figure out how to get it back to original output like ludwig.
- Exporting to neuropod - This is supposed to handle pre-processing and post-processing of data , and has gpu support which I can use for inferencing in production. Lack of documentation and any tutorial (even possible deprecation of ludwig support -> https://github.com/uber/neuropod/commit/5726bfebbaf742c7f8569eb9d9927a1165742f94 as mentioned in legacy tf-inference ludwig integration-tests -> https://github.com/ludwig-ai/ludwig/blob/tf-inference/tests/integration_tests/test_neuropod.py, neuropod doesn't load ludwig model without for _always_use_native=False ) has made difficult to actually get any output from this path.
Hi @smiraldr,
I was able to get a working setup with a similar BERT-based text classification model on Ray using master (torch), after fixing a few issues, namely https://github.com/ludwig-ai/ludwig/pull/1691.
We've made many GPU-related improvements to torch/master so it's quite possible that the latency is an issue that we'll only exists on tf-legacy. I'll have an update about this later today or tomorrow.
If we see the same GPU non-utilization behavior on torch, then we'll invest more time into a fix. If not, we'll recommend moving off of tf-legacy.
I trained and deployed a couple of different huggingface-bert text classification models on GPUs with ray and ludwig serve and measured inference times.
I've observed that the very first call to a fresh server is quite slow (7-8 seconds), but subsequent calls to the server execute rather quickly, around 200-400 ms. Batch prediction exhibits the same behavior, with a batch size of 20 taking around 500 ms, after the first call.
Batch predict:
time curl http://0.0.0.0:8000/batch_predict -X POST -F 'dataset={"columns": ["description"], "data": [["\"Blame @xaiax, Inspired by @MakingInvisible, using cmu phonetic data to produce incongruous matches."], ["Some images via Lorem Flickr.\""], ["Photographing the American West since 1980. I specialize in location portraits & events, both indoors & outside, using natural light & portable studio lighting."], ["\"Scruffy looking nerf herder and @twitch broadcaster"], ["Contact \ud83d\udcec [email protected]\""], ["Wife.Godmother.Friend.Feline Fanatic! Assistant Principal, Vestavia Hills Elementary Liberty Park"], ["Loan coach at @mancity & Aspiring DJ"], ["\u0644\u0627 \u0625\u0644\u0647 \u0625\u0644\u0627 \u0627\u0644\u0644\u0647 . \u0641\u0639\u0644 \u0627\u0644\u062a\u0646\u0628\u064a\u0647\u0627\u062a \ud83c\udf37\ud83c\udf96\u0639\u0627\u062f\u0644 ...\u062d\u0633\u0627\u0628\u064a \u0627\u0644\u0622\u062e\u0631 @__o0h"], ["Co-Host of @TheTalkCBS, Judge on @DancingABC, Founder https://t.co/qYbW5GlP0C, Dancer, Choreographer, Animal rescuer/lover. Instagram: @CarrieAnnInaba"], ["Internationally-acclaimed Egyptian Singer."], ["Man Utd fan. mostly here for football. Takes photos. Ex care worker, does stuff with computers often sarcastic. \ud83d\udc1d\ud83c\uddec\ud83c\udde7\ud83c\uddea\ud83c\uddfa"], ["Stay hungry, Stay foolish."], ["micropoetry bot \u00b6 license: https://t.co/vuvanDSlgC \u00b6 also on mastodon: https://t.co/Xy5L4acJva"], ["Fashion and style news(+occasional sport) from Hilary Alexander"], ["soon"], ["Femminista, animalista, antiproibizionista... E altre -ista che ora non ricordo. Ora dirigo il sito de #LeIene \ud83d\udc4a\ud83c\udfff"], ["\"Mi viene da vomitare. "], [" "], ["Dove sono le mie Jordan?\""], ["amfolowback"]]}'
{"index":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19],"columns":["account_type_predictions","account_type_probabilities","account_type_probability","account_type_probabilities_<UNK>","account_type_probabilities_human","account_type_probabilities_bot"],"data":[["human",[0.011442052200436592,0.7697554230690002,0.21880251169204712],0.7697554230690002,0.011442052200436592,0.7697554230690002,0.21880251169204712],["human",[0.011442422866821289,0.7697575688362122,0.21880005300045013],0.7697575688362122,0.011442422866821289,0.7697575688362122,0.21880005300045013],["human",[0.011442478746175766,0.7697569727897644,0.21880054473876953],0.7697569727897644,0.011442478746175766,0.7697569727897644,0.21880054473876953],["human",[0.011442234739661217,0.7697573900222778,0.21880042552947998],0.7697573900222778,0.011442234739661217,0.7697573900222778,0.21880042552947998],["human",[0.011442234739661217,0.7697566151618958,0.2188011258840561],0.7697566151618958,0.011442234739661217,0.7697566151618958,0.2188011258840561],["human",[0.011442418210208416,0.7697583436965942,0.21879926323890686],0.7697583436965942,0.011442418210208416,0.7697583436965942,0.21879926323890686],["human",[0.011442265473306179,0.7697564959526062,0.21880124509334564],0.7697564959526062,0.011442265473306179,0.7697564959526062,0.21880124509334564],["human",[0.011442516930401325,0.7697587609291077,0.21879874169826508],0.7697587609291077,0.011442516930401325,0.7697587609291077,0.21879874169826508],["human",[0.01144131924957037,0.7697508931159973,0.2188078761100769],0.7697508931159973,0.01144131924957037,0.7697508931159973,0.2188078761100769],["human",[0.011442331597208977,0.7697573304176331,0.2188004106283188],0.7697573304176331,0.011442331597208977,0.7697573304176331,0.2188004106283188],["human",[0.011442277580499649,0.7697573304176331,0.2188004106283188],0.7697573304176331,0.011442277580499649,0.7697573304176331,0.2188004106283188],["human",[0.011442305520176888,0.7697566151618958,0.21880115568637848],0.7697566151618958,0.011442305520176888,0.7697566151618958,0.21880115568637848],["human",[0.011442414484918118,0.7697566747665405,0.21880091726779938],0.7697566747665405,0.011442414484918118,0.7697566747665405,0.21880091726779938],["human",[0.011442373506724834,0.7697571516036987,0.21880044043064117],0.7697571516036987,0.011442373506724834,0.7697571516036987,0.21880044043064117],["human",[0.011442414484918118,0.7697573900222778,0.21880024671554565],0.7697573900222778,0.011442414484918118,0.7697573900222778,0.21880024671554565],["human",[0.011442380025982857,0.7697583436965942,0.21879921853542328],0.7697583436965942,0.011442380025982857,0.7697583436965942,0.21879921853542328],["human",[0.011442485265433788,0.7697581052780151,0.21879948675632477],0.7697581052780151,0.011442485265433788,0.7697581052780151,0.21879948675632477],["human",[0.01144240889698267,0.7697570323944092,0.2188006043434143],0.7697570323944092,0.01144240889698267,0.7697570323944092,0.2188006043434143],["human",[0.011442501097917557,0.7697585225105286,0.21879902482032776],0.7697585225105286,0.011442501097917557,0.7697585225105286,0.21879902482032776],["human",[0.011442499235272408,0.7697575688362122,0.21879993379116058],0.7697575688362122,0.011442499235272408,0.7697575688362122,0.21879993379116058]]}
real 0m0.502s
user 0m0.008s
sys 0m0.000s
Single predict:
(base) ray@ip-172-31-9-204:/ludwig$ time curl http://0.0.0.0:8000/predict -X POST -F 'description=Best twitter profile ever'
{"account_type_predictions":"human","account_type_probabilities":[0.011442153714597225,0.7697559595108032,0.218801811337471],"account_type_probability":0.7697559595108032,"account_type_probabilities_<UNK>":0.011442153714597225,"account_type_probabilities_human":0.7697559595108032,"account_type_probabilities_bot":0.218801811337471}
real 0m7.518s
user 0m0.004s
sys 0m0.004s
(base) ray@ip-172-31-9-204:/ludwig$ time curl http://0.0.0.0:8000/predict -X POST -F 'description=Best twitter profile ever'
{"account_type_predictions":"human","account_type_probabilities":[0.011442153714597225,0.7697559595108032,0.218801811337471],"account_type_probability":0.7697559595108032,"account_type_probabilities_<UNK>":0.011442153714597225,"account_type_probabilities_human":0.7697559595108032,"account_type_probabilities_bot":0.218801811337471}
real 0m0.267s
user 0m0.005s
sys 0m0.005s
(base) ray@ip-172-31-9-204:/ludwig$ time curl http://0.0.0.0:8000/predict -X POST -F 'description=Best twitter profile ever and ever'
{"account_type_predictions":"human","account_type_probabilities":[0.01144220121204853,0.7697561979293823,0.21880164742469788],"account_type_probability":0.7697561979293823,"account_type_probabilities_<UNK>":0.01144220121204853,"account_type_probabilities_human":0.7697561979293823,"account_type_probabilities_bot":0.21880164742469788}
real 0m0.267s
user 0m0.004s
sys 0m0.004s
(base) ray@ip-172-31-9-204:/ludwig$ time curl http://0.0.0.0:8000/predict -X POST -F 'description=New York is the best place for food'
{"account_type_predictions":"human","account_type_probabilities":[0.011442417278885841,0.7697572112083435,0.2188003957271576],"account_type_probability":0.7697572112083435,"account_type_probabilities_<UNK>":0.011442417278885841,"account_type_probabilities_human":0.7697572112083435,"account_type_probabilities_bot":0.2188003957271576}
real 0m0.264s
user 0m0.000s
sys 0m0.008s
I see the same behavior on torch, with similar cold-start and warmed-up latency times.
@smiraldr, could you check to see that you are also able to get substantially faster latency times on subsequent calls to a fresh server?
Going forward, it's probably useful to add warm-up queries as part of standing up the server to get low latency times right away.
@justinxzhao I am aware about the inital 7-8 sec latency but I can confirm that im getting high latency in the subsequent requests as well. The only difference in your experiment is the use of ray for model training, my ludwig models were trained on pandas as dataset format on default backend, will try this setup once with my models and let you know. Just for verification it would be great if you could try ludwig serve with a sample model i've trained. (https://drive.google.com/drive/folders/1OAyhisFihZFZ6AT_t2ygrTph2TScpwaQ?usp=sharing)
Here I've used ludwig serve with the model posted above with same time curl for batch predict:
real 0m15.404s user 0m0.008s sys 0m0.002s
real 0m8.901s user 0m0.005s sys 0m0.005s
Here is my nvidia-smi output for my system info:
nvidia-smiThu Jan 20 10:38:42 2022 +-----------------------------------------------------------------------------+| NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 ||-------------------------------+----------------------+----------------------+| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC || Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. || | | MIG M. ||===============================+======================+======================|| 0 Tesla T4 On | 00000000:00:1E.0 Off | 0 || N/A 31C P0 26W / 70W | 620MiB / 15109MiB | 0% Default || | | N/A |+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 263047 C /usr/bin/python3 617MiB |
+-----------------------------------------------------------------------------+
Also incase we feel that this is a system config issue i've deployed ludwig in a containerized environement with all necessary nvidia drivers to aws eks gpu ami pods where I am getting the same inference speed.
I used this one as the sample for model posted:
time curl http://0.0.0.0:8000/batch_predict -X POST -F 'dataset={"columns": ["clean_title"], "data": [["\"Blame @xaiax, Inspired by @MakingInvisible, using cmu phonetic data to produce incongruous matches."], ["Some images via Lorem Flickr.\""], ["Photographing the American West since 1980. I specialize in location portraits & events, both indoors & outside, using natural light & portable studio lighting."], ["\"Scruffy looking nerf herder and @twitch broadcaster"], ["Contact \ud83d\udcec [email protected]\""], ["Wife.Godmother.Friend.Feline Fanatic! Assistant Principal, Vestavia Hills Elementary Liberty Park"], ["Loan coach at @mancity & Aspiring DJ"], ["\u0644\u0627 \u0625\u0644\u0647 \u0625\u0644\u0627 \u0627\u0644\u0644\u0647 . \u0641\u0639\u0644 \u0627\u0644\u062a\u0646\u0628\u064a\u0647\u0627\u062a \ud83c\udf37\ud83c\udf96\u0639\u0627\u062f\u0644 ...\u062d\u0633\u0627\u0628\u064a \u0627\u0644\u0622\u062e\u0631 @__o0h"], ["Co-Host of @TheTalkCBS, Judge on @DancingABC, Founder https://t.co/qYbW5GlP0C, Dancer, Choreographer, Animal rescuer/lover. Instagram: @CarrieAnnInaba"], ["Internationally-acclaimed Egyptian Singer."], ["Man Utd fan. mostly here for football. Takes photos. Ex care worker, does stuff with computers often sarcastic. \ud83d\udc1d\ud83c\uddec\ud83c\udde7\ud83c\uddea\ud83c\uddfa"], ["Stay hungry, Stay foolish."], ["micropoetry bot \u00b6 license: https://t.co/vuvanDSlgC \u00b6 also on mastodon: https://t.co/Xy5L4acJva"], ["Fashion and style news(+occasional sport) from Hilary Alexander"], ["soon"], ["Femminista, animalista, antiproibizionista... E altre -ista che ora non ricordo. Ora dirigo il sito de #LeIene \ud83d\udc4a\ud83c\udfff"], ["\"Mi viene da vomitare. "], [" "], ["Dove sono le mie Jordan?\""], ["amfolowback"]]}'
{"index":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19],"columns":["account_type_predictions","account_type_probabilities","account_type_probability","account_type_probabilities_<UNK>","account_type_probabilities_human","account_type_probabilities_bot"],"data":[["human",[0.011442052200436592,0.7697554230690002,0.21880251169204712],0.7697554230690002,0.011442052200436592,0.7697554230690002,0.21880251169204712],["human",[0.011442422866821289,0.7697575688362122,0.21880005300045013],0.7697575688362122,0.011442422866821289,0.7697575688362122,0.21880005300045013],["human",[0.011442478746175766,0.7697569727897644,0.21880054473876953],0.7697569727897644,0.011442478746175766,0.7697569727897644,0.21880054473876953],["human",[0.011442234739661217,0.7697573900222778,0.21880042552947998],0.7697573900222778,0.011442234739661217,0.7697573900222778,0.21880042552947998],["human",[0.011442234739661217,0.7697566151618958,0.2188011258840561],0.7697566151618958,0.011442234739661217,0.7697566151618958,0.2188011258840561],["human",[0.011442418210208416,0.7697583436965942,0.21879926323890686],0.7697583436965942,0.011442418210208416,0.7697583436965942,0.21879926323890686],["human",[0.011442265473306179,0.7697564959526062,0.21880124509334564],0.7697564959526062,0.011442265473306179,0.7697564959526062,0.21880124509334564],["human",[0.011442516930401325,0.7697587609291077,0.21879874169826508],0.7697587609291077,0.011442516930401325,0.7697587609291077,0.21879874169826508],["human",[0.01144131924957037,0.7697508931159973,0.2188078761100769],0.7697508931159973,0.01144131924957037,0.7697508931159973,0.2188078761100769],["human",[0.011442331597208977,0.7697573304176331,0.2188004106283188],0.7697573304176331,0.011442331597208977,0.7697573304176331,0.2188004106283188],["human",[0.011442277580499649,0.7697573304176331,0.2188004106283188],0.7697573304176331,0.011442277580499649,0.7697573304176331,0.2188004106283188],["human",[0.011442305520176888,0.7697566151618958,0.21880115568637848],0.7697566151618958,0.011442305520176888,0.7697566151618958,0.21880115568637848],["human",[0.011442414484918118,0.7697566747665405,0.21880091726779938],0.7697566747665405,0.011442414484918118,0.7697566747665405,0.21880091726779938],["human",[0.011442373506724834,0.7697571516036987,0.21880044043064117],0.7697571516036987,0.011442373506724834,0.7697571516036987,0.21880044043064117],["human",[0.011442414484918118,0.7697573900222778,0.21880024671554565],0.7697573900222778,0.011442414484918118,0.7697573900222778,0.21880024671554565],["human",[0.011442380025982857,0.7697583436965942,0.21879921853542328],0.7697583436965942,0.011442380025982857,0.7697583436965942,0.21879921853542328],["human",[0.011442485265433788,0.7697581052780151,0.21879948675632477],0.7697581052780151,0.011442485265433788,0.7697581052780151,0.21879948675632477],["human",[0.01144240889698267,0.7697570323944092,0.2188006043434143],0.7697570323944092,0.01144240889698267,0.7697570323944092,0.2188006043434143],["human",[0.011442501097917557,0.7697585225105286,0.21879902482032776],0.7697585225105286,0.011442501097917557,0.7697585225105286,0.21879902482032776],["human",[0.011442499235272408,0.7697575688362122,0.21879993379116058],0.7697575688362122,0.011442499235272408,0.7697575688362122,0.21879993379116058]]}
I have trained a model and tried to use ludwig serve for ludwigai/ludwig-ray-gpu:sha-89d1836
`(base) ray@8d674c62190d:/ludwig$ time curl http://0.0.0.0:8000/batch_predict -X POST -F 'dataset={"columns": ["clean_title"], "data": [["\"Blame @xaiax, Inspired by @MakingInvisible, using cmu phonetic data to produce incongruous matches."], ["Some images via Lorem Flickr.\""], ["Photographing the American West since 1980. I specialize in location portraits & events, both indoors & outside, using natural light & portable studio lighting."], ["\"Scruffy looking nerf herder and @twitch broadcaster"], ["Contact \ud83d\udcec [email protected]\""], ["Wife.Godmother.Friend.Feline Fanatic! Assistant Principal, Vestavia Hills Elementary Liberty Park"], ["Loan coach at @mancity & Aspiring DJ"], ["\u0644\u0627 \u0625\u0644\u0647 \u0625\u0644\u0627 \u0627\u0644\u0644\u0647 . \u0641\u0639\u0644 \u0627\u0644\u062a\u0646\u0628\u064a\u0647\u0627\u062a \ud83c\udf37\ud83c\udf96\u0639\u0627\u062f\u0644 ...\u062d\u0633\u0627\u0628\u064a \u0627\u0644\u0622\u062e\u0631 @__o0h"], ["Co-Host of @TheTalkCBS, Judge on @DancingABC, Founder https://t.co/qYbW5GlP0C, Dancer, Choreographer, Animal rescuer/lover. Instagram: @CarrieAnnInaba"], ["Internationally-acclaimed Egyptian Singer."], ["Man Utd fan. mostly here for football. Takes photos. Ex care worker, does stuff with computers often sarcastic. \ud83d\udc1d\ud83c\uddec\ud83c\udde7\ud83c\uddea\ud83c\uddfa"], ["Stay hungry, Stay foolish."], ["micropoetry bot \u00b6 license: https://t.co/vuvanDSlgC \u00b6 also on mastodon: https://t.co/Xy5L4acJva"], ["Fashion and style news(+occasional sport) from Hilary Alexander"], ["soon"], ["Femminista, animalista, antiproibizionista... E altre -ista che ora non ricordo. Ora dirigo il sito de #LeIene \ud83d\udc4a\ud83c\udfff"], ["\"Mi viene da vomitare. "], [" "], ["Dove sono le mie Jordan?\""], ["amfolowback"]]}'
{"index":[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19],"columns":["category_predictions","category_probabilities","category_probability","category_probabilities_Politics","category_probabilities_Life","category_probabilities_Religious","category_probabilities_<UNK>","category_probabilities_Occassion","category_probabilities_Governance","category_probabilities_Love","category_probabilities_Society","category_probabilities_Personal","category_probabilities_Covid","category_probabilities_Sports","category_probabilities_Lifestyle","category_probabilities_Entertainment","category_probabilities_Daily","category_probabilities_Technology","category_probabilities_Crime","category_probabilities_Wishes","category_probabilities_Up Elections","category_probabilities_Business","category_probabilities_Environment","category_probabilities_Motivation","category_probabilities_Education","category_probabilities_Others","category_probabilities_Patriotic","category_probabilities_Accident","category_probabilities_Defence","category_probabilities_Literature"],"data":[["Business",[0.015386789105832577,0.01703220047056675,0.00011299122706986964,2.372478602463701e-10,0.00016026220691855997,0.0013723885640501976,0.0017946403240785003,0.009639977477490902,0.11924412101507187,0.017161261290311813,0.23283083736896515,0.0065123517997562885,0.044030122458934784,0.010641652159392834,0.09573473036289215,1.4062178706808481e-05,0.0008689211099408567,0.0006638766499236226,0.3647991418838501,0.0006315859500318766,0.001594710280187428,0.029117055237293243,0.029376348480582237,9.581725316820666e-05,4.231247032748797e-07,0.0006753873312845826,0.0005083681899122894],0.3647991418838501,0.015386789105832577,0.01703220047056675,0.00011299122706986964,2.372478602463701e-10,0.00016026220691855997,0.0013723885640501976,0.0017946403240785003,0.009639977477490902,0.11924412101507187,0.017161261290311813,0.23283083736896515,0.0065123517997562885,0.044030122458934784,0.010641652159392834,0.09573473036289215,1.4062178706808481e-05,0.0008689211099408567,0.0006638766499236226,0.3647991418838501,0.0006315859500318766,0.001594710280187428,0.029117055237293243,0.029376348480582237,9.581725316820666e-05,4.231247032748797e-07,0.0006753873312845826,0.0005083681899122894],["Personal",[0.11034643650054932,0.0638287141919136,0.011411873623728752,6.84515388815754e-10,0.03115183487534523,0.02328626625239849,0.01386620756238699,0.04739256948232651,0.3322354555130005,0.005323889199644327,0.06419036537408829,0.025236355140805244,0.05819571390748024,0.009920960292220116,0.04527776315808296,0.0023493303451687098,0.009386242367327213,0.00663850037381053,0.018928375095129013,0.0022469800896942616,0.0063302419148385525,0.008718006312847137,0.0921110212802887,0.005052013322710991,0.00030677314498461783,0.003977716434746981,0.002290343167260289],0.3322354555130005,0.11034643650054932,0.0638287141919136,0.011411873623728752,6.84515388815754e-10,0.03115183487534523,0.02328626625239849,0.01386620756238699,0.04739256948232651,0.3322354555130005,0.005323889199644327,0.06419036537408829,0.025236355140805244,0.05819571390748024,0.009920960292220116,0.04527776315808296,0.0023493303451687098,0.009386242367327213,0.00663850037381053,0.018928375095129013,0.0022469800896942616,0.0063302419148385525,0.008718006312847137,0.0921110212802887,0.005052013322710991,0.00030677314498461783,0.003977716434746981,0.002290343167260289],["Personal",[0.08146890997886658,0.010483277030289173,0.00153391994535923,1.8486833175401785e-09,0.03999938443303108,0.016369538381695747,0.0011502396082505584,0.053555913269519806,0.17999444901943207,0.01513746753334999,0.057623203843832016,0.06512518227100372,0.11517660319805145,0.010915324091911316,0.12632177770137787,0.00024191231932491064,0.008291730657219887,0.0027832244522869587,0.0981234535574913,0.001995154656469822,0.005750773474574089,0.021984098479151726,0.07406023889780045,0.003528898349031806,8.763798177824356e-06,0.005506580695509911,0.0028699692338705063],0.17999444901943207,0.08146890997886658,0.010483277030289173,0.00153391994535923,1.8486833175401785e-09,0.03999938443303108,0.016369538381695747,0.0011502396082505584,0.053555913269519806,0.17999444901943207,0.01513746753334999,0.057623203843832016,0.06512518227100372,0.11517660319805145,0.010915324091911316,0.12632177770137787,0.00024191231932491064,0.008291730657219887,0.0027832244522869587,0.0981234535574913,0.001995154656469822,0.005750773474574089,0.021984098479151726,0.07406023889780045,0.003528898349031806,8.763798177824356e-06,0.005506580695509911,0.0028699692338705063],["Personal",[0.044849101454019547,0.06522180885076523,0.008527419529855251,6.131390950514515e-10,0.03716127574443817,0.00747926440089941,0.01321093738079071,0.03168867900967598,0.38244062662124634,0.011589424684643745,0.06167714297771454,0.04847592115402222,0.05822913721203804,0.020824598148465157,0.06493888050317764,0.0005146617186255753,0.007757126819342375,0.0010654787765815854,0.014363054186105728,0.003976409789174795,0.011617370881140232,0.007026127073913813,0.0898180902004242,0.0029547237791121006,0.00015367283776868135,0.002351096598431468,0.0020879642106592655],0.38244062662124634,0.044849101454019547,0.06522180885076523,0.008527419529855251,6.131390950514515e-10,0.03716127574443817,0.00747926440089941,0.01321093738079071,0.03168867900967598,0.38244062662124634,0.011589424684643745,0.06167714297771454,0.04847592115402222,0.05822913721203804,0.020824598148465157,0.06493888050317764,0.0005146617186255753,0.007757126819342375,0.0010654787765815854,0.014363054186105728,0.003976409789174795,0.011617370881140232,0.007026127073913813,0.0898180902004242,0.0029547237791121006,0.00015367283776868135,0.002351096598431468,0.0020879642106592655],["Personal",[0.02848409302532673,0.04069185629487038,0.018072953447699547,6.684272579882133e-10,0.007140201050788164,0.01897851936519146,0.0026235724799335003,0.10251262784004211,0.4367724061012268,0.001819672412239015,0.007281921803951263,0.03505050018429756,0.01547456905245781,0.007296254858374596,0.08498993515968323,0.00040612308657728136,0.003894519293680787,0.0024917887058109045,0.023105941712856293,0.00045860957470722497,0.004482207354158163,0.04357537627220154,0.09734924882650375,0.003981417044997215,3.6355337215354666e-05,0.0009075540583580732,0.012121761217713356],0.4367724061012268,0.02848409302532673,0.04069185629487038,0.018072953447699547,6.684272579882133e-10,0.007140201050788164,0.01897851936519146,0.0026235724799335003,0.10251262784004211,0.4367724061012268,0.001819672412239015,0.007281921803951263,0.03505050018429756,0.01547456905245781,0.007296254858374596,0.08498993515968323,0.00040612308657728136,0.003894519293680787,0.0024917887058109045,0.023105941712856293,0.00045860957470722497,0.004482207354158163,0.04357537627220154,0.09734924882650375,0.003981417044997215,3.6355337215354666e-05,0.0009075540583580732,0.012121761217713356],["Politics",[0.31808143854141235,0.0163495521992445,0.03757079318165779,1.0238388609140614e-10,0.00328794470988214,0.09078020602464676,0.00458951061591506,0.13637161254882812,0.1287156641483307,0.003460746956989169,0.0008784598903730512,0.023223239928483963,0.03308425843715668,7.135207852115855e-05,0.007443105801939964,0.10503877699375153,0.0060997651889920235,0.014559760689735413,0.006892054341733456,0.0017273282865062356,0.000244837166974321,0.007392305880784988,0.043444931507110596,0.0019485116936266422,0.005273555405437946,0.0011803711531683803,0.002289943629875779],0.31808143854141235,0.31808143854141235,0.0163495521992445,0.03757079318165779,1.0238388609140614e-10,0.00328794470988214,0.09078020602464676,0.00458951061591506,0.13637161254882812,0.1287156641483307,0.003460746956989169,0.0008784598903730512,0.023223239928483963,0.03308425843715668,7.135207852115855e-05,0.007443105801939964,0.10503877699375153,0.0060997651889920235,0.014559760689735413,0.006892054341733456,0.0017273282865062356,0.000244837166974321,0.007392305880784988,0.043444931507110596,0.0019485116936266422,0.005273555405437946,0.0011803711531683803,0.002289943629875779],["Others",[0.07124420255422592,0.015647470951080322,0.00785285048186779,4.701553035779682e-10,0.0037962417118251324,0.057358838617801666,0.005611460655927658,0.010619701817631721,0.11561738699674606,0.019321409985423088,0.07579614967107773,0.069851353764534,0.05423692241311073,0.006159496959298849,0.05158674344420433,0.005701480899006128,0.0015607852255925536,0.0010666747111827135,0.1343025267124176,0.050607409328222275,0.0002523927250877023,0.013271583244204521,0.2151339054107666,0.0012648190604522824,0.0034690385218709707,0.008230513893067837,0.00043869524961337447],0.2151339054107666,0.07124420255422592,0.015647470951080322,0.00785285048186779,4.701553035779682e-10,0.0037962417118251324,0.057358838617801666,0.005611460655927658,0.010619701817631721,0.11561738699674606,0.019321409985423088,0.07579614967107773,0.069851353764534,0.05423692241311073,0.006159496959298849,0.05158674344420433,0.005701480899006128,0.0015607852255925536,0.0010666747111827135,0.1343025267124176,0.050607409328222275,0.0002523927250877023,0.013271583244204521,0.2151339054107666,0.0012648190604522824,0.0034690385218709707,0.008230513893067837,0.00043869524961337447],["Religious",[0.04974572733044624,0.15529559552669525,0.6249916553497314,1.93983246132845e-10,0.01699259877204895,0.0027132052928209305,0.013082485646009445,0.0359211266040802,0.023236414417624474,0.0014093875652179122,0.00045526164467446506,0.027435695752501488,0.0017663296312093735,0.011261474341154099,0.0016337225679308176,0.0001513998577138409,0.0006749792955815792,0.0019006902584806085,0.0016155503690242767,0.0013708630576729774,0.010233915410935879,0.003455098019912839,0.004084711894392967,0.00798716489225626,0.00015766487922519445,0.000361471000360325,0.002065864158794284],0.6249916553497314,0.04974572733044624,0.15529559552669525,0.6249916553497314,1.93983246132845e-10,0.01699259877204895,0.0027132052928209305,0.013082485646009445,0.0359211266040802,0.023236414417624474,0.0014093875652179122,0.00045526164467446506,0.027435695752501488,0.0017663296312093735,0.011261474341154099,0.0016337225679308176,0.0001513998577138409,0.0006749792955815792,0.0019006902584806085,0.0016155503690242767,0.0013708630576729774,0.010233915410935879,0.003455098019912839,0.004084711894392967,0.00798716489225626,0.00015766487922519445,0.000361471000360325,0.002065864158794284],["Personal",[0.0655311569571495,0.18401360511779785,0.019767316058278084,1.3292894562866309e-09,0.0172585379332304,0.012324826791882515,0.009504093788564205,0.07955258339643478,0.30961549282073975,0.0050682732835412025,0.0025702430866658688,0.02542119100689888,0.023801380768418312,0.01865152269601822,0.05891970172524452,0.0001985232811421156,0.009145262651145458,0.004594298545271158,0.05983084440231323,0.004178567323833704,0.03349835053086281,0.008998671546578407,0.037939637899398804,0.0028845411725342274,0.000121297751320526,0.0012612428981810808,0.005348900333046913],0.30961549282073975,0.0655311569571495,0.18401360511779785,0.019767316058278084,1.3292894562866309e-09,0.0172585379332304,0.012324826791882515,0.009504093788564205,0.07955258339643478,0.30961549282073975,0.0050682732835412025,0.0025702430866658688,0.02542119100689888,0.023801380768418312,0.01865152269601822,0.05891970172524452,0.0001985232811421156,0.009145262651145458,0.004594298545271158,0.05983084440231323,0.004178567323833704,0.03349835053086281,0.008998671546578407,0.037939637899398804,0.0028845411725342274,0.000121297751320526,0.0012612428981810808,0.005348900333046913],["Personal",[0.13331611454486847,0.09263188391923904,0.010411563329398632,1.7462749302143976e-10,0.0029599396511912346,0.013953900896012783,0.007654794957488775,0.06189091503620148,0.30902740359306335,0.013907522894442081,0.035490505397319794,0.03363647311925888,0.040665823966264725,0.008835984393954277,0.05765684321522713,0.001871685148216784,0.0063026295974850655,0.0025817332789301872,0.026537781581282616,0.004239215981215239,0.003877918468788266,0.016706932336091995,0.10665019601583481,0.004701138474047184,0.00034587664413265884,0.00272168917581439,0.0014235146809369326],0.30902740359306335,0.13331611454486847,0.09263188391923904,0.010411563329398632,1.7462749302143976e-10,0.0029599396511912346,0.013953900896012783,0.007654794957488775,0.06189091503620148,0.30902740359306335,0.013907522894442081,0.035490505397319794,0.03363647311925888,0.040665823966264725,0.008835984393954277,0.05765684321522713,0.001871685148216784,0.0063026295974850655,0.0025817332789301872,0.026537781581282616,0.004239215981215239,0.003877918468788266,0.016706932336091995,0.10665019601583481,0.004701138474047184,0.00034587664413265884,0.00272168917581439,0.0014235146809369326],["Technology",[0.05694687366485596,0.05057508498430252,0.0039607989601790905,9.246317045352725e-10,0.0022462469059973955,0.017332157120108604,0.013885204680263996,0.055237311869859695,0.18007409572601318,0.009139497764408588,0.007845532149076462,0.02819829434156418,0.03416833654046059,0.005056693218648434,0.34046417474746704,0.003017116105183959,0.019243838265538216,0.0020730176474899054,0.04087181016802788,0.007435115519911051,0.0029371485579758883,0.009371968917548656,0.09740903228521347,0.0035350946709513664,0.00013419312017504126,0.005811097100377083,0.0030301830265671015],0.34046417474746704,0.05694687366485596,0.05057508498430252,0.0039607989601790905,9.246317045352725e-10,0.0022462469059973955,0.017332157120108604,0.013885204680263996,0.055237311869859695,0.18007409572601318,0.009139497764408588,0.007845532149076462,0.02819829434156418,0.03416833654046059,0.005056693218648434,0.34046417474746704,0.003017116105183959,0.019243838265538216,0.0020730176474899054,0.04087181016802788,0.007435115519911051,0.0029371485579758883,0.009371968917548656,0.09740903228521347,0.0035350946709513664,0.00013419312017504126,0.005811097100377083,0.0030301830265671015],["Personal",[0.17925311625003815,0.017334233969449997,0.0334966816008091,4.5404130455395375e-10,0.011804630048573017,0.021500682458281517,0.007027176674455404,0.03415476903319359,0.188046395778656,0.008655411191284657,0.12489955872297287,0.045826662331819534,0.13545925915241241,0.00583340460434556,0.01453433372080326,0.0009822715073823929,0.001690560020506382,0.008870493620634079,0.04037803038954735,0.004024912137538195,0.004148499574512243,0.01845669187605381,0.08276260644197464,0.005086015444248915,0.0009857282275334,0.004371962510049343,0.0004159246454946697],0.188046395778656,0.17925311625003815,0.017334233969449997,0.0334966816008091,4.5404130455395375e-10,0.011804630048573017,0.021500682458281517,0.007027176674455404,0.03415476903319359,0.188046395778656,0.008655411191284657,0.12489955872297287,0.045826662331819534,0.13545925915241241,0.00583340460434556,0.01453433372080326,0.0009822715073823929,0.001690560020506382,0.008870493620634079,0.04037803038954735,0.004024912137538195,0.004148499574512243,0.01845669187605381,0.08276260644197464,0.005086015444248915,0.0009857282275334,0.004371962510049343,0.0004159246454946697],["Personal",[0.006909011397510767,0.0628538578748703,0.00626063346862793,4.3322959109026726e-10,0.0011985759483650327,0.005682818125933409,0.017602043226361275,0.022711487486958504,0.437953919172287,0.002455070847645402,0.0027007597964257,0.018727675080299377,0.04098314791917801,0.004488564562052488,0.24078702926635742,0.0003568698593880981,0.00548082310706377,0.00017676963761914521,0.06648869812488556,0.0013769158395007253,0.001275288057513535,0.0054696486331522465,0.042788367718458176,0.0018493147799745202,0.00020427408162504435,0.001256075338460505,0.0019623474217951298],0.437953919172287,0.006909011397510767,0.0628538578748703,0.00626063346862793,4.3322959109026726e-10,0.0011985759483650327,0.005682818125933409,0.017602043226361275,0.022711487486958504,0.437953919172287,0.002455070847645402,0.0027007597964257,0.018727675080299377,0.04098314791917801,0.004488564562052488,0.24078702926635742,0.0003568698593880981,0.00548082310706377,0.00017676963761914521,0.06648869812488556,0.0013769158395007253,0.001275288057513535,0.0054696486331522465,0.042788367718458176,0.0018493147799745202,0.00020427408162504435,0.001256075338460505,0.0019623474217951298],["Personal",[0.034389544278383255,0.048427335917949677,0.018171297386288643,1.1077571082651616e-09,0.027580730617046356,0.021118756383657455,0.0043109445832669735,0.08837131410837173,0.2721204161643982,0.009739474393427372,0.01585376262664795,0.05665288493037224,0.05918670818209648,0.010540924966335297,0.07272591441869736,0.001390421180985868,0.01343112625181675,0.004106380511075258,0.02765091136097908,0.003560488112270832,0.006516182795166969,0.01958397403359413,0.17093244194984436,0.004853458609431982,0.00035066285636276007,0.0032344867940992117,0.0051994831301271915],0.2721204161643982,0.034389544278383255,0.048427335917949677,0.018171297386288643,1.1077571082651616e-09,0.027580730617046356,0.021118756383657455,0.0043109445832669735,0.08837131410837173,0.2721204161643982,0.009739474393427372,0.01585376262664795,0.05665288493037224,0.05918670818209648,0.010540924966335297,0.07272591441869736,0.001390421180985868,0.01343112625181675,0.004106380511075258,0.02765091136097908,0.003560488112270832,0.006516182795166969,0.01958397403359413,0.17093244194984436,0.004853458609431982,0.00035066285636276007,0.0032344867940992117,0.0051994831301271915],["Personal",[0.02401704527437687,0.06975264847278595,0.046979960054159164,1.5501076811030856e-10,0.002687020692974329,0.0011945570586249232,0.013933095149695873,0.026858244091272354,0.38165292143821716,0.0015467380871996284,0.021275609731674194,0.042876098304986954,0.007139107212424278,0.0496034100651741,0.002825135365128517,0.00011261014878982678,0.0007397997542284429,0.0017310434486716986,0.007611921988427639,0.003647088771685958,0.0016609056619927287,0.0026640656869858503,0.2873634099960327,0.0008006145362742245,9.180985944112763e-05,0.0003741669934242964,0.0008610134827904403],0.38165292143821716,0.02401704527437687,0.06975264847278595,0.046979960054159164,1.5501076811030856e-10,0.002687020692974329,0.0011945570586249232,0.013933095149695873,0.026858244091272354,0.38165292143821716,0.0015467380871996284,0.021275609731674194,0.042876098304986954,0.007139107212424278,0.0496034100651741,0.002825135365128517,0.00011261014878982678,0.0007397997542284429,0.0017310434486716986,0.007611921988427639,0.003647088771685958,0.0016609056619927287,0.0026640656869858503,0.2873634099960327,0.0008006145362742245,9.180985944112763e-05,0.0003741669934242964,0.0008610134827904403],["Politics",[0.21223095059394836,0.18945671617984772,0.01906316913664341,4.501425898695288e-10,0.005271339789032936,0.01567271538078785,0.057085923850536346,0.05698266252875328,0.15955597162246704,0.010863534174859524,0.039870575070381165,0.035458724945783615,0.03713318333029747,0.006506608799099922,0.025727421045303345,0.0006879853317514062,0.003616643138229847,0.004919392988085747,0.00877247005701065,0.006567496340721846,0.024727776646614075,0.022514676675200462,0.03942742198705673,0.01396985538303852,9.485549526289105e-05,0.002704054582864046,0.001117895939387381],0.21223095059394836,0.21223095059394836,0.18945671617984772,0.01906316913664341,4.501425898695288e-10,0.005271339789032936,0.01567271538078785,0.057085923850536346,0.05698266252875328,0.15955597162246704,0.010863534174859524,0.039870575070381165,0.035458724945783615,0.03713318333029747,0.006506608799099922,0.025727421045303345,0.0006879853317514062,0.003616643138229847,0.004919392988085747,0.00877247005701065,0.006567496340721846,0.024727776646614075,0.022514676675200462,0.03942742198705673,0.01396985538303852,9.485549526289105e-05,0.002704054582864046,0.001117895939387381],["Personal",[0.09880542010068893,0.13245262205600739,0.035283997654914856,2.095340151386793e-10,0.019674470648169518,0.009128544479608536,0.029239995405077934,0.03978702798485756,0.21588057279586792,0.010021929629147053,0.08929992467164993,0.11819068342447281,0.03583170473575592,0.01454381737858057,0.012014285661280155,0.00044096895726397634,0.0009386934689246118,0.00261066947132349,0.017104119062423706,0.004825078882277012,0.0110172713175416,0.01150162611156702,0.08578062802553177,0.0031282473355531693,0.0007651911582797766,0.0012152711860835552,0.0005172007950022817],0.21588057279586792,0.09880542010068893,0.13245262205600739,0.035283997654914856,2.095340151386793e-10,0.019674470648169518,0.009128544479608536,0.029239995405077934,0.03978702798485756,0.21588057279586792,0.010021929629147053,0.08929992467164993,0.11819068342447281,0.03583170473575592,0.01454381737858057,0.012014285661280155,0.00044096895726397634,0.0009386934689246118,0.00261066947132349,0.017104119062423706,0.004825078882277012,0.0110172713175416,0.01150162611156702,0.08578062802553177,0.0031282473355531693,0.0007651911582797766,0.0012152711860835552,0.0005172007950022817],["Others",[0.021595489233732224,0.03807642310857773,0.03388509154319763,1.467381216535557e-10,0.001439449260942638,0.0022465926595032215,0.0038624515291303396,0.02585119754076004,0.18548822402954102,0.0006981634069234133,0.014912690035998821,0.022241946309804916,0.00578704196959734,0.01298671867698431,0.0011152385268360376,0.0002509073819965124,0.0006123724160715938,0.002463065553456545,0.003964530769735575,0.0017461852403357625,0.0009366318699903786,0.0019724019803106785,0.6162986755371094,0.00033525138860568404,5.234919808572158e-05,0.0002910990151576698,0.0008897390798665583],0.6162986755371094,0.021595489233732224,0.03807642310857773,0.03388509154319763,1.467381216535557e-10,0.001439449260942638,0.0022465926595032215,0.0038624515291303396,0.02585119754076004,0.18548822402954102,0.0006981634069234133,0.014912690035998821,0.022241946309804916,0.00578704196959734,0.01298671867698431,0.0011152385268360376,0.0002509073819965124,0.0006123724160715938,0.002463065553456545,0.003964530769735575,0.0017461852403357625,0.0009366318699903786,0.0019724019803106785,0.6162986755371094,0.00033525138860568404,5.234919808572158e-05,0.0002910990151576698,0.0008897390798665583],["Personal",[0.09333116561174393,0.13470837473869324,0.05593375861644745,8.283754238114227e-10,0.027683375403285027,0.00742826983332634,0.03358197584748268,0.04195341095328331,0.21320563554763794,0.004122791346162558,0.09430955350399017,0.05759508162736893,0.06550431996583939,0.04350877180695534,0.011902217753231525,0.0002627517969813198,0.002881949534639716,0.0013249219628050923,0.00818016566336155,0.0060113887302577496,0.012678250670433044,0.007907996885478497,0.06759226322174072,0.005277949385344982,0.00044725631596520543,0.0020106686279177666,0.0006556762964464724],0.21320563554763794,0.09333116561174393,0.13470837473869324,0.05593375861644745,8.283754238114227e-10,0.027683375403285027,0.00742826983332634,0.03358197584748268,0.04195341095328331,0.21320563554763794,0.004122791346162558,0.09430955350399017,0.05759508162736893,0.06550431996583939,0.04350877180695534,0.011902217753231525,0.0002627517969813198,0.002881949534639716,0.0013249219628050923,0.00818016566336155,0.0060113887302577496,0.012678250670433044,0.007907996885478497,0.06759226322174072,0.005277949385344982,0.00044725631596520543,0.0020106686279177666,0.0006556762964464724],["Sports",[0.04698852822184563,0.01843077875673771,0.005428779404610395,8.303837617518184e-10,0.006925484631210566,0.014472984708845615,0.0033424508292227983,0.012185703031718731,0.1325760781764984,0.035293545573949814,0.20493485033512115,0.0558832623064518,0.028647568076848984,0.021277746185660362,0.08022530376911163,0.000875011261086911,0.000986237428151071,0.0030560807790607214,0.11164052784442902,0.015706973150372505,0.001566136721521616,0.022521978244185448,0.17144401371479034,0.0010448915418237448,0.00036131119122728705,0.0034802891314029694,0.0007035211892798543],0.20493485033512115,0.04698852822184563,0.01843077875673771,0.005428779404610395,8.303837617518184e-10,0.006925484631210566,0.014472984708845615,0.0033424508292227983,0.012185703031718731,0.1325760781764984,0.035293545573949814,0.20493485033512115,0.0558832623064518,0.028647568076848984,0.021277746185660362,0.08022530376911163,0.000875011261086911,0.000986237428151071,0.0030560807790607214,0.11164052784442902,0.015706973150372505,0.001566136721521616,0.022521978244185448,0.17144401371479034,0.0010448915418237448,0.00036131119122728705,0.0034802891314029694,0.0007035211892798543]]}
real 0m11.735s
user 0m0.005s
sys 0m0.004s`
Have the same latencies for subsequent requests as well.
@justinxzhao would you mind posting the model config and system config you've used for training and serving?
@smiraldr and I met to discuss this issue in a bit more detail. We didn't reach any hard conclusions, but here were our observations.
-
Our dataset sequence lengths were about the same. Smiral's data uses marginally more categories, but overall data alone doesn't explain the latency increase.
-
I was not able to serve Smiral's ludwig model from my Ray cluster with a TF image. This is likely due to different backends that are chosen automatically based whether or not Ray is in the environment. Smiral's model was trained and saved in a non-Ray environment. Running
ludwig servewith his model on my ray cluster worked, but fails at runtime with the following error:
INFO: 127.0.0.1:50450 - "POST /batch_predict HTTP/1.1" 500 Internal Server Error
Failed to parse batch_predict form
Traceback (most recent call last):
File "/home/ubuntu/ludwig/ludwig/serve.py", line 95, in batch_predict
form = await request.form()
File "/home/ray/anaconda3/lib/python3.7/site-packages/starlette/requests.py", line 240, in form
self._form = await multipart_parser.parse()
File "/home/ray/anaconda3/lib/python3.7/site-packages/starlette/formparsers.py", line 195, in parse
async for chunk in self.stream:
File "/home/ray/anaconda3/lib/python3.7/site-packages/starlette/requests.py", line 214, in stream
raise ClientDisconnect()
starlette.requests.ClientDisconnect
Failed to run batch_predict: {}
Traceback (most recent call last):
File "/home/ubuntu/ludwig/ludwig/serve.py", line 105, in batch_predict
resp, _ = model.predict(dataset=data_df)
File "/home/ubuntu/ludwig/ludwig/api.py", line 727, in predict
callbacks=self.callbacks,
File "/home/ubuntu/ludwig/ludwig/data/preprocessing.py", line 1681, in preprocess_for_prediction
dataset, features, preprocessing_params, training_set_metadata, backend, callbacks
File "/home/ubuntu/ludwig/ludwig/data/preprocessing.py", line 243, in preprocess_for_prediction
mode="prediction",
File "/home/ubuntu/ludwig/ludwig/data/preprocessing.py", line 1033, in build_dataset
dataset_df = backend.df_engine.persist(dataset_df)
File "/home/ubuntu/ludwig/ludwig/data/dataframe/dask.py", line 62, in persist
return data.persist() if self._persist else data
File "/home/ray/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 5139, in __getattr__
return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'persist'
INFO: 127.0.0.1:50454 - "POST /batch_predict HTTP/1.1" 500 Internal Server Error
Thus I was not able to reproduce the high latency that Smiral observed in his non-Ray environment, but GPU environment.
-
With
nvidia-smi, we observed that GPU utilization was indeed 0 during an infinite curl in Smiral's environment with Smiral's model. However, I was able to observe ~50% utilization using the same curl with my model in my Ray environment. Both of us observed near 100% GPU utilization during training. -
Smiral (please correct me if I read this incorrectly) was able to serve my ray-trained model on his non-Ray environment. His latency numbers were around ~2 seconds, so still 10X slower than what I observed on my Ray cluster, but also significantly faster than 9-10 sec originally.
-
We observed similar patterns in Pytorch.
Potential culprits and next steps:
- Ray vs. non-Ray. Should serving a ludwig model trained with/without Ray perform the same in a Ray/non-Ray GPU (dockerized) environment? Smiral will look into whether he's able to get better latency numbers using Ray.
- Ludwig 0.4 vs. 0.4.1 (tf-legacy head). Potentially some differences between 0.4 and 0.4.1, and may also be why I wasn't able to use
ludwig servewith Smiral's model.
Final RCA - Due to weights being initialised at a later stage , we had every prediction call pinging the HF servers for config (was not being picked up from cache). This action blocked the prediction flow due to which GPU was not being leveraged efficiently.
Tried 0.5.2 release but it still makes external calls to HF which has every inference 7-8 sec even on gpu
Hi @smiraldr, this should be fully addressed with #2131, which was not in the 0.5.2 release but will be included in the 0.5.3 release, which should land shortly.
I'll ping back here when the release is complete and it would be great if you could try the fix with v0.5.3.