rasa memory usage keep growing when model updating

[zhnglicho]

Rasa Open Source version

3.0.9

Rasa SDK version

3.1.0

Python version

3.8

What operating system are you using?

Linux

What happened?

models: url: http://my-server.com/models/default_core@latest
wait_time_between_pulls: 10 # [optional](default: 100)

I use this way to update rasa model, when I have a new model, rasa will pull the latest model automatically, and finish update. it is great, but I encounter an issue that memory usage increase when new model is pulled and updated.
for example rasa serving launch, and finish first model loading, the rasa memory useage is about 1.4G, when I change model to another version, it will pull the model automatically and finish changing, in this time, the rasa memory usage usage is 2.1G, and doesn't release . config.yml ` version: "3.0" recipe: default.v1 language: zh pipeline:

• name: JiebaTokenizer
• name: LanguageModelFeaturizer model_name: "bert" model_weights: "hfl/chinese-roberta-wwm-ext"
• name: RegexFeaturizer use_word_boundaries: True case_sensitive: True number_additional_patterns: 10
• name: RegexEntityExtractor case_sensitive: False
• name: DIETClassifier epochs: 25 batch_size: 256 number_of_transformer_layers: 1 embedding_dimension: 30 learning_rate: 5e-3 transformer_size: 128 number_of_attention_heads: 2
• name: EntitySynonymMapper
• name: FallbackClassifier threshold: 0.7 ambiguity_threshold: 0.1 policies:
• name: MemoizationPolicy
• name: TEDPolicy max_history: 30 epochs: 30 batch_size: 50
• name: RulePolicy

`

Command / Request

a

Relevant log output

2

[losterloh]

Hello @zhnglicho , I could not reproduce this bug. Can you provide more details? Can you supply your config, domain, and training data files? How did you setup the model server that supplies the model in your case? What command do you use to run rasa?

[zhnglicho]

Hi @losterloh , thank you for your resposne. which os did you use ? can you try to reproduce in ubuntu 16.04 . Here is the command I run "rasa run --debug --port 5015 --log-file run.log" . can you give me an email , I will send train data and model to you .

[nyejon]

I have the same issue. Deployed using the rasa docker image/helm charts, and rasa x community as model server.

[losterloh]

Thanks for the additional info about the setup @nyejon, that may give us enough pointers to try again to reproduce. @zhnglicho will ping you in case we still can't reproduce and would like your data files there.

[kalpa277]

@zhnglicho How I will train the model so that it should be done around 30 seconds. I am using GPU but it takes around 2-4 mins to train whereas in CPU it takes 12-20 mins

[zhnglicho]

@losterloh can you reproduce it ? do you still need me to provider the data files ? @kalpa277 Just using rasa train command, I dont know which cpu model and gpu model that you use now , in cpu envrioment, you can try to add ----num-threads {cpu cores number}

[kalpa277]

sure @losterloh . I am having current 500 lines of training code.My GPU server

powered by [AMD Radeon Instinct MI25](https://www.amd.com/en/products/professional-graphics/instinct-mi25) GPUs and AMD EPYC 7V12(Rome) CPUs with a base frequency of 2.45GHz, all-cores peak frequency of 3.1GHzency of 3.1GHz** ```

Is this better one or we have to any other GPU like NVIDIA or GPU upper variant like NPU.Becacuse when new model is trained only epochs takes time to load and Epochs are neural network which takes time to load.

[ancalita]

@zhnglicho If you could create a public repo with the project files which we can use to reproduce the issue and debug further, that would be invaluable 🙏🏻

[sync-by-unito[bot]]

➤ Maxime Verger commented:

:bulb: Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

:arrow_right: More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.