rasa
rasa copied to clipboard
[Draft PR only for MR tests] TF 2.5 Update
Proposed changes:
- ...
Status (please check what you already did):
Hey @dakshvar22! :wave: To run model regression tests, comment with the /modeltest command and a configuration.
Tips :bulb:: The model regression test will be run on push events. You can re-run the tests by re-add status:model-regression-tests label or use a Re-run jobs button in Github Actions workflow.
Tips :bulb:: Every time when you want to change a configuration you should edit the comment with the previous configuration.
You can copy this in your comment and customize:
/modeltest
```yml ########## ## Available datasets ########## # - "Carbon Bot" (NLU) # - "Hermit" (NLU) # - "Private 1" (NLU) # - "Private 2" (NLU) # - "Private 3" (NLU) # - "Sara" (NLU, Core) # - "financial-demo" (NLU, Core) # - "helpdesk-assistant" (NLU, Core) # - "insurance-demo" (NLU, Core) # - "retail-demo" (NLU, Core) ########## ## Available NLU configurations ########## # - "BERT + DIET(bow) + ResponseSelector(bow)" # - "BERT + DIET(seq) + ResponseSelector(t2t)" # - "Spacy + DIET(bow) + ResponseSelector(bow)" # - "Spacy + DIET(seq) + ResponseSelector(t2t)" # - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)" # - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)" # - "Sparse + DIET(bow) + ResponseSelector(bow)" # - "Sparse + DIET(seq) + ResponseSelector(t2t)" # - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)" # - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)" ########## ## Available Core configurations ########## # - "Rules" # - "Rules + AugMemo" # - "Rules + AugMemo + TED" # - "Rules + Memo" # - "Rules + Memo + TED" # - "Rules + TED" ## Example configuration #################### syntax ################# ## include: ## - dataset: ["<dataset_name>"] ## config: ["<configuration_name>"] # ## Example: ## include: ## - dataset: ["Carbon Bot"] ## config: ["Sparse + DIET(bow) + ResponseSelector(bow)"] # ## Shortcut: ## You can use the "all" shortcut to include all available configurations or datasets # ## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration ## for all available datasets ## include: ## - dataset: ["all"] ## config: ["Sparse + DIET(bow) + ResponseSelector(bow)"] # ## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets ## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and ## "BERT + DIET + ResponseSelector(T2T)" configurations: ## include: ## - dataset: ["Carbon Bot", "Sara"] ## config: ["all"] ## - dataset: ["Hermit"] ## config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"] # ## Example: Define a branch name to check-out for a dataset repository. Default branch is 'main' ## dataset_branch: "test-branch" ## include: ## - dataset: ["Carbon Bot", "Sara"] ## config: ["all"] ## ## Shortcuts: ## You can use the "all" shortcut to include all available configurations or datasets. ## You can use the "all-nlu" shortcut to include all available NLU configurations or datasets. ## You can use the "all-core" shortcut to include all available core configurations or datasets. include: - dataset: ["Carbon Bot"] config: ["Sparse + DIET(bow) + ResponseSelector(bow)"] ```
/modeltest
##########
## Available datasets
##########
# - "Carbon Bot" (NLU)
# - "Hermit" (NLU)
# - "Private 1" (NLU)
# - "Private 2" (NLU)
# - "Private 3" (NLU)
# - "Sara" (NLU, Core)
# - "financial-demo" (NLU, Core)
# - "helpdesk-assistant" (NLU, Core)
# - "insurance-demo" (NLU, Core)
# - "retail-demo" (NLU, Core)
##########
## Available NLU configurations
##########
# - "BERT + DIET(bow) + ResponseSelector(bow)"
# - "BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Spacy + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"
##########
## Available Core configurations
##########
# - "Rules"
# - "Rules + AugMemo"
# - "Rules + AugMemo + TED"
# - "Rules + Memo"
# - "Rules + Memo + TED"
# - "Rules + TED"
## Example configuration
#################### syntax #################
## include:
## - dataset: ["<dataset_name>"]
## config: ["<configuration_name>"]
#
## Example:
## include:
## - dataset: ["Carbon Bot"]
## config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Shortcut:
## You can use the "all" shortcut to include all available configurations or datasets
#
## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration
## for all available datasets
## include:
## - dataset: ["all"]
## config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets
## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and
## "BERT + DIET + ResponseSelector(T2T)" configurations:
## include:
## - dataset: ["Carbon Bot", "Sara"]
## config: ["all"]
## - dataset: ["Hermit"]
## config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"]
#
## Example: Define a branch name to check-out for a dataset repository. Default branch is 'main'
## dataset_branch: "test-branch"
## include:
## - dataset: ["Carbon Bot", "Sara"]
## config: ["all"]
##
## Shortcuts:
## You can use the "all" shortcut to include all available configurations or datasets.
## You can use the "all-nlu" shortcut to include all available NLU configurations or datasets.
## You can use the "all-core" shortcut to include all available core configurations or datasets.
include:
- dataset: ["Private 3"]
config: ["Spacy + DIET(seq) + ResponseSelector(t2t)"]
The model regression tests have started. It might take a while, please be patient. As soon as results are ready you'll see a new comment with the results.
Used configuration can be found in the comment.
Commit: eafde326b96826ecee28d2bf6c56101465e58f69, The full report is available as an artifact.
Dataset: Private 3, Dataset repository branch: main, commit: 624f54ebc82536b144d8eebf40c27369c93fa99d
| Configuration | Intent Classification Micro F1 | Entity Recognition Micro F1 | Response Selection Micro F1 |
|---|---|---|---|
Spacy + DIET(seq) + ResponseSelector(t2t)test: 37s, train: 1m48s, total: 2m24s |
0.6214 (0.00) | no data |
no data |
This PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 4 committers have signed the CLA.
:white_check_mark: Ghostvv
:white_check_mark: koernerfelicia
:white_check_mark: samsucik
:x: Daksh
Daksh seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.