dialogy
dialogy copied to clipboard
Published
20 hours ago •
skit-ai
skit-ai
Fix amount <> interval clash
Sometimes (randomly?) this test on application collections combined is failing and that's resulting in a failed CI/CD pipeline.
This patch currently bypasses the dim=amount-of-money and type=interval coupling. (amount-of-money expects type to be value, not interval).
Log of make test on SLU succeeding and then make test failing, without any change to the codebase.
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$ make test
========================================== test session starts ==========================================
platform linux -- Python 3.9.13, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/nilesh/slu-test-acc-v1/application-collections-entity-improvements, configfile: pytest.ini
plugins: cov-2.12.1, anyio-3.6.1
collected 8 items
tests/test_controller/test_predict_api.py ........ [100%]
---------- coverage: platform linux, python 3.9.13-final-0 -----------
Name Stmts Miss Cover
----------------------------------------------------------
slu/dev/cli.py 74 74 0%
slu/dev/dev.py 69 69 0%
slu/dev/dir_setup.py 18 18 0%
slu/dev/io/mp/__init__.py 16 16 0%
slu/dev/io/reader/csv.py 35 35 0%
slu/dev/io/reader/pickle.py 15 15 0%
slu/dev/io/reader/sqlite.py 17 17 0%
slu/dev/prepare_entity_tags.py 101 101 0%
slu/dev/repl.py 50 50 0%
slu/dev/test.py 105 77 27%
slu/dev/train.py 114 114 0%
slu/src/api/__init__.py 3 3 0%
slu/src/api/endpoints.py 78 78 0%
slu/src/api/models.py 9 9 0%
slu/src/controller/custom_plugins.py 597 161 73%
slu/src/controller/prediction.py 73 24 67%
slu/src/controller/processors.py 53 5 91%
slu/utils/calibration.py 16 16 0%
slu/utils/config.py 97 21 78%
slu/utils/decorators.py 22 22 0%
slu/utils/error.py 4 4 0%
slu/utils/error_response.py 19 19 0%
slu/utils/ignore.py 3 3 0%
slu/utils/make_test_cases.py 23 23 0%
slu/utils/s3.py 20 20 0%
slu/utils/sentry.py 4 4 0%
slu/utils/slack.py 30 30 0%
----------------------------------------------------------
TOTAL 1819 1028 43%
10 files skipped due to complete coverage.
Coverage HTML written to dir htmlcov
========================================== 8 passed in 53.35s ===========================================
The tests pass!
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$ make test
========================================== test session starts ==========================================
platform linux -- Python 3.9.13, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/nilesh/slu-test-acc-v1/application-collections-entity-improvements, configfile: pytest.ini
plugins: cov-2.12.1, anyio-3.6.1
collected 8 items
tests/test_controller/test_predict_api.py .......F [100%]
=============================================== FAILURES ================================================
___________________________________ test_classifier_on_training_data ____________________________________
cls = <class 'dialogy.types.entity.deserialize.EntityDeserializer'>
duckling_entity_dict = {'body': '3-4', 'dim': 'amount-of-money', 'end': 23, 'latent': False, ...}
alternative_index = 0, reference_time = 1666295139132, timezone = 'Asia/Kolkata'
duration_cast_operator = None, constraints = None
@classmethod
def deserialize_duckling(
cls,
duckling_entity_dict: Dict[str, Any],
alternative_index: int,
reference_time: Optional[int] = None,
timezone: str = "UTC",
duration_cast_operator: Optional[str] = None,
constraints: Optional[Dict[str, Any]] = None,
) -> Optional[BaseEntity]:
cls.validate(duckling_entity_dict)
entity_class_name = cls.get_entity_class_str(duckling_entity_dict)
EntityClass: BaseEntity = cls.entitiy_classes[entity_class_name]
try:
> entity = EntityClass.from_duckling(
duckling_entity_dict,
alternative_index,
constraints=constraints,
duration_cast_operator=duration_cast_operator,
timezone=timezone,
reference_time=reference_time,
)
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/types/entity/deserialize/__init__.py:66:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'dialogy.types.entity.amount_of_money.CurrencyEntity'>
d = {'body': '3-4', 'dim': 'amount-of-money', 'end': 23, 'latent': False, ...}, alternative_index = 0
kwargs = {'constraints': None, 'duration_cast_operator': None, 'reference_time': 1666295139132, 'timezone': 'Asia/Kolkata'}
@classmethod
def from_duckling(
cls, d: Dict[str, Any], alternative_index: int, **kwargs: Any
) -> CurrencyEntity:
> value = d[const.VALUE][const.VALUE]
E KeyError: 'value'
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/types/entity/amount_of_money/__init__.py:61: KeyError
The above exception was the direct cause of the following exception:
slu_api = <function get_predictions.<locals>.predict at 0x7f8bd36428b0>
def test_classifier_on_training_data(slu_api):
"""
Evaluate the workflow with all the embedded plugins.
Plugins can be evaluated individually for fine-tuning but since there are interactions
between them, we need to evaluate them all together. This helps in cases where these interactions
are a cause of a model's poor performance.
This method doesn't mutate the given test dataset, instead we produce results with the same `id_`
so that they can be joined and studied together.
"""
project_config_map = YAMLLocalConfig().generate()
config: Config = list(project_config_map.values()).pop()
dataset = config.get_dataset(const.CLASSIFICATION, f"{const.TRAIN}.csv")
test_df = pd.read_csv(dataset).sample(n=100)
predictions = []
config.tasks.classification.threshold = 0
for _, row in tqdm(test_df.iterrows(), total=test_df.shape[0]):
> output = slu_api(
**{
const.ALTERNATIVES: json.loads(row[const.ALTERNATIVES]),
const.CONTEXT: {},
const.LANG: "en",
"ignore_test_case": True,
}
)
tests/test_controller/test_predict_api.py:146:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
slu/src/controller/prediction.py:110: in predict
_, output = workflow.run(input_)
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/workflow/workflow.py:285: in run
return self.execute().flush()
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/workflow/workflow.py:260: in execute
plugin(self)
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/base/plugin/__init__.py:292: in __call__
value = self.utility(workflow.input, workflow.output)
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:992: in utility
return self.parse(
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:970: in parse
entities = self.apply_entity_classes(
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:907: in apply_entity_classes
self._reshape(
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/plugins/text/duckling_plugin/__init__.py:788: in _reshape
entity = EntityDeserializer.deserialize_duckling(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'dialogy.types.entity.deserialize.EntityDeserializer'>
duckling_entity_dict = {'body': '3-4', 'dim': 'amount-of-money', 'end': 23, 'latent': False, ...}
alternative_index = 0, reference_time = 1666295139132, timezone = 'Asia/Kolkata'
duration_cast_operator = None, constraints = None
@classmethod
def deserialize_duckling(
cls,
duckling_entity_dict: Dict[str, Any],
alternative_index: int,
reference_time: Optional[int] = None,
timezone: str = "UTC",
duration_cast_operator: Optional[str] = None,
constraints: Optional[Dict[str, Any]] = None,
) -> Optional[BaseEntity]:
cls.validate(duckling_entity_dict)
entity_class_name = cls.get_entity_class_str(duckling_entity_dict)
EntityClass: BaseEntity = cls.entitiy_classes[entity_class_name]
try:
entity = EntityClass.from_duckling(
duckling_entity_dict,
alternative_index,
constraints=constraints,
duration_cast_operator=duration_cast_operator,
timezone=timezone,
reference_time=reference_time,
)
except KeyError as e:
> raise ValueError(
f"Failed to deserialize {EntityClass} "
f"from duckling response: {duckling_entity_dict}. Exception: {e}"
) from e
E ValueError: Failed to deserialize <class 'dialogy.types.entity.amount_of_money.CurrencyEntity'> from duckling response: {'body': '3-4', 'start': 20, 'value': {'to': {'value': 4, 'unit': 'unknown'}, 'from': {'value': 3, 'unit': 'unknown'}, 'type': 'interval'}, 'end': 23, 'dim': 'amount-of-money', 'latent': False}. Exception: 'value'
../../anaconda3/envs/acc-tmp/lib/python3.9/site-packages/dialogy/types/entity/deserialize/__init__.py:75: ValueError
----------------------------------------- Captured stdout call ------------------------------------------
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_merchant_issue
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_service_issue_other
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_other
PaymentHesitationReasonFiller: rlabel = reason_service_issue_complaints_and_customer_care
PaymentHesitationReasonFiller: rlabel = reason_another_company
PaymentHesitationReasonFiller: rlabel = no_reason
PaymentHesitationReasonFiller: passing...
PaymentHesitationReasonFiller: rlabel = reason_no_reminder
PaymentHesitationReasonFiller: rlabel = reason_service_issue_other
PaymentHesitationReasonFiller: rlabel = reason_loan_not_taken
PaymentHesitationReasonFiller: rlabel = reason_other
PaymentHesitationReasonFiller: rlabel = reason_service_issue_no_statement
PaymentHesitationReasonFiller: rlabel = reason_service_issue_complaints_and_customer_care
PaymentHesitationReasonFiller: rlabel = reason_other
PaymentHesitationReasonFiller: rlabel = reason_i_was_not_available
PaymentHesitationReasonFiller: rlabel = reason_service_issue_no_statement
----------------------------------------- Captured stderr call ------------------------------------------
/home/nilesh/slu-test-acc-v1/application-collections-entity-improvements/tests/test_controller/test_predict_api.py:140: DtypeWarning: Columns (3,6,13,14,15,17,18,19,20,21,22,23,24,25,28,29,30,32,34,36,45,48,49) have mixed types. Specify dtype option on import or set low_memory=False.
test_df = pd.read_csv(dataset).sample(n=100)
79%|███████▉ | 79/100 [00:28<00:07, 2.74it/s]
---------- coverage: platform linux, python 3.9.13-final-0 -----------
Name Stmts Miss Cover
----------------------------------------------------------
slu/dev/cli.py 74 74 0%
slu/dev/dev.py 69 69 0%
slu/dev/dir_setup.py 18 18 0%
slu/dev/io/mp/__init__.py 16 16 0%
slu/dev/io/reader/csv.py 35 35 0%
slu/dev/io/reader/pickle.py 15 15 0%
slu/dev/io/reader/sqlite.py 17 17 0%
slu/dev/prepare_entity_tags.py 101 101 0%
slu/dev/repl.py 50 50 0%
slu/dev/test.py 105 84 20%
slu/dev/train.py 114 114 0%
slu/src/api/__init__.py 3 3 0%
slu/src/api/endpoints.py 78 78 0%
slu/src/api/models.py 9 9 0%
slu/src/controller/custom_plugins.py 597 165 72%
slu/src/controller/prediction.py 73 23 68%
slu/src/controller/processors.py 53 5 91%
slu/utils/calibration.py 16 16 0%
slu/utils/config.py 97 21 78%
slu/utils/decorators.py 22 22 0%
slu/utils/error.py 4 4 0%
slu/utils/error_response.py 19 19 0%
slu/utils/ignore.py 3 3 0%
slu/utils/make_test_cases.py 23 23 0%
slu/utils/s3.py 20 20 0%
slu/utils/sentry.py 4 4 0%
slu/utils/slack.py 30 30 0%
----------------------------------------------------------
TOTAL 1819 1038 43%
10 files skipped due to complete coverage.
Coverage HTML written to dir htmlcov
======================================== short test summary info ========================================
FAILED tests/test_controller/test_predict_api.py::test_classifier_on_training_data - ValueError: Faile...
===================================== 1 failed, 7 passed in 46.13s ======================================
make: *** [Makefile:15: test] Error 1
(acc-tmp) nilesh@e2e-95-223:~/slu-test-acc-v1/application-collections-entity-improvements$