SQUAD logits to probabilities

[Rajmehta123]

A very basic question.

What is the logit score in the Squad pre-trained model? [158.80136108398438]

I know logit is a log of the odds. But how should I convert logits to probabilities? If that is not possible, how can I infer the logit to decide if this is an accurate answer, eg: 158 logits might resemble highly accurate.

[Rajmehta123]

@danielkornev @IgnatovFedor Any help would be appreciated. Thanks :)

[vaskonov]

@Rajmehta123 you are correct, based on logits only you cannot assess the model's confidence. Please mention the configuration file that has been used.

[alphabetagamer]

Have the same question , I tried using config.squad.squad_bert and it allowed me to output the score by changing "out": ["ans_start_predicted", "ans_end_predicted", "logits"] }, { "class_name": "squad_bert_ans_postprocessor", "in": ["ans_start_predicted", "ans_end_predicted", "context_raw", "bert_features", "subtok2chars"], "out": ["ans_predicted", "ans_start_predicted", "ans_end_predicted"] } ], "out": ["ans_predicted", "ans_start_predicted", "logits"] to "out": ["ans_start_predicted", "ans_end_predicted", "logits","score"] }, { "class_name": "squad_bert_ans_postprocessor", "in": ["ans_start_predicted", "ans_end_predicted", "context_raw", "bert_features", "subtok2chars"], "out": ["ans_predicted", "ans_start_predicted", "ans_end_predicted"] } ], "out": ["ans_predicted", "ans_start_predicted", "logits","score"]

However, it raised errors when I tried the same changes in config.squad.squad. **Changes : **

"out": ["ans_start_predicted", "ans_end_predicted", "logits","score"] }, { "class_name": "squad_ans_postprocessor", "id": "squad_ans_postprepr", "in": ["ans_start_predicted", "ans_end_predicted", "context_raw", "c_p2r", "spans"], "out": ["ans_predicted", "ans_start_predicted", "ans_end_predicted"] } ], "out": ["ans_predicted", "ans_start_predicted", "logits","score"] Error log:

Traceback (most recent call last): File "C:\Users\lokin.conda\envs\dp\lib\site-packages\flask\app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "C:\Users\lokin.conda\envs\dp\lib\site-packages\flask\app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "C:\Users\lokin.conda\envs\dp\lib\site-packages\flask_cors\extension.py", line 165, in wrapped_function return cors_after_request(app.make_response(f(*args, **kwargs))) File "C:\Users\lokin.conda\envs\dp\lib\site-packages\flask\app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "C:\Users\lokin.conda\envs\dp\lib\site-packages\flask_compat.py", line 39, in reraise raise value File "C:\Users\lokin.conda\envs\dp\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "C:\Users\lokin.conda\envs\dp\lib\site-packages\flask\app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "c:/Users/lokin/Desktop/2021/Cimpa/para-finder/server_parafinder.py", line 1993, in onlyparagraph_answers ou.append({'query':z,'para':txt,'dp':model_dp([txt],[z])}) File "C:\Users\lokin.conda\envs\dp\lib\site-packages\deeppavlov\core\common\chainer.py", line 207, in call return self._compute(*args, param_names=self.in_x, pipe=self.pipe, targets=self.out_params) File "C:\Users\lokin.conda\envs\dp\lib\site-packages\deeppavlov\core\common\chainer.py", line 236, in _compute res = [mem[k] for k in targets] File "C:\Users\lokin.conda\envs\dp\lib\site-packages\deeppavlov\core\common\chainer.py", line 236, in res = [mem[k] for k in targets] KeyError: 'score'

[IgnatovFedor]

@Rajmehta123, thank you for your interest! We tested squad_bert config from the DeepPavlov version 1.2.0 and didn’t encounter the described problem. If you are still having difficulties with this, please let us know.