ask-cli icon indicating copy to clipboard operation
ask-cli copied to clipboard

ask smapi get-result-for-nlu-evaluations result is missing the values for expected slot names

Open arisen1145 opened this issue 3 years ago • 1 comments

I'm submitting a...


[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report  
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request 
[ ] Other... Please describe: 

Expected Behavior

The API response of ask smapi get-result-for-nlu-evaluations includes the correct data for the property "expected" (same as defined in the NLU annotation).

Current Behavior

API response of ask smapi get-result-for-nlu-evaluations has a wrong (missing) data in the property "expected". The slot name and value which is defined in the NLU annotation is missing. The test is still passing, but the actual and expected are not matching. Same behavior is in the UI of the in NLU Evaluation Results, see attached Screenshot

{
  "_links": {
        ...
    }
  },
  "paginationContext": {
    "totalCount": 1
  },
  "testCases": [
    {
      "actual": {
        "intent": {
          "confirmationStatus": "NONE",
          "name": "HelloWorldIntent",
          "slots": {
            "name": {
              "confirmationStatus": "NONE",
              "name": "name",
              "slotValue": {
                "type": "Simple",
                "value": "mike"
              }
            }
          }
        }
      },
      "expected": [
        {
          "intent": {
            "name": "HelloWorldIntent",
            "slots": {
              "name": {}
            }
          }
        }
      ],
      "inputs": {
        "utterance": "hello mike"
      },
      "status": "PASSED"
    }
  ],
  "totalFailed": 0
}

CLI Snapshot If applicable, add screenshots to help explain your problem.

Steps to Reproduce (for bugs)

  • Create new sample skill.
  • Go to Build -> Intents -> HelloWorldIntent -> add an utterance "hello {name}" with slot Type Amazon.Firstname
  • Go to Build -> Intents -> Annotation Sets -> Create a NLU Annotation Set with Utterance: hello mike Expected Intent: HelloWorldIntent Expected slots: slot name=name, slot value=mike
  • Go to Evaluate Model -> NLU Evaluation -> Select your Annotation and click Run evaluation
  • Copy the evaluationId and do this cli request ask smapi get-result-for-nlu-evaluations -s amzn1.ask.skill.xyz--evaluation-id xyz

Possible Solution

Your Environment and Context

  • ask-cli version: 2.22.4
  • Operating System and version: macOS 11.2.2
  • Node.js version used for development: v12.16.1
  • NPM version used for development: 6.13.4 Screenshot 2021-03-09 at 17 42 59

arisen1145 avatar Mar 09 '21 16:03 arisen1145

Hi @arisen1145, thank you for creating this bug report. This seems like a problem with the SMAPI service, so we will try to investigate.

CamdenFoucht avatar Apr 28 '22 20:04 CamdenFoucht