Information on Bot Framework and Composer's support of Language Studio (Conversational Language Understanding, Custom Question Answering))

[cwhitten]

12/14/22 update: See this link for more information on using a custom CQA recognizer: https://github.com/microsoft/BotFramework-Composer/issues/9400#issuecomment-1351756216

Hello everyone,

With the recent announcements around the retirement of QnA Maker and GA release of CQA (Custom Question Answering), we wanted to provide some guidance to our community. We have a sample for using CQA with the BF (Bot Framework) SDK today and the link is below and for Composer we are looking to provide a package that will allow for CQA as a custom recognizer in the coming weeks. To provide some relief, we have extended the ability to create new QnA resources temporarily for those subscriptions where we identified active use of Composer for QnA authoring between 9/1/2022 and 10/1/2022. The extension will be provided to these customers until 1/1/2023. Premier Support Customers that feel that they need more time to provision QnA Maker resources that were not already approved will be required to open a support request to ask for an exception that can be evaluated by our PG for approval. We will be communicating guidance on moving away from QnA Maker for BF Composer as a resource in the coming weeks as there are great advantages to our new product offering.

Here are some documentation links that are available currently and we will post again when we have more to share:

SDK CQA Sample

BotBuilder-Samples/samples/csharp_dotnetcore/12.customQABot at main · microsoft/BotBuilder-Samples (github.com)

BotBuilder-Samples/samples/csharp_dotnetcore/48.customQABot-all-features at main · microsoft/BotBuilder-Samples (github.com)

JavaScript

https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/javascript_nodejs/12.customQABot

https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/javascript_nodejs/48.customQABot-all-features

Csharp

BotBuilder-Samples/samples/csharp_dotnetcore/12.customQABot at main · microsoft/BotBuilder-Samples (github.com)

BotBuilder-Samples/samples/csharp_dotnetcore/48.customQABot-all-features at main · microsoft/BotBuilder-Samples (github.com)

CQA Documentation

What is question answering? - Azure

How to migrate QnaMaker Cognitive Services - Microsoft Learn

Question Answering tutorial

We will continue to collect your feedback and share updates to this pinned github issue as we have them. Thank you

[nghivo]

Hi, Thank you so much for your information @cwhitten. How about if any plan for Migrate from Language Understanding (LUIS) to conversational language understanding (CLU), that can work for Bot Composer?

[fausttiger007]

btw, one of your C# specific links points to JS... although the correct link is also listed at top

[martinlarosa]

@cwhitten will there be support for CLU and its multilingual capabilities?

[Dewain27]

CLU guidance will be coming in the months ahead. We are actively working with the Cognitive Services Language Team on the best guidance as it relates to Composer / SDK bot building. We will be providing communication once that guidance is finalized.

[primle]

CLU guidance will be coming in the months ahead. We are actively working with the Cognitive Services Language Team on the best guidance as it relates to Composer / SDK bot building. We will be providing communication once that guidance is finalized.

Thanks @cwhitten and @Dewain27 for the update about the custom CQA recognizer and potential future updates on CLU.

We are currently developing a new dialog architecture of our existing "flow-bot", written in Composer (*) and want to use CLU & Question Answering. We are in a multilingual setup with 4 languages and would love to have something similar to the existing LUIS/QnA solution with a recognizer set and some build-in support for CLU.

@Dewain27 Do you already know if there will be some built-in solutions or "extensions" like a custom recognizer?

In the meantime, would you recommend:

1. Building our own custom recognizer? Can be implemented like custom actions. Currently, there is no good documentation, but some examples can be found here: iciclecreek.bot/source/Libraries at master · tomlm/iciclecreek.bot · GitHub and there is the mentioned implementation for the Bot Framework SDK.

2. Building a custom middleware, which will capture the out-going request to LUIS and calls the CLU Rest API instead and stores the results accordingly. (So BFC thinks its based on LUIS)

3. Something else (or waiting for the provided solution)

(*) PVA is currently no valid option for us, as we do not have the procurement basis and need customization like connecting via mTLS to an on-premise network.

[peterswimm]

@primle also check out the usage of custom recognizer in these docs: https://learn.microsoft.com/en-us/composer/concept-language-understanding?source=recommendations&tabs=v2x#to-add-language-understanding-to-a-dialog

[PhoenixJayeju]

Hi what is the update on this one @cwhitten your original post did say "coming weeks" back on the 12th of Oct, it's the end of November. What is the deal.

[spmoreno]

Hi! Is there any update on this issue? We really need to make a decision about what tools we're gonna use moving fodward.

[fausttiger007]

I have Custom QA working fine via API calls from my Composer bot project, and processing of multi-prompts. Just Regular Expression recognizer and Custom Question Answering. HOWEVER, can NOT PUBLISH to an Azure Bot, as it continues to ask for LUIS language understanding /authoring and QnAmaker resources. As these are not used in bot (they are just default/dummy values in Bot Composer appsettings.json which works fine in Composer webchat or emulator.

How do we keep Publish from requiring actual LUIS and QNA resources? They are not used nor required.

The bot was buily from EMPTY BOT Template created from Bot Composer.

"luis": {
  "authoringKey": "",
  "authoringEndpoint": "",
  "endpointKey": "",
  "endpoint": "",
  "region": "West Europe"
},
"qna": {
  "subscriptionKey": "<subscription key>",
  "endpoint": "<endpoint>"
},
"MicrosoftAppId": "<app id from Bot Framework registration>",
"MicrosoftAppPassword": "<app password from Bot Framework registration>"

Appears to make no difference whether I include or exclude the LUIS / QNA settings in publish profile... Publish tries to validate. So want to know how I disable them, as they are not used.

[nghivo]

Hi @cwhitten , We are waiting from many weeks, it is so important for our plans using Microsoft Bot Composer with Enterprise level support. Can you support us with the new update?

Thanks.

[PhoenixJayeju]

@fausttiger, i feel your pain, I've been down this exact road ☹. To fix it, you need to remove all references to QnA knowledgebase from your composer settings/project. These include QnA dialogs (and perhaps triggers, i only had to do the dialogs) in places such as the chitchat dialog at the parent level and any other dialogs referencing QnA don't forget the chitchat dialog in Bot responses section as well. You may need to go into the directory containing the bot project and do the same. Note until you remove the content of the chitchat dialog under your bot responses the issue will not go away. I found this part a bit tricky as the problem persisted even after i'd deleted the chit chat responses within the Bot response section on the canvas, in the end i had to open the file within the bot project folder, where I selected and deleted all the contents. This did the trick.
You may have already done this but check that after doing all of the above, ensure that QnA resource is deselected from the resources you want to provision.

[cwhitten]

Hi everyone,

Thank you for your patience as we're worked through this. We're very close to enabling a CQA integration through a new Bot Framework component package. A PR is in review here: https://github.com/microsoft/botframework-components/pull/1405, afterwhich we'll publish it to a public nuget feed.

[fausttiger007]

@Intelliteckai

You mentioned QnAmaker refs, but did you remove your LUIS references also or are still using LUIS (not really needed with CustomQA and Regex)

Not sure which references to QnA or LUIS... Default Recognizer for bot is Regular Expression Recognizer for all dialogs is Regular Expression.

no dialogs reference QnA... (I have no ChitChat dialog as this was based on emptyBot template). I think LUIS and QnA references are just generated from the code built into Composer that has configuration for them even if not used.

There are (blank empty) files like CQA_Bot.en-us.qna or CQA_Bot.en-us.lu
in folders such as knowledge-base or language-understanding, but those are all empty (but maybe still referenced)

removing unused settings from appsettings.json doesn't do it either.

I'l note the EmptyBot template created from Composer isn't even supposed to have QnA or LUIS components (but it still requires (or validates) them for Publish)

[fausttiger007]

@cwhitten

this is still only for a full code solution, and not a composer based solution?

I have my composer based solution working, but can't deploy to azure due to dependencies wired into Composer it seems.

Hi everyone,

Thank you for your patience as we're worked through this. We're very close to enabling a CQA integration through a new Bot Framework component package. A PR is in review here: microsoft/botframework-components#1405, afterwhich we'll publish it to a public nuget feed.

[nickfrancis]

I have Custom QA working fine via API calls from my Composer bot project, and processing of multi-prompts. Just Regular Expression recognizer and Custom Question Answering. HOWEVER, can NOT PUBLISH to an Azure Bot, as it continues to ask for LUIS language understanding /authoring and QnAmaker resources. As these are not used in bot (they are just default/dummy values in Bot Composer appsettings.json which works fine in Composer webchat or emulator.

How do we keep Publish from requiring actual LUIS and QNA resources? They are not used nor required.

The bot was buily from EMPTY BOT Template created from Bot Composer.

"luis": {
  "authoringKey": "",
  "authoringEndpoint": "",
  "endpointKey": "",
  "endpoint": "",
  "region": "West Europe"
},
"qna": {
  "subscriptionKey": "<subscription key>",
  "endpoint": "<endpoint>"
},
"MicrosoftAppId": "<app id from Bot Framework registration>",
"MicrosoftAppPassword": "<app password from Bot Framework registration>"

Appears to make no difference whether I include or exclude the LUIS / QNA settings in publish profile... Publish tries to validate. So want to know how I disable them, as they are not used.

@fausttiger would love to see how you got this set up & working. Could you share your set-up / screenshots?

[PhoenixJayeju]

@fausttiger the approach for QnA is the same for LUIS. I decided to continue using LUIS instead of CLU because it is more complicated to setup than QnA if you consider that you have to create http actions for each intent in trigger in LUIS. Also LUIS has not been taken offline like the QnA service. You can see from the attached image that for the unknown intent where i called the http action, i defined a multi switch options for the two possible utterances including the default, (Help and None). To set it up for language, you will need to add all your intents in the switch and then follow it up with the actions for the bot to take. If you consider the way the LUIS works, the users says something (utterance), it gets sent to the LUIS to determine the intent from the keyphrase. The LUIS returns the probable intent (top intent) confidence thresholds score to the bot, the bot then looks up what to do next as defined in the dialog. it's the same here.

[cwhitten]

I have Custom QA working fine via API calls from my Composer bot project, and processing of multi-prompts. Just Regular Expression recognizer and Custom Question Answering. HOWEVER, can NOT PUBLISH to an Azure Bot, as it continues to ask for LUIS language understanding /authoring and QnAmaker resources. As these are not used in bot (they are just default/dummy values in Bot Composer appsettings.json which works fine in Composer webchat or emulator. How do we keep Publish from requiring actual LUIS and QNA resources? They are not used nor required. The bot was buily from EMPTY BOT Template created from Bot Composer.

"luis": {
  "authoringKey": "",
  "authoringEndpoint": "",
  "endpointKey": "",
  "endpoint": "",
  "region": "West Europe"
},
"qna": {
  "subscriptionKey": "<subscription key>",
  "endpoint": "<endpoint>"
},
"MicrosoftAppId": "<app id from Bot Framework registration>",
"MicrosoftAppPassword": "<app password from Bot Framework registration>"

Appears to make no difference whether I include or exclude the LUIS / QNA settings in publish profile... Publish tries to validate. So want to know how I disable them, as they are not used.

@fausttiger would love to see how you got this set up & working. Could you share your set-up / screenshots?

@nickfrancis are there any .qna files in your project? You should be able to delete the knowledge-base directory in your project if so. If you have no publishing profile that provisioning QnAMaker I don't see why it would attempt to publish any QnA resources. Please try this

[fausttiger007]

@cwhitten

I'll see what I can post.

Trying to get Publish from Bot to work (talking with MS shortly) as can't yet publish from Composer to Azure via ImportResources. (MS person mentions Composer bug, Composer assuming types of bots/resourcs to be deployed, but don't have details).

Basically -I'm using API POST to call custom question answering. -If MultiPrompt count nonzero.. then Additional process to retrieve Prompt name(s) and QNAid for each, and present to user (HeroCard),

• then just QNAid query to retrieve and display selected prompt answer.

QUESTION Does Regular Expression evaluation use LUIS? (so if all recognizers ar Regex is LUIS still used?). As I don't specify any LUIS in my local Bot, I wouldn't think so.

@fausttiger would love to see how you got this set up & working. Could you share your set-up / screenshots?

[fausttiger007]

If you consider the way the LUIS works, the users says something (utterance), it gets sent to the LUIS to determine the intent from the keyphrase. The LUIS returns the probable intent (top intent) confidence thresholds score to the bot, the bot then looks up what to do next as defined in the dialog. it's the same her

@Intelliteckai ,

if only Regex recognizer used for recognizer, is LUIS still used?

Unknown intent can't use LUIS as I have none defined. I never use Default REcognizer only regex (including Cancel/Help/etc. dialogs)

But yes, I might just add an unused LUIS resource if needed to avoid Publish validation issues (bot doesn't require it)

[fausttiger007]

are there any .qna files in your project? You should be able to delete the knowledge-base directory in your project if so. If you have no publishing profile that provisioning QnAMaker I don't see why it would attempt to publish any QnA resources. Please try this

@cwhitten

does same apply for .LU files? There are a number in my project, but they are all empty files.

i.e. Bot_CQA\language-understanding\en-us\CQA_Bot.en-us.lu Bot_CQA\knowledge-base\en-us\CQA_Bot.en-us.qna etc.

[fausttiger007]

OK, with just Custom QA and Regex recognizers we got the bot deployed from Composer into Azure Bot.

Our current bot project was already working find in Bot Composer web chat or emulator... we just couldn't Import Resources (created all in Azure) to publish bot code to Azure Bot.

We basically just had composer create basic bot requirements + LUIS; Had language create language components, and manually add some reporting related elements in Azure

This is how we provisioned/published (I'll have to get some details how we have Customer QA working in our bot composer project later)...

1. From Azure Create Language project - Custom Question Answering which creates Search service Storage account

2. From Composer (my working project) new publish profile / create resources (just bot, required and LUIS, no QnA) Required Defaults -Azure Bot -App Service plan -App Service Plus Options: -Luis Authoring -Luis Prediction

continue which provisions resources in Azure

3. From Bot Composer, publish code bot to Azure Works in Azure.

4. In Composer bot appsettings.json

• qna = {}
• luis and Luis Authoring settings : false or default empty, not sure if it matters

5. Manually in Azure create -Application Insights -Log Analytics workspace add keys/etc. as needed to connect to Language/other resources

Still need to determine best setup/queries to get log analytics/application insights data for telemetry /reporting in Power BI as different compared with QnAMaker.

[AhmedEl-nagar]

@cwhitten

Any update regarding this issue?

I am using Bot Composer, I tried to use the new package, but I have an issue

dialog error: Type Microsoft.Bot.Components.Recognizers.CustomQuestionAnsweringRecognizer not registered in factory.

[fausttiger007]

@fausttiger would love to see how you got this set up & working. Could you share your set-up / screenshots?

@cwhitten

Basically this in Composer below is the main query and prompts processing... (I already posted how I created Azure resources with unused LUIS) This bot was built on top of Composer generated Empty Bot.

---

CustomQA Bot in Composer.

This includes a ChangeTopic dialog to get a CurrentTopic value from user prompt selection.

Only regex recognizer and CustomQA Language query from HTML Post.

I filter Custom QA via Metadata filter to currentTopic, Help, ChitChat with QA pairs that have metadata tags TOPIC:xxx for each.

Topic:Help being questions that are not topic specific ("that's not what I was asking", etc.)

unknown intent

HTTP POST METHOD = ${settings.customQuestionAnswer.LanguageEndpointPredictionURL}

= https://customqalanguageprojectxxxxx.cognitiveservices.azure.com/language/:query-knowledgebases?projectName=XXXXXX&api-version=2021-10-01&deploymentName=production

where metadata filtered BODY = { "top": 1, "question": "${turn.activity.text}", "includeUnstructuredSources": true, "confidenceScoreThreshold": "${user.scoreThreshold}", "rankerType": "QuestionOnly", "answerSpanRequest": { "enable": true, "topAnswersWithSpan": 1, "confidenceScoreThreshold": "${user.scoreThresholdPrompt}" }, "filters": { "metadataFilter": { "logicalOperation": "OR", "metadata": [ { "key": "topic", "value": "${user.currentTopic}" }, { "key": "topic", "value": "chitchat" }, { "key": "topic", "value": "help" } ] } } }

. . . Check if good short answer and optionally display

Display top answer

Check if answer had metadata topic:help and query general help answer (which also calls Topic specific help question/answer)

. . . Check for presence of follow-on prompts and call dialog to process

. . .

= = = = = = = = =

Multi follow-on Prompts

including… convert 3 x N prompts array to 1 x N name and 1xN QnAid vectors for user selection (probably a tighter way to do this, but it works)

. . . Get desired prompt from user, and query that specific QnA ID

BODY = { "qnaId": "${dialog.prompt.qnaId}", "question": "${dialog.prompt.displayText}", "includeUnstructuredSources": true, "answerSpanRequest": { "enable": true, "topAnswersWithSpan": 1, "confidenceScoreThreshold": "${user.scoreThresholdPromptShort}" } }

. . . Display answer and query if additional prompts exist in answer

[cwhitten]

Hi everyone.

Again, thank you very much for your patience. Yesterday we published a C#-based Bot Framework package that Composer users can install into their C#-based bot project created from Composer. Note: Node.js bots cannot install or use this package.

Please see the README.md for instructions on how to configure and use this custom recognizer, and if you have issues when following these instructions, open a github issue here, where the team will help troubleshoot.

Note that package enables CQA runtime inferencing for intent recognition, and any knowledge-base authoring will need to be done in the Language Studio. Additionally, using this CQA recognizer is not compatible with any existing knowledge-bases that are authored directly in Composer. You will still create and use the QnA Intent Recognized trigger on the dialog which you've enabled the custom CQA recognizer.

[nghivo]

Thanks @cwhitten , Will try and let you know in case of any bugs

[spmoreno]

Hi everyone.

Again, thank you very much for your patience. Yesterday we published a C#-based Bot Framework package that Composer users can install into their C#-based bot project created from Composer. Note: Node.js bots cannot install or use this package.

Please see the README.md for instructions on how to configure and use this custom recognizer, and if you have issues when following these instructions, open a github issue here, where the team will help troubleshoot.

Note that package enables CQA runtime inferencing for intent recognition, and any knowledge-base authoring will need to be done in the Language Studio. Additionally, using this CQA recognizer is not compatible with any existing knowledge-bases that are authored directly in Composer. You will still create and use the QnA Intent Recognized trigger on the dialog which you've enabled the custom CQA recognizer.

Thanks @cwhitten I have done some tests and it's working as you described.

[fausttiger007]

Has anyone been updating their analytics to digest the Application Insights (workspace not old classic) and Log Analytics Workspace telemetry to work with the new Custom Question Answering or CLU?

we previously modified the Virtual Assistant Analytics template https://microsoft.github.io/botframework-solutions/solution-accelerators/tutorials/view-analytics/1-intro/ with new Workspace App Insights, however the table names have changed, so out of the box things break, but working to remedy that. Also separate api.loganalytics.io queries to Language Resource" and "Azure Bot resource" are required now it seems as data is still separated and needs to be joined.

[nghivo]

Hi @cwhitten and @spmoreno , We have tested and it works well, so far we have not received any error. But I have 2 more questions:

• When can we remove the settings for QnA on the Configuration settings and the Publish Profile?
• How is about the CLU for Bot Framework Composer?

Thanks.

[nghivo]

Hi, Is there any update how to use the new CLU together with CQA?

[hannadeng]

Hi @cwhitten and @spmoreno , We have tested and it works well, so far we have not received any error. But I have 2 more questions:

• When can we remove the settings for QnA on the Configuration settings and the Publish Profile?
• How is about the CLU for Bot Framework Composer?

Thanks.

Hi @cwhitten , could you help check these questions mentioned by @nghivo ?