Support for content filters in Azure OpenAI

[dkmiller]

Is there an existing issue for this?

• [X] I have searched the existing issues

Community Note

• Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Description

Azure OpenAI has introduced the ability to configure content filters (How to configure content filters with Azure OpenAI Service).

Terraform already supports specifying a custom policy name via rai_policy_name. However, it does not appear that Terraform currently supports creating custom policies.

The underlying REST call made to create a new content filter is looks like below.

PUT https://management.azure.com/subscriptions/5001b330-4456-4782-8a95-5be13fe6f00a/resourceGroups/my-resource-group/providers/Microsoft.CognitiveServices/accounts/my-openai-account/raiPolicies/MyCustomRaiPolicy?api-version=2023-06-01-preview

with a body like

{
  "name": "Test",
  "displayName": "",
  "properties": {
    "basePolicyName": "Microsoft.Default",
    "type": "UserManaged",
    "contentFilters": [
      {
        "name": "hate",
        "blocking": false,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "prompt"
      },
      {
        "name": "sexual",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "high",
        "source": "prompt"
      },
      {
        "name": "selfharm",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "low",
        "source": "prompt"
      },
      {
        "name": "violence",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "prompt"
      },
      {
        "name": "hate",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "low",
        "source": "completion"
      },
      {
        "name": "sexual",
        "blocking": false,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "completion"
      },
      {
        "name": "selfharm",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "completion"
      },
      {
        "name": "violence",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "high",
        "source": "completion"
      }
    ]
  }
}

New or Affected Resource(s)/Data Source(s)

azurerm_cognitive_rai_policy

Potential Terraform Configuration

resource "azurerm_cognitive_rai_policy" "my_policy" {
  name                 = "MyCustomRaiPolicy"
  display_name         = "My custom RAI policy"
  cognitive_account_id = "/subscriptions/5001b330-4456-4782-8a95-5be13fe6f00a/resourceGroups/my-resource-group/providers/Microsoft.CognitiveServices/accounts/my-openai-account"
  content_filters = [
    {
      name = "hate"
      blocking = false
      enabled = true // This is the default.
      allowed_content_level = "high"
      source = "prompt"
    },
    {
      name = "sexual"
      blocking = true // This is the default.
      // Omit default value for `enabled`.
      allowed_content_level = "high"
      source = "completion"
    }
  ]
}

References

RBAC docs for Microsoft.CognitiveServices/accounts/raiPolicies/*: https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations

Original, incorrectly-filed, issue: https://github.com/Azure/terraform-azurerm-openai/issues/28.

[liuwuliuyun]

Hi @dkmiller, thanks for raising this enhancement request. I will track this request and try to implement it. For anyone needs this right now, I would recommend using azapi provider to deploy resource with preview API version.

[dkmiller]

@liuwuliuyun TIL the azapi provider exists, thanks so much! For anyone wishing to use Terraform to create RAI policies in the interim, here's the code I used:

terraform {
  required_providers {
    azapi = {
      source  = "azure/azapi"
      version = "~> 1.8.0"
    }
  }
}

resource "azapi_resource" "no_moderation_policy" {
  type      = "Microsoft.CognitiveServices/accounts/raiPolicies@2023-06-01-preview"
  name      = "NoModerationPolicy"
  parent_id = var.parent_id

  schema_validation_enabled = false

  body = jsonencode({
    displayName = ""
    properties = {
      basePolicyName = "Microsoft.Default"
      type           = "UserManaged"
      contentFilters = [
        { name = "hate", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "sexual", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "selfharm", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "violence", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "hate", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
        { name = "sexual", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
        { name = "selfharm", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
        { name = "violence", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
      ]
    }
  })
}

[tombuildsstuff]

hey @dkmiller

Thanks for opening this issue.

Taking a quick look into this one we're currently using API Version 2023-05-01 for CognitiveServices - however this functionality is only available in API Version 2023-06-01-preview (per above, and it not being present within 2023-05-01).

The SDK we use (hashicorp/go-azure-sdk) is generated from the Azure API Definitions found within the Azure/azure-rest-api-specs repository - however unfortunately the Preview API version isn't available at this time, and so unfortunately we're unable to take advantage of this API version at this point-in-time.

As such I've gone ahead and labelled this issue to indicate that it's blocked on the API Definitions becoming available - but once those are available then we can take a look into supporting this natively within the AzureRM Provider.

Thanks!

[dkmiller]

Thanks @tombuildsstuff! I'm kicking off a conversation with our contacts in Azure, will circle back as soon as there is an update.

[dsoltz1]

Hi @tombuildsstuff and @dkmiller ! I wondered if there is now a possibility to create the content filter with Azurerm provider or if the only valid way is to use Azapi. Thanks!

[illgitthat]

Hi @tombuildsstuff and @dkmiller !

I wondered if there is now a possibility to create the content filter with Azurerm provider or if the only valid way is to use Azapi. Thanks!

It looks like 2023-05-01 is still being used. @liuwuliuyun Are you the right person to ask if there is any update on this enhancement?

[dkmiller]

@dsoltz1 — sadly, to the best of my knowledge Azapi is still the only valid way. The sample snippet I shared above should work.

We have an open ask out to Azure to publish API schemas including content filters, after which Terraform can pick those up. cc @cenhao, who is also interested in this from Airbnb.

[liuwuliuyun]

@illgitthat, thanks for reaching out, I will comment in this thread once there is update on this issue. In the meanwhile, I would suggest using AzAPI provider for any resource in preview API.

[kamikaze]

I'm getting following error:

│ --------------------------------------------------------------------------------
│ RESPONSE 400: 400 Bad Request
│ ERROR CODE: Resource has invalid content filter
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "Resource has invalid content filter",
│     "message": "Policy does not have necessary permission to override base policy. Please check aka.ms/oai/rai/exceptions"
│   }
│ }
│ --------------------------------------------------------------------------------
│ 
│ 
│   with module.azure_infra.azapi_resource.no_moderation_policy,
│   on modules/azure_infra/02-cognitive.tf line 9, in resource "azapi_resource" "no_moderation_policy":
│    9: resource "azapi_resource" "no_moderation_policy" {
│ 
╵

[illgitthat]

@kamikaze you can't disable content filters entirely without first getting approval. Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new

[LeonKalt]

Would it be possible to also activate async streaming mode via terraform?

[mymeatball]

How to disable content flters and acticate async straming mode via Go SDK?

[DanHyland]

@liuwuliuyun any updates given the new 2024-02-01 API release? Having azurerm support will make things like dependency management easier.

[beliven-daniele-sarnari]

Support 👍

[liuwuliuyun]

Hi everyone, I would like to answer some questions in this thread by Q&A format.

Q: How could we create content filter in AOAI now? A: I just checked that as of now, content filter in AOAI is still only able to create by preview versions of API. So I would still recommend that using AzAPI provider to create this resource.

Q: Where do we know which API version I should use and is there a stable version 2024-02-01 released? A: Here is the swagger repo that shows API versions of Cognitive resource provider. You could see that latest stable API version 2023-05-01, creation of this resource is still not supported. There is no version 2024-02-01, we only support cognitive management plane APIs here, version 2024-02-01 could be a data plane API that used for inference. You could use 2023-06-01-preview or later preview versions to create this resource.

Q: Why not support preview versions of API here? A: There might be potential regression issues to have AzureRM support Azure preview features, because preview features do not guarantee its backward and forward feature compatibility. E.g. If AzureRM supports it now but this feature will have new versions in the future which however would include breaking changes to the current preview versions, it would be uneasy for AzureRM to catch up with that new version because it's on purpose to bring as few breaking changes as possible to Terraform AzureRM.

Hope this helps~

[Mithun2507]

Would it be possible to also activate async streaming mode via terraform? you can achieve this using below parameter in the API request mode = "Deferred"

[vjayarag]

Thanks below code worked for me

resource "azapi_resource" "content_filter" {
  type      = "Microsoft.CognitiveServices/accounts/raiPolicies@2023-10-01-preview"
  name      = var.content_filter_name
  parent_id = azurerm_cognitive_account.example.id

  schema_validation_enabled = false

  body = jsonencode({
    properties = {
      mode           = "Default",
      basePolicyName = "Microsoft.Default",
      contentFilters = [
        { name = "hate", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "sexual", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "selfharm", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "violence", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "hate", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "sexual", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "selfharm", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "violence", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "jailbreak", blocking = true, enabled = true, source = "Prompt" },
        { name = "indirect_attack", blocking = true, enabled = true, source = "Prompt" },
        { name = "protected_material_text", blocking = true, enabled = true, source = "Completion" },
        { name = "protected_material_code", blocking = true, enabled = true, source = "Completion" }
      ]
    }
  })
  depends_on = [
    azurerm_cognitive_account.example
  ]
}

And passing the value from var.content_filter_name into var.rai_policy_name as below deployment config

resource "azurerm_cognitive_deployment" "example" {
  name                 = var.name
  cognitive_account_id = var.openai_account_id
  model {
    format  = "OpenAI"
    name    = var.model_name
    version = var.model_version
  }
  scale {
    type = var.scale
    capacity = var.model_capacity
  }
  rai_policy_name = var.rai_policy_name
}```

[DanHyland]

@vjayarag the issue is not the ability to manage this via azapi its that we want azurerm support for multiple reasons including:

• Proper resource dependency management via TF graph
• Remove the need to manage API versions

The latter issue is being seen today within my environment when using the 2023-06-01-preview API, given that preview APIs are mutable.....

The naming convention for the categories (or names) seems to have changed and now are upper case for Hate, Sexual and Violence, where selfharm is still lower case. This type of churn causes a perpetual diff within our environment without us mutating anything on our end. This is extremely sub optimal.

[dkmiller]

Hey @tombuildsstuff, I finally heard back from Microsoft! They've got an updated API version now GA at 2024-06-01.

It should have all latest features including latest content filter annotations.

Hopefully that should unblock implementing this feature request?

[nerddtvg]

@dkmiller: The API you linked is for Azure AI Services and does not include Cognitive Services Accounts -> raiPolicies.

However! There is a new 2024-10-01 Cognitive Services API in stable which does:

https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/resource-manager/Microsoft.CognitiveServices/stable/2024-10-01/cognitiveservices.json#L2241-L2298

[nerddtvg]

It also looks like this was included in hashicorp/go-azure-sdk last week as well: https://github.com/hashicorp/go-azure-sdk/tree/main/resource-manager/cognitive/2024-10-01

[dkmiller]

This is fantastic news! cc @qiaqiatic who is also at Airbnb and using the old "azapi" approach.

[NikhilKosare]

@dkmiller instead of using microsoft.default can i use a custom filter and change the settings inside content filters? i tried updating the value and it throws an error "message": "Policy does not have necessary permission to override base policy. Please check aka.ms/oai/rai/exceptions" "

[DanHyland]

@NikhilKosare I suspect you need to be approved for content filter modifications here: https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUMlBQNkZMR0lFRldORTdVQzQ0TEI5Q1ExOSQlQCN0PWcu

per https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters

[DanHyland]

Another potential reason for having true Azurerm support would be to eliminate this issue. Via azapi, I sporadically get this bug in case sensitive for the categories, but I have not figured out how to fix the perpetual diff. No matter what I supply to the API (along with trying different API versions), I get a diff due to case sensitivity. (apologies not sure why the code paste indentation is getting removed)

~ resource "azapi_resource" "disable_content_filtering_policy" { ~ body = jsonencode( ~ { ~ properties = { ~ contentFilters = [ ~ { ~ name = "Hate" -> "hate" # (4 unchanged attributes hidden) }, ~ { ~ name = "Sexual" -> "sexual" # (4 unchanged attributes hidden) }, { allowedContentLevel = "High" blocking = false enabled = false name = "selfharm" source = "Prompt" }, ~ { ~ name = "Violence" -> "violence" # (4 unchanged attributes hidden) }, ~ { ~ name = "Hate" -> "hate" # (4 unchanged attributes hidden) }, ~ { ~ name = "Sexual" -> "sexual" # (4 unchanged attributes hidden) }, { allowedContentLevel = "High" blocking = false enabled = false name = "selfharm" source = "Completion" }, ~ { ~ name = "Violence" -> "violence" # (4 unchanged attributes hidden) }, ] # (2 unchanged attributes hidden) } # (1 unchanged attribute hidden) } )

[jordyjwilliams]

@dkmiller instead of using microsoft.default can i use a custom filter and change the settings inside content filters? i tried updating the value and it throws an error "message": "Policy does not have necessary permission to override base policy. Please check aka.ms/oai/rai/exceptions" "

TL;DR: What's happening here?

• You are not able to assign blocking = false
  • Unless you have the approval mentioned above.
  • What you can do from this doc is set the SeverityThreshold to High.
  • But you will need to keep blocking = true otherwise you will. be overwriting the base policy (Microsoft.Default).
  • And you will see the above error.

How can I set blocking = false?

• AFAIK You are not able to do this without prior approval for Content Filter modifications.
• EG need to bump from Medium (default) to High... Must keep all default options

NOTE:

• Below snippet assumes using latest azapi (2.0.1 at the time of writing): https://registry.terraform.io/providers/Azure/azapi/latest
  • This does not require the use of jsonencode (since latest version)

Creating content_filter_policy

resource "azapi_resource" "content_filter_policy" {
  # Use latest version here.
  type      = "Microsoft.CognitiveServices/accounts/raiPolicies@2024-10-01"
  name      = "HighModerationPolicy"
  parent_id = azurerm_cognitive_account.example_acct.id

  schema_validation_enabled = false

  body = {
    displayName = ""
    properties = {
      basePolicyName = "Microsoft.Default"
      type           = "UserManaged"
      # NOTE: High -> Allow low and medium. Filter on High only.
      # cannot completely remove without approval from Azure.
      # REF: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters#understand-content-filter-configurability
      contentFilters = [
        # Sadly to turn off `blocking` We require permissions from `Azure`.
        # REF: aka.ms/oai/rai/exceptions
        # Inputs
        { name = "hate", blocking = true, enabled = true, SeverityThreshold = "High", source = "Prompt" },
        { name = "sexual", blocking = true, enabled = true, SeverityThreshold = "High", source = "Prompt" },
        { name = "selfharm", blocking = true, enabled = true, SeverityThreshold = "High", source = "Prompt" },
        { name = "violence", blocking = true, enabled = true, SeverityThreshold = "High", source = "Prompt" },
        # Outputs
        { name = "hate", blocking = true, enabled = true, SeverityThreshold = "High", source = "Completion" },
        { name = "sexual", blocking = true, enabled = true, SeverityThreshold = "High", source = "Completion" },
        { name = "selfharm", blocking = true, enabled = true, SeverityThreshold = "High", source = "Completion" },
        { name = "violence", blocking = true, enabled = true, SeverityThreshold = "High", source = "Completion" }
      ]
    }
  }

  depends_on = [
    azurerm_cognitive_account.example_acct
  ]
}

Referencing in a model azurerm_cognitive_deployment

resource "azurerm_cognitive_deployment" "example" {
  name                 = example
  cognitive_account_id = azurerm_cognitive_account.example_acct.id
  model {
    format  = "OpenAI"
    name    = var.model_name
    version = var.model_version
  }
  scale {
    type = var.scale
    capacity = var.model_capacity
  }
  # Can pull directly from object or set by a local var.
  # Depending on how your infra is setup this style may avoid race conditions
  rai_policy_name = azapi_resource.content_filter_policy.name

}