Expand WordPress Intelligence support to other locales

[kean]

Fixes CMM-762: Excerpts are created in the system language not the content language and CMM-798: Add support for other locales

I smoke-tested it by generating a post in Spanish and testing that the excerpts are also in Spanish. It should work for any other scenarios as well. If there are any issues, I suggest addressing them forward.

[wpmobilebot]

📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.

	App Name	Jetpack
	Configuration	Release-Alpha
	Build Number	30139
	Version	PR #25034
	Bundle ID	com.jetpack.alpha
	Commit	6599afe1bc58540c0bd5944ab2331159eef1a709
	Installation URL	1huohpqk4dgh8

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

[wpmobilebot]

📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.

	App Name	WordPress
	Configuration	Release-Alpha
	Build Number	30139
	Version	PR #25034
	Bundle ID	org.wordpress.alpha
	Commit	6599afe1bc58540c0bd5944ab2331159eef1a709
	Installation URL	6hbimlng5did8

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

[crazytonyli]

I'd expect the generated summary to be in the post's language, rather than the device's language. What do you think?

[crazytonyli]

Oh, sorry, I missed that there are updates to other prompts too.

[crazytonyli]

The post summary generation is not updated in this PR. Probably because that went to a separate code path: LanguageModelHelper.makeGenerateExcerptPrompt.

[crazytonyli]

I used a news article in Chinese for testing. The post tags generation in the "Post Settings" page does not work for me.

Here are some debug prints:

IntelligenceService.suggestTags executed in 2771.9709873199463 ms
Printing description of instructions:
The person's locale is en_NZ.
You are helping a WordPress user add tags to a post or a page.
**Parameters**
- POST_CONTENT: contents of the post (HTML or plain text)
- SITE_TAGS: case-sensitive comma-separated list of the existing tags used elsewhere on the site (not always relevant to the post)
- EXISTING_POST_TAGS: tags already added to the post
**Steps**
- 1. Identify the specific formatting pattern used (e.g., lowercase with underscores, capitalized words with spaces, etc)
- 2. Identify the language used in SITE_TAGS and POST_CONTENT
- 3. Generate a list of ten most relevant suggested tags based on POST_CONTENT and SITE_TAGS relevant to the content.
**Requirements**
- You MUST generate tags in the same language as SITE_TAGS and POST_CONTENT
- Tags MUST match the formatting pattern and language of existing tags
- Do not include any tags from EXISTING_POST_TAGS
- If there are no relevant suggestions, returns an empty list
- Do not produce any output other than the final list of tag
Printing description of prompt:
Suggest up to ten tags for a post.
POST_CONTENT: '''
<p>香港警方表示，周一（12月1日）新发现5具大埔宏福苑火灾遗体，死亡人数增至151人。警方称有些遗体已烧化灰烬，难以辨认。</p>
<p>中国国务院港澳办网站消息，港澳办主任夏宝龙今日在深圳听取大埔火灾有关情况的详细汇报。</p>
<p>火灾后有市民成立“大埔宏福苑火灾关注组”发起联署，提出四大诉求，包括持续支援受灾居民，成立独立调查委员会，全面彻查潜利益输送，以及全力追究监管疏忽，问责政府官员。</p>
<p>其中一名联署发起人关靖丰11月29日被传因涉嫌煽动被香港警方国安处拘捕；翌日，再有消息传出两名男女被捕，其一人为前屯门区议员张锦雄，另一名女子则为义工。</p>
<p>警方未有交代拘捕行动，但回覆BBC中文称，警方采取任何行动，均会按实际情况，依法处理。</p>
<p>12月1日（周一）下午，香港媒体披露关靖丰离开警署的画面，称他获准保释候查。</p>
<p>目前，警方以涉嫌误杀拘捕13人，各人来自以下公司，包括：大判工程（总包商）、工程顾问、二判（分包商）搭棚及外墙工程；调查贪污的廉政公署拘捕12人，包括：工程顾问、承建商及搭棚判头。</p>
<p>现任大埔区区议员黄碧娇12月1日就宏福苑大火发声明称，将亲自向香港警务处报案，要求立即立案调查事件是否涉及现届法团渎职、隐瞒安全风险等严重罪行， 同时向廉政公署（ICAC）要求立案调查，彻查现届法团是否存在贪污渎职，导致消防设备长期失修多年未被执管。黄曾担任宏福苑旧法团顾问，去年9月卸任。</p>
'''
SITE_TAGS: 'high, on, rf, jonnie, hobbs, hello, an, great, bob, he'll'
EXISTING_POST_TAGS: ''
Printing description of response:
(FoundationModels.LanguageModelSession.Response<WordPressShared.SuggestedTagsResult>) response = {
  userPrompt = "Suggest up to ten tags for a post.\n\nPOST_CONTENT: \'\'\'\n<p>香港警方表示，周一（12月1日）新发现5具大埔宏福苑火灾遗体，死亡人数增至151人。警方称有些遗体已烧化灰烬，难以辨认。</p>\n<p>中国国务院港澳办网站消息，港澳办主任夏宝龙今日在深圳听取大埔火灾有关情况的详细汇报。</p>\n<p>火灾后有市民成立“大埔宏福苑火灾关注组”发起联署，提出四大诉求，包括持续支援受灾居民，成立独立调查委员会，全面彻查潜利益输送，以及全力追究监管疏忽，问责政府官员。</p>\n<p>其中一名联署发起人关靖丰11月29日被传因涉嫌煽动被香港警方国安处拘捕；翌日，再有消息传出两名男女被捕，其一人为前屯门区议员张锦雄，另一名女子则为义工。</p>\n<p>警方未有交代拘捕行动，但回覆BBC中文称，警方采取任何行动，均会按实际情况，依法处理。</p>\n<p>12月1日（周一）下午，香港媒体披露\u{e5}"...
  duration = 2.7653398340000002
  feedbackAttachment = nil
  content = {
    tags = 10 values {
      [0] = "high"
      [1] = "on"
      [2] = "rf"
      [3] = "jonnie"
      [4] = "hobbs"
      [5] = "hello"
      [6] = "an"
      [7] = "great"
      [8] = "bob"
      [9] = "he\'ll"
    }
  }
  rawContent = {
    value = {
      storage = object {
        object = {
          properties = 1 key/value pair {
            [0] = {
              key = "tags"
              value = {
                storage = array {
                  array = 10 values {
                    [0] = {
                      storage = string (string = "high")
                      isComplete = true
                    }
                    [1] = {
                      storage = string (string = "on")
                      isComplete = true
                    }
                    [2] = {
                      storage = string (string = "rf")
                      isComplete = true
                    }
                    [3] = {
                      storage = string (string = "jonnie")
                      isComplete = true
                    }
                    [4] = {
                      storage = string (string = "hobbs")
                      isComplete = true
                    }
                    [5] = {
                      storage = string (string = "hello")
                      isComplete = true
                    }
                    [6] = {
                      storage = string (string = "an")
                      isComplete = true
                    }
                    [7] = {
                      storage = string (string = "great")
                      isComplete = true
                    }
                    [8] = {
                      storage = string (string = "bob")
                      isComplete = true
                    }
                    [9] = {
                      storage = string (string = "he\'ll")
                      isComplete = true
                    }
                  }
                }
                isComplete = true
              }
            }
          }
          order = nil
        }
      }
      isComplete = true
    }
    id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
  }
  transcriptEntries = 1 value {
    [2] = response {
      response = {
        id = "13EB559F-C1B9-4309-A3AB-B93B5C2C9E24"
        assetIDs = 3 values {
          [0] = "com.apple.fm.language.instruct_3b.fm_api_generic_12.0.0.13.101733,0"
          [1] = "com.apple.fm.language.instruct_3b.tokenizer_12.0.0.13.202233,0"
          [2] = "com.apple.fm.language.instruct_3b.fm_api_generic.draft_12.0.81307.13.202252,0"
        }
        segments = 1 value {
          [0] = structure {
            structure = {
              id = "0"
              source = "SuggestedTagsResult"
              content = {
                value = {
                  storage = object {
                    object = {
                      properties = 1 key/value pair {
                        [0] = {
                          key = "tags"
                          value = {
                            storage = array {
                              array = 10 values {
                                [0] = {
                                  storage = string (string = "high")
                                  isComplete = true
                                }
                                [1] = {
                                  storage = string (string = "on")
                                  isComplete = true
                                }
                                [2] = {
                                  storage = string (string = "rf")
                                  isComplete = true
                                }
                                [3] = {
                                  storage = string (string = "jonnie")
                                  isComplete = true
                                }
                                [4] = {
                                  storage = string (string = "hobbs")
                                  isComplete = true
                                }
                                [5] = {
                                  storage = string (string = "hello")
                                  isComplete = true
                                }
                                [6] = {
                                  storage = string (string = "an")
                                  isComplete = true
                                }
                                [7] = {
                                  storage = string (string = "great")
                                  isComplete = true
                                }
                                [8] = {
                                  storage = string (string = "bob")
                                  isComplete = true
                                }
                                [9] = {
                                  storage = string (string = "he\'ll")
                                  isComplete = true
                                }
                              }
                            }
                            isComplete = true
                          }
                        }
                      }
                      order = nil
                    }
                  }
                  isComplete = true
                }
                id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
              }
              rawValue = "{\"tags\": [\"high\", \"on\", \"rf\", \"jonnie\", \"hobbs\", \"hello\", \"an\", \"great\", \"bob\", \"he\'ll\"]}"
            }
          }
        }
      }
    }
  }
}
Printing description of response:
(FoundationModels.LanguageModelSession.Response<WordPressShared.SuggestedTagsResult>) response = {
  userPrompt = "Suggest up to ten tags for a post.\n\nPOST_CONTENT: \'\'\'\n<p>香港警方表示，周一（12月1日）新发现5具大埔宏福苑火灾遗体，死亡人数增至151人。警方称有些遗体已烧化灰烬，难以辨认。</p>\n<p>中国国务院港澳办网站消息，港澳办主任夏宝龙今日在深圳听取大埔火灾有关情况的详细汇报。</p>\n<p>火灾后有市民成立“大埔宏福苑火灾关注组”发起联署，提出四大诉求，包括持续支援受灾居民，成立独立调查委员会，全面彻查潜利益输送，以及全力追究监管疏忽，问责政府官员。</p>\n<p>其中一名联署发起人关靖丰11月29日被传因涉嫌煽动被香港警方国安处拘捕；翌日，再有消息传出两名男女被捕，其一人为前屯门区议员张锦雄，另一名女子则为义工。</p>\n<p>警方未有交代拘捕行动，但回覆BBC中文称，警方采取任何行动，均会按实际情况，依法处理。</p>\n<p>12月1日（周一）下午，香港媒体披露\u{e5}"...
  duration = 2.7653398340000002
  feedbackAttachment = nil
  content = {
    tags = 10 values {
      [0] = "high"
      [1] = "on"
      [2] = "rf"
      [3] = "jonnie"
      [4] = "hobbs"
      [5] = "hello"
      [6] = "an"
      [7] = "great"
      [8] = "bob"
      [9] = "he\'ll"
    }
  }
  rawContent = {
    value = {
      storage = object {
        object = {
          properties = 1 key/value pair {
            [0] = {
              key = "tags"
              value = {
                storage = array {
                  array = 10 values {
                    [0] = {
                      storage = string (string = "high")
                      isComplete = true
                    }
                    [1] = {
                      storage = string (string = "on")
                      isComplete = true
                    }
                    [2] = {
                      storage = string (string = "rf")
                      isComplete = true
                    }
                    [3] = {
                      storage = string (string = "jonnie")
                      isComplete = true
                    }
                    [4] = {
                      storage = string (string = "hobbs")
                      isComplete = true
                    }
                    [5] = {
                      storage = string (string = "hello")
                      isComplete = true
                    }
                    [6] = {
                      storage = string (string = "an")
                      isComplete = true
                    }
                    [7] = {
                      storage = string (string = "great")
                      isComplete = true
                    }
                    [8] = {
                      storage = string (string = "bob")
                      isComplete = true
                    }
                    [9] = {
                      storage = string (string = "he\'ll")
                      isComplete = true
                    }
                  }
                }
                isComplete = true
              }
            }
          }
          order = nil
        }
      }
      isComplete = true
    }
    id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
  }
  transcriptEntries = 1 value {
    [2] = response {
      response = {
        id = "13EB559F-C1B9-4309-A3AB-B93B5C2C9E24"
        assetIDs = 3 values {
          [0] = "com.apple.fm.language.instruct_3b.fm_api_generic_12.0.0.13.101733,0"
          [1] = "com.apple.fm.language.instruct_3b.tokenizer_12.0.0.13.202233,0"
          [2] = "com.apple.fm.language.instruct_3b.fm_api_generic.draft_12.0.81307.13.202252,0"
        }
        segments = 1 value {
          [0] = structure {
            structure = {
              id = "0"
              source = "SuggestedTagsResult"
              content = {
                value = {
                  storage = object {
                    object = {
                      properties = 1 key/value pair {
                        [0] = {
                          key = "tags"
                          value = {
                            storage = array {
                              array = 10 values {
                                [0] = {
                                  storage = string (string = "high")
                                  isComplete = true
                                }
                                [1] = {
                                  storage = string (string = "on")
                                  isComplete = true
                                }
                                [2] = {
                                  storage = string (string = "rf")
                                  isComplete = true
                                }
                                [3] = {
                                  storage = string (string = "jonnie")
                                  isComplete = true
                                }
                                [4] = {
                                  storage = string (string = "hobbs")
                                  isComplete = true
                                }
                                [5] = {
                                  storage = string (string = "hello")
                                  isComplete = true
                                }
                                [6] = {
                                  storage = string (string = "an")
                                  isComplete = true
                                }
                                [7] = {
                                  storage = string (string = "great")
                                  isComplete = true
                                }
                                [8] = {
                                  storage = string (string = "bob")
                                  isComplete = true
                                }
                                [9] = {
                                  storage = string (string = "he\'ll")
                                  isComplete = true
                                }
                              }
                            }
                            isComplete = true
                          }
                        }
                      }
                      order = nil
                    }
                  }
                  isComplete = true
                }
                id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
              }
              rawValue = "{\"tags\": [\"high\", \"on\", \"rf\", \"jonnie\", \"hobbs\", \"hello\", \"an\", \"great\", \"bob\", \"he\'ll\"]}"
            }
          }
        }
      }
    }
  }
}

[dangermattic]

	2 Warnings
:warning:	Modules/Package.swift was changed without updating its corresponding Package.resolved. If the change includes adding, removing, or editing a dependency please resolve the Swift packages as appropriate to your project setup (e.g. in Xcode or by running swift package resolve). If the change to the Package.swift did not modify dependencies, ignoring this warning should be safe, but we recommend double checking and running the package resolution just in case. .
:warning:	This PR is larger than 500 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by :no_entry_sign: Danger

[kean]

I added a significant amount of automated tests, improved the prompts to get it to produce the correct language output at a much higher rate, and updated the PR description.

I also have another upcoming PR with additional LLM-based evaluation for automation.

[sonarqubecloud[bot]]

Quality Gate passed

Issues
11 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.9% Duplication on New Code

See analysis details on SonarQube Cloud