[Azure] Add client secret (Oauth2) support for eventhub filebeat input

[kaiyan-sheng]

Proposed commit message

This PR is to enhance Azure Event Hub input plugin for Elastic Agent with RBAC authorization (OAuth2) due to security requirements. Previously we only support shared access key (with connection string) for authentication.

The implementation added a new config parameter called auth_type for users to specify authentication method: When auth_type is set to connection_string, or leave it blank: connection_string is required. When auth_typeis set toclient_secret`, oauth2 is used.

Note: We do expect users to use the same auth type for both eventhub and storage account.

OAuth2 specific Configuration Parameters (auth_type=client_secret)

When using OAuth2 authentication, the following parameters are required:

• eventhub_namespace: Fully qualified namespace (e.g., namespace.servicebus.windows.net)
• tenant_id: Azure AD tenant ID
• client_id: Azure AD application (client) ID
• client_secret: Azure AD application client secret
• authority_host: Azure AD authority host (optional, defaults to Azure Public Cloud) https://login.microsoftonline.com is the default.

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] I have made corresponding change to the default configuration files
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

N/A

How to test this PR locally

Setups on Azure side

1. Setup environment variables for setting up eventhub

export RESOURCE_GROUP="kaiyan-resource-group"
export LOCATION="eastus"
export EVENTHUB_NAMESPACE="kaiyan-filebeat-test-ns"
export EVENTHUB_NAME="kaiyan-test-logs"
export STORAGE_ACCOUNT="kaiyanfbstorage"
export STORAGE_CONTAINER="kaiyan-fb-container"
export APP_NAME="filebeat-eventhub-app"

2. Create resource group, eventhub namespace, eventhub

az group create --name $RESOURCE_GROUP --location $LOCATION
az eventhubs namespace create \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --location $LOCATION \
    --sku Standard
az eventhubs eventhub create \
    --resource-group $RESOURCE_GROUP \
    --namespace-name $EVENTHUB_NAMESPACE \
    --name $EVENTHUB_NAME \
    --partition-count 4

3. Create storage account, storage container

az storage account create \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --location $LOCATION \
    --sku Standard_LRS
az storage container create --name $STORAGE_CONTAINER --account-name $STORAGE_ACCOUNT

4. Create Azure AD application, service principle

APP_ID=$(az ad app create \
    --display-name $APP_NAME \
    --query appId --output tsv)
az ad sp create --id $APP_ID

5. Assign eventhub role

EVENTHUB_RESOURCE_ID=$(az eventhubs namespace show \
    --resource-group $RESOURCE_GROUP \
    --name $EVENTHUB_NAMESPACE \
    --query id --output tsv)
az role assignment create \
    --assignee $APP_ID \
    --role "Azure Event Hubs Data Receiver" \
    --scope $EVENTHUB_RESOURCE_ID

6. Get storage account connection string and client secret

STORAGE_CONNECTION_STRING=$(az storage account show-connection-string \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query connectionString --output tsv)
CLIENT_SECRET=$(az ad app credential reset \
    --id $APP_ID \
    --years 1 \
    --query password --output tsv)

OR Instead of getting storage account connection string, assign storage account role:

STORAGE_RESOURCE_ID=$(az storage account show \
    --resource-group $RESOURCE_GROUP \
    --name $STORAGE_ACCOUNT \
    --query id --output tsv)

az role assignment create \
    --assignee $APP_ID \
    --role "Storage Blob Data Contributor" \
    --scope $STORAGE_RESOURCE_ID

7. Create an elastic cloud deployment and get the credentials for testing Filebeat

cloud.id: test-filebeat:foo
cloud.auth: elastic:bar

8. Build and run Filebeat locally

mage update; mage build; ./filebeat -e

9. Get tenant ID:

az account show --query tenantId --output tsv

When no connection_string is specified and no auth_type is specified:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns"
    tenant_id: "<redacted>"
    client_id: "<redacted>"
    client_secret: "<redacted>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_connection_string: "<redacted>"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"

We get error log when starting Filebeat:

Exiting: Failed to start crawler: starting input failed: error while initializing input: reading azure-eventhub input config: connection_string is required when auth_type is empty or set to connection_string accessing 'filebeat.inputs.0' (source:'filebeat.yml')

testing backwards compatibility:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    connection_string:  "<redacted>"
    storage_account: "kaiyanfbstorage"
    storage_account_key: "<redacted>"
    storage_account_container: "kaiyan-fb-container"

Without auth_type specified, by default we are using connection_string to keep backwards compatible. This config still works.

testing with oauth2 for both eventhub and SA:

filebeat.inputs:
  - type: azure-eventhub
    eventhub: "kaiyan-test-logs"
    consumer_group: "$Default"
    eventhub_namespace: "kaiyan-filebeat-test-ns.servicebus.windows.net"
    tenant_id: "<your-tenant-id>" 
    client_id: "<your-app-id>"
    client_secret: "<your-secret>"
    authority_host: "https://login.microsoftonline.com"
    storage_account: "kaiyanfbstorage"
    storage_account_container: "kaiyan-fb-container"
    processor_version: "v2"
    auth_type: "client_secret"

Screenshots

I can see logs getting ingested from Eventhub to elasticsearch with Filebeat:

Logs

I see this in the filebeat log when testing:

{"log.level":"info","@timestamp":"2025-10-28T17:28:38.082-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createContainerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":771},"message":"successfully created container client with OAuth2 authentication","service.name":"filebeat","storage_account":"kaiyanfbstorage","container":"kaiyan-fb-container","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-10-28T17:28:39.858-0600","log.logger":"input.azure-eventhub.oauth2","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/azureeventhub.createConsumerClientWithOAuth2","file.name":"azureeventhub/v2_input.go","file.line":727},"message":"successfully created consumer client with OAuth2 authentication","service.name":"filebeat","namespace":"kaiyan-filebeat-test-ns.servicebus.windows.net","eventhub":"kaiyan-test-logs","tenant_id":"aa40685b-417d-4664-b4ec-8f7640719adb","client_id":"b7a30122-496d-4d84-9200-4f24066b6045","ecs.version":"1.6.0"}

[github-actions[bot]]

:robot: GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify[bot]]

This pull request does not have a backport label. If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏. For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[github-actions[bot]]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-azure-eventhub.md

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[kaiyan-sheng]

@colleenmcginnis Thank you so much for the comments! Yes this change will likely to go in for 9.3.0. Definitely no backports since it is a feature.

[kaiyan-sheng]

Since we'll soon have 3+ authentication types, we should probably consider adding an "auth type" configuration setting for explicit selection. Inferring the authentication type from the option values can become tricky pretty quickly—also considering that we'll have event hub and storage account.

@zmoog I totally forgot we talked about this in our meeting. Yes I will add the auth_type here. I think for backwards compatibility, I will just have connection_string as the default auth_type if auth_type is not specified.

[kaiyan-sheng]

@zmoog While making the changes, I realize we also have storage account auth type to consider. Should we allow customers to have different auth types for storage account and eventhub? Will the customer in any use case perfers that?

[kaiyan-sheng]

@zmoog and I talked offline and decided to only have one auth_type for both eventhub and storage account.

[a03nikki]

@kaiyan-sheng , @zmoog : Can we get this integration updated to include the ability to pass additional OAuth 2.0 parameters as well?

In particular, I am looking for these two

grant_type: refresh_token
refresh_token:  '1234'

There are a number of other Microsoft integrations and inputs that have had the "OAuth2 Endpoint Params" added to them already; these were https://github.com/elastic/integrations/pull/14924 and https://github.com/elastic/integrations/pull/15667.

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b oauth2_azure upstream/oauth2_azure
git merge upstream/main
git push upstream oauth2_azure

[zmoog]

Can we get this integration updated to include the ability to pass additional OAuth 2.0 parameters as well?

Yeah, we can take a look for sure.

The azure-eventhub input will use these service principal/OAuth 2.0 settings to authenticate with both Event Hubs and Storage Account.

@a03nikki, do you have a specific scenario in mind where users would need to customize OAuth 2.0 params? That would help a lot with testing.

[kaiyan-sheng]

@zmoog @a03nikki This PR is focusing on adding the basic oauth2 support with client credentials. Maybe I can add the support for refresh_token in a separate PR?

[zmoog]

@zmoog @a03nikki This PR is focusing on adding the basic oauth2 support with client credentials. Maybe I can add the support for refresh_token in a separate PR?

Makes sense.

@kaiyan-sheng, does the service principal / OAuth2 part in the Azure SDK support passing additional parameters?

[kaiyan-sheng]

@kaiyan-sheng, does the service principal / OAuth2 part in the Azure SDK support passing additional parameters?

@zmoog Are you referring to specifically refresh_token?

[zmoog]

Are you referring to specifically refresh_token?

Not specifically to refresh_token, more to the additional OAuth 2.0 parameters in general.

I have two questions:

• Do Event Hubs and Storage Account need additional OAuth 2.0 parameters? What's the use case here, @a03nikki?
• Does the azidentity package offer enough customization options to add additional OAuth 2.0 parameters, in case we need them?

[kaiyan-sheng]

Does the azidentity package offer enough customization options to add additional OAuth 2.0 parameters, in case we need them?

@zmoog I think azidentity doesn't offer specifically the refresh_token parameter. But we can use azcore.TokenCredential instead. I wonder what is the use case here for refresh_token instead of client_credentials.

[LJaganathan]

Hello! @kaiyan-sheng @zmoog "Azure Event Hub to Elasticsearch Integration Using Azure Client Credentials"

With respect to the integration between Azure Event Hub and Elasticsearch using Azure Active Directory authentication (Client ID and Client Secret), it is understood that this capability will be introduced in version 9.3. Could you please confirm the expected availability timeline for the upgraded version for customers?

Thanks, -Jaganathan L

[github-actions[bot]]

@mergifyio backport 8.19 9.1 9.2 9.3

[mergify[bot]]

backport 8.19 9.1 9.2 9.3

❌ No backport have been created

• #48142 [8.19](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input has been created for branch 8.19 but encountered conflicts
• #48143 [9.1](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input has been created for branch 9.1 but encountered conflicts
• #48144 [9.2](backport #47256) [Azure] Add client secret (Oauth2) support for eventhub filebeat input has been created for branch 9.2 but encountered conflicts
• Backport to branch 9.3 failed

GitHub error: Branch not found