beats [updatecli] update elastic stack version for testing 8.14.0-8783136e

Generated automatically with https://github.com/elastic/beats/actions/runs/8235069466

Bump elastic-stack to latest snapshot version

Update snapshot.yml

1 file(s) updated with "$1:8.14.0-d7334334-SNAPSHOT": * testing/environments/snapshot.yml

GitHub Action workflow link

Created automatically by Updatecli

Options:

Most of Updatecli configuration is done via its manifest(s).

If you close this pull request, Updatecli will automatically reopen it, the next time it runs.
If you close this pull request and delete the base branch, Updatecli will automatically recreate it, erasing all previous commits made.

Feel free to report any issues at github.com/updatecli/updatecli.
If you find this tool useful, do not hesitate to star our GitHub repository as a sign of appreciation, and/or to tell us directly on our chat!

Feb 23 '24 15:02 apmmachine

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-03-13T18:56:29.457+0000
Duration: 136 min 34 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	29167
Skipped	2046
Total	31213

:green_heart: Flaky test report

Tests succeeded.

:robot: GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Feb 28 '24 21:02 elasticmachine

@elastic/obs-infraobs-integrations could you have a look at the failing integration tests?

[2024-03-04T16:23:14.736Z] FAILED tests/system/test_modules.py::Test::test_fileset_file_047_iis - AssertionError: The following expected object doesn't match:
[2024-03-04T16:23:14.736Z]    Diff:
[2024-03-04T16:23:14.736Z]   {'dictionary_item_removed': [root['url.extension']]}, full object: 
[2024-03-04T16:23:14.737Z]   {'log.offset': 0, 'destination.address': '192.168.101.101', 'destination.port': 443, 'destination.ip': '192.168.101.101', 'source.geo.region_iso_code': 'GB-ENG', 'source.geo.continent_name': 'Europe', 'source.geo.city_name': 'London', 'source.geo.country_iso_code': 'GB', 'source.geo.country_name': 'United Kingdom', 'source.geo.region_name': 'England', 'source.geo.location.lon': -0.0931, 'source.geo.location.lat': 51.5142, 'source.address': '81.2.69.145', 'source.port': 12345, 'source.ip': '81.2.69.145', 'fileset.name': 'error', 'url.path': '12.2.1', 'url.original': '12.2.1', 'input.type': 'log', 'iis.error.reason_phrase': 'URL', '@timestamp': '2018-05-05T05:05:55.000Z', 'related.ip': ['81.2.69.145', '192.168.101.101'], 'service.type': 'iis', 'http.request.method': 't3', 'http.response.status_code': 400, 'http.version': '0.9', 'event.original': '2018-05-05 05:05:55 81.2.69.145 12345 192.168.101.101 443 HTTP/0.9 t3 12.2.1 400 - URL -', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'}
[2024-03-04T16:23:14.737Z] assert 1 == 0
[2024-03-04T16:23:14.737Z]  +  where 1 = len({'dictionary_item_removed': [root['url.extension']]})

Mar 04 '24 16:03 rdner

@elastic/obs-infraobs-integrations could you have a look at the failing integration tests?

[2024-03-04T16:23:14.736Z] FAILED tests/system/test_modules.py::Test::test_fileset_file_047_iis - AssertionError: The following expected object doesn't match:
[2024-03-04T16:23:14.736Z]    Diff:
[2024-03-04T16:23:14.736Z]   {'dictionary_item_removed': [root['url.extension']]}, full object: 
[2024-03-04T16:23:14.737Z]   {'log.offset': 0, 'destination.address': '192.168.101.101', 'destination.port': 443, 'destination.ip': '192.168.101.101', 'source.geo.region_iso_code': 'GB-ENG', 'source.geo.continent_name': 'Europe', 'source.geo.city_name': 'London', 'source.geo.country_iso_code': 'GB', 'source.geo.country_name': 'United Kingdom', 'source.geo.region_name': 'England', 'source.geo.location.lon': -0.0931, 'source.geo.location.lat': 51.5142, 'source.address': '81.2.69.145', 'source.port': 12345, 'source.ip': '81.2.69.145', 'fileset.name': 'error', 'url.path': '12.2.1', 'url.original': '12.2.1', 'input.type': 'log', 'iis.error.reason_phrase': 'URL', '@timestamp': '2018-05-05T05:05:55.000Z', 'related.ip': ['81.2.69.145', '192.168.101.101'], 'service.type': 'iis', 'http.request.method': 't3', 'http.response.status_code': 400, 'http.version': '0.9', 'event.original': '2018-05-05 05:05:55 81.2.69.145 12345 192.168.101.101 443 HTTP/0.9 t3 12.2.1 400 - URL -', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'}
[2024-03-04T16:23:14.737Z] assert 1 == 0
[2024-03-04T16:23:14.737Z]  +  where 1 = len({'dictionary_item_removed': [root['url.extension']]})

@muthu-mps,

Can you take a look at this failure?

Mar 05 '24 05:03 lalit-satapathy

@elastic/obs-infraobs-integrations could you have a look at the failing integration tests?

[2024-03-04T16:23:14.736Z] FAILED tests/system/test_modules.py::Test::test_fileset_file_047_iis - AssertionError: The following expected object doesn't match:
[2024-03-04T16:23:14.736Z]    Diff:
[2024-03-04T16:23:14.736Z]   {'dictionary_item_removed': [root['url.extension']]}, full object: 
[2024-03-04T16:23:14.737Z]   {'log.offset': 0, 'destination.address': '192.168.101.101', 'destination.port': 443, 'destination.ip': '192.168.101.101', 'source.geo.region_iso_code': 'GB-ENG', 'source.geo.continent_name': 'Europe', 'source.geo.city_name': 'London', 'source.geo.country_iso_code': 'GB', 'source.geo.country_name': 'United Kingdom', 'source.geo.region_name': 'England', 'source.geo.location.lon': -0.0931, 'source.geo.location.lat': 51.5142, 'source.address': '81.2.69.145', 'source.port': 12345, 'source.ip': '81.2.69.145', 'fileset.name': 'error', 'url.path': '12.2.1', 'url.original': '12.2.1', 'input.type': 'log', 'iis.error.reason_phrase': 'URL', '@timestamp': '2018-05-05T05:05:55.000Z', 'related.ip': ['81.2.69.145', '192.168.101.101'], 'service.type': 'iis', 'http.request.method': 't3', 'http.response.status_code': 400, 'http.version': '0.9', 'event.original': '2018-05-05 05:05:55 81.2.69.145 12345 192.168.101.101 443 HTTP/0.9 t3 12.2.1 400 - URL -', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'}
[2024-03-04T16:23:14.737Z] assert 1 == 0
[2024-03-04T16:23:14.737Z]  +  where 1 = len({'dictionary_item_removed': [root['url.extension']]})

@muthu-mps,

Can you take a look at this failure?

I am looking into the issue.

Mar 05 '24 05:03 muthu-mps

Observations

The python integrations test failure reported in the IIS error log module. I have performed triaging on the error first to know is this happening only with the IIS logs.

No, The failure is reported with the other modules as well which is using the uri_parts ingest node processor.

Logs producing the Error

IIS Error log module [2024-03-04T16:23:14.737Z] {'log.offset': 0, 'destination.address': '192.168.101.101', 'destination.port': 443, 'destination.ip': '192.168.101.101', 'source.geo.region_iso_code': 'GB-ENG', 'source.geo.continent_name': 'Europe', 'source.geo.city_name': 'London', 'source.geo.country_iso_code': 'GB', 'source.geo.country_name': 'United Kingdom', 'source.geo.region_name': 'England', 'source.geo.location.lon': -0.0931, 'source.geo.location.lat': 51.5142, 'source.address': '81.2.69.145', 'source.port': 12345, 'source.ip': '81.2.69.145', 'fileset.name': 'error', 'url.path': '12.2.1', 'url.original': '12.2.1', 'input.type': 'log', 'iis.error.reason_phrase': 'URL', '@timestamp': '2018-05-05T05:05:55.000Z', 'related.ip': ['81.2.69.145', '192.168.101.101'], 'service.type': 'iis', 'http.request.method': 't3', 'http.response.status_code': 400, 'http.version': '0.9', 'event.original': '2018-05-05 05:05:55 81.2.69.145 12345 192.168.101.101 443 HTTP/0.9 t3 12.2.1 400 - URL -', 'event.kind': 'event', 'event.module': 'iis', 'event.category': ['web', 'network'], 'event.type': ['connection'], 'event.dataset': 'iis.error', 'event.outcome': 'failure'}
O365 audit log module 'log.offset': 10504, 'rule.name': 'Low volume of content detected test', 'rule.id': '8398c03a-a00d-42bb-8f80-ead0ad04e1df', 'fileset.name': 'audit', 'url.path': '/testsiem2.onmicrosoft.com/sharepoint', 'url.original': 'https://example.net/testsiem2.onmicrosoft.com/sharepoint', 'url.scheme': 'https', 'url.domain': 'example.net', 'tags': ['forwarded'], 'o365.audit.ObjectId': '<AM0PR05MB4803CDA6206C2F2FEB36DB5AB8EC0@AM0PR05MB4803.eurprd05.prod.outlook.com>', 'o365.audit.UserKey': '1153801116545789462', 'o365.audit.OrganizationId': '0e1dddce-163e-4b0b-9e33-87ba56ac4655', 'o365.audit.Operation': 'DlpRuleMatch', 'o365.audit.IncidentId': 'c1dc582b-fa61-6020-1800-08d7b966ec64', 'o365.audit.SensitiveInfoDetectionIsIncluded': False, 'o365.audit.Workload': 'Exchange', 'o365.audit.RecordType': 13, 'o365.audit.Version': 1, 'o365.audit.UserId': 'DlpAgent', 'o365.audit.CreationTime': '2020-02-24T20:11:15', 'o365.audit.SharePointMetaData.itemCreationTime': '2020-02-20T11:23:45', 'o365.audit.SharePointMetaData.UniqueID': '8e103f2f-b293-4062-38b8-08d7b965b2fa', 'o365.audit.SharePointMetaData.FileName': 'Company-Internal-Financial.docx', 'o365.audit.SharePointMetaData.FilePathUrl': 'https://example.net/testsiem2.onmicrosoft.com/sharepoint', 'o365.audit.SharePointMetaData.LastModifiedTime': '2020-02-24T12:13:14Z', 'o365.audit.SharePointMetaData.FileOwner': '[email protected]', 'o365.audit.SharePointMetaData.From': '[email protected]', 'o365.audit.PolicyDetails': [{'PolicyName': 'test', 'Rules': [{'Actions': ['NotifyUser'], 'RuleMode': 'Enable', 'RuleId': '8398c03a-a00d-42bb-8f80-ead0ad04e1df', 'ConditionsMatched': {'SensitiveInformation': [{'UniqueCount': 1, 'Confidence': 75, 'Count': 1, 'Location': 'Message Body', 'SensitiveType': '419f449f-6d9d-4be1-a154-b531f7a91b41'}, {'UniqueCount': 1, 'Confidence': 75, 'Count': 1, 'Location': 'Message Body', 'SensitiveType': 'b8fe86d1-c056-453b-bfaa-9fe698699ecc'}], 'OtherConditions': [{'Value': 'IncludeExternalUsers', 'Name': 'AccessScope'}]}, 'Severity': 'Low', 'RuleName': 'Low volume of content detected test'}], 'PolicyId': '88956b36-45b3-4828-bf53-78603c0e5f58'}], 'o365.audit.Id': 'a42123a9-1c07-4dde-9be6-ac71cb9fd16b', 'o365.audit.UserType': 4, 'input.type': 'log', '@timestamp': '2020-02-24T20:11:15.000Z', 'file.inode': '8e103f2f-b293-4062-38b8-08d7b965b2fa', 'file.owner': '[email protected]', 'file.name': 'Company-Internal-Financial.docx', 'file.mtime': '2020-02-24T12:13:14.000Z', 'related.user': ['alice', '[email protected]'], 'service.type': 'o365', 'organization.id': '0e1dddce-163e-4b0b-9e33-87ba56ac4655', 'host.name': 'testsiem2.onmicrosoft.com', 'host.id': '0e1dddce-163e-4b0b-9e33-87ba56ac4655', 'event.severity': 2, 'event.code': 'ComplianceDLPExchange', 'event.provider': 'Exchange', 'event.kind': 'alert', 'event.module': 'o365', 'event.action': 'DlpRuleMatch', 'event.id': 'a42123a9-1c07-4dde-9be6-ac71cb9fd16b', 'event.type': 'access', 'event.category': 'file', 'event.dataset': 'o365.audit', 'event.outcome': 'success', 'user.domain': 'testsiem2.onmicrosoft.com', 'user.name': 'alice', 'user.id': '[email protected]', 'user.email': '[email protected]'

Root Cause

Performed the following steps to identify the root cause of the issue,

From the above log trace dictionary_item_removed for url.extension. I have performed the test run against the previous build versions. There is no build error but I was able to trace that the url.extension has incorrect values captured when the url has dot notation in between.
But the above issue is there for quite long. This is not causing the build failure then I have started looking into the Elasticsearch ingest node uri_parts processor to verify if something has changed.

Simulate uri_parts processor

8.14.SNAPSHOT
8.12
We are almost there, In 8.14-SNAPSHOT we can see that the url.extension field is removed from the processor itself. This fixes the issue for the logs which generates incorrect extension values.
Here is the issue which fixes the incorrect extension values from the uri_parts processor.https://github.com/elastic/elasticsearch/pull/105689

Solution

Verify and remove the url.extension field from the sample events which has incorrect values. This applies to both the Beats and integrations. Currently based on test failure the issue is observed in IIS and O365 modules.

Mar 07 '24 06:03 muthu-mps

The failing test were fixed in https://github.com/elastic/beats/pull/38216

Mar 13 '24 18:03 cmacknz

:broken_heart: Build Failed

Buildkite Build
Commit: b1ef407825fcbe3291d48f17587ddbb8fe954f09

Failed CI Steps

History

:broken_heart: Build #2304 failed 21310e126fc2ce34fa0586e210de238ededa8a07
:broken_heart: Build #2256 failed 5ed26080a13dd432b42e7d5990809cb0ba9eac01
:broken_heart: Build #2216 failed d8c7a78bac1f87f031953ebf5b04a63d922d9203
:broken_heart: Build #2177 failed 288c37b81d12174a9400e0191cec751cefa0d077

cc @apmmachine

Mar 13 '24 19:03 elasticmachine

:broken_heart: Build Failed

Buildkite Build
Commit: b1ef407825fcbe3291d48f17587ddbb8fe954f09

Failed CI Steps

History

:broken_heart: Build #3645 failed 21310e126fc2ce34fa0586e210de238ededa8a07
:broken_heart: Build #3597 failed 5ed26080a13dd432b42e7d5990809cb0ba9eac01
:broken_heart: Build #3557 failed d8c7a78bac1f87f031953ebf5b04a63d922d9203
:broken_heart: Build #3518 failed 288c37b81d12174a9400e0191cec751cefa0d077

cc @apmmachine

Mar 13 '24 19:03 elasticmachine

:green_heart: Build Succeeded

Buildkite Build
Commit: b1ef407825fcbe3291d48f17587ddbb8fe954f09

History

:green_heart: Build #2007 succeeded 21310e126fc2ce34fa0586e210de238ededa8a07
:green_heart: Build #1959 succeeded 5ed26080a13dd432b42e7d5990809cb0ba9eac01
:broken_heart: Build #1919 failed d8c7a78bac1f87f031953ebf5b04a63d922d9203
:broken_heart: Build #1881 failed 288c37b81d12174a9400e0191cec751cefa0d077
:green_heart: Build #1832 succeeded 9541c0511bf00b00e61df4aa0d210275af3bf7d8
:green_heart: Build #1797 succeeded 569e4f3497ef4d9c62cc56071903a3ffccf494ca

cc @apmmachine

Mar 13 '24 19:03 elasticmachine

:green_heart: Build Succeeded

Buildkite Build
Commit: b1ef407825fcbe3291d48f17587ddbb8fe954f09

History

:green_heart: Build #2014 succeeded 21310e126fc2ce34fa0586e210de238ededa8a07
:green_heart: Build #1966 succeeded 5ed26080a13dd432b42e7d5990809cb0ba9eac01
:green_heart: Build #1926 succeeded d8c7a78bac1f87f031953ebf5b04a63d922d9203
:broken_heart: Build #1888 failed 288c37b81d12174a9400e0191cec751cefa0d077
:green_heart: Build #1839 succeeded 9541c0511bf00b00e61df4aa0d210275af3bf7d8
:green_heart: Build #1804 succeeded 569e4f3497ef4d9c62cc56071903a3ffccf494ca

cc @apmmachine

Mar 13 '24 19:03 elasticmachine

:green_heart: Build Succeeded

Buildkite Build
Commit: b1ef407825fcbe3291d48f17587ddbb8fe954f09

History

:green_heart: Build #1170 succeeded 21310e126fc2ce34fa0586e210de238ededa8a07
:green_heart: Build #1122 succeeded 5ed26080a13dd432b42e7d5990809cb0ba9eac01
:green_heart: Build #1082 succeeded d8c7a78bac1f87f031953ebf5b04a63d922d9203
:green_heart: Build #1044 succeeded 288c37b81d12174a9400e0191cec751cefa0d077
:green_heart: Build #995 succeeded 9541c0511bf00b00e61df4aa0d210275af3bf7d8
:green_heart: Build #960 succeeded 569e4f3497ef4d9c62cc56071903a3ffccf494ca

cc @apmmachine

Mar 13 '24 19:03 elasticmachine

:green_heart: Build Succeeded

Buildkite Build
Commit: b1ef407825fcbe3291d48f17587ddbb8fe954f09

History

:broken_heart: Build #3221 failed 21310e126fc2ce34fa0586e210de238ededa8a07
:green_heart: Build #3173 succeeded 5ed26080a13dd432b42e7d5990809cb0ba9eac01
:green_heart: Build #3133 succeeded d8c7a78bac1f87f031953ebf5b04a63d922d9203
:broken_heart: Build #3095 failed 288c37b81d12174a9400e0191cec751cefa0d077
:green_heart: Build #3046 succeeded 9541c0511bf00b00e61df4aa0d210275af3bf7d8

cc @apmmachine

Mar 13 '24 19:03 elasticmachine

beats beats copied to clipboard

[updatecli] update elastic stack version for testing 8.14.0-8783136e

Bump elastic-stack to latest snapshot version

:green_heart: Build Succeeded

Build stats

Test stats :test_tube:

:green_heart: Flaky test report

:robot: GitHub comments

Observations

Logs producing the Error

Root Cause

Simulate uri_parts processor

Solution

:broken_heart: Build Failed

Failed CI Steps

History

:broken_heart: Build Failed

Failed CI Steps

History

:green_heart: Build Succeeded

History

:green_heart: Build Succeeded

History

:green_heart: Build Succeeded

History

:green_heart: Build Succeeded

History

beats
beats copied to clipboard