[8.19](backport #45088) [filebeat][GCS] - Relax content-type restrictions along with some basic cleanup and refactor

[mergify[bot]]

Type of change

• Enhancement
• Cleanup
• Docs

Note

This PR is best reviewed commit wise.

This change by itself does not allow new content-types/encoding, but relaxes the strict content-type rejection error due to the object metadata not being populated. It allows the users to specify the correct content-type/encoding within the bounds of the current restrictions.

Proposed commit message

Sometimes we require to specify the content-type/encoding of objects due to 
limitations on the users end in accessing the remote storage. This PR adds
a generic way to validate config structs for container level overrides and 
introduces support for specifying custom content-types and encodings.

Checklist

• [x] My code follows the style guidelines of this project
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes https://github.com/elastic/beats/issues/44628
• Relates https://github.com/elastic/beats/issues/41107, https://github.com/elastic/beats/pull/44583

Use cases

Screenshots

Logs

---

This is an automatic backport of pull request #45088 done by [Mergify](https://mergify.com).

[mergify[bot]]

Cherry-pick of 045d0d83add6b847cc0045c55aa5361f128b5f1b has failed:

On branch mergify/bp/8.19/pr-45088
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 045d0d83a.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc
	modified:   x-pack/filebeat/include/list.go
	modified:   x-pack/filebeat/input/gcs/mock/mock.go
	modified:   x-pack/filebeat/input/gcs/scheduler.go
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/array-at-root.json -> x-pack/filebeat/input/gcs/testdata/array-at-root.json
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/events-array.json -> x-pack/filebeat/input/gcs/testdata/events-array.json
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/log.json -> x-pack/filebeat/input/gcs/testdata/log.json
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/log.ndjson -> x-pack/filebeat/input/gcs/testdata/log.ndjson
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/multiline.json -> x-pack/filebeat/input/gcs/testdata/multiline.json
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/multiline.json.gz -> x-pack/filebeat/input/gcs/testdata/multiline.json.gz
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/nested-arrays.json -> x-pack/filebeat/input/gcs/testdata/nested-arrays.json
	renamed:    x-pack/filebeat/input/gcs/mock/testdata/txn1.csv -> x-pack/filebeat/input/gcs/testdata/txn1.csv

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   docs/reference/filebeat/filebeat-input-gcs.md
	both modified:   x-pack/filebeat/input/gcs/config.go
	both modified:   x-pack/filebeat/input/gcs/input.go
	both modified:   x-pack/filebeat/input/gcs/input_test.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[mergify[bot]]

This pull request is now in conflicts. Could you fix it? 🙏 To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mergify/bp/8.19/pr-45088 upstream/mergify/bp/8.19/pr-45088
git merge upstream/8.19
git push upstream mergify/bp/8.19/pr-45088

[mergify[bot]]

This pull request has not been merged yet. Could you please review and merge it @ShourieG? 🙏