[Meta][ABS] - Improvements and addition of new features to the Azure Blob Storage input

[ShourieG]

Describe the enhancement: This issue is to track various improvements and addition of new features to the Azure Blob Storage input.

Describe a specific use case for the enhancement or feature: With our Azure Blob Storage input being slowly adopted across various integrations, we need to make it more robust and optimal in terms of performance, failure tracking and scalability, hence this Meta issue is created to bring in the necessary changes overtime.

Improvements:

• [x] Add metrics to the Azure Blob Storage input. Separate issue here
• [x] Segregate batch_size from worker count (currently worker count is used as the batch_size to distribute jobs evenly)
• [ ] Segregate cursor save op from event publish and add support for detecting elasticsearch acknowledgement signal and use that to update cursor state.
• [x] Relax Content-Type Restrictions.

New Features

• [x] Make existing retry options user configurable.
• [x] Add support for filtering by prefix expressions.
• [ ] Add support for Azure Service Bus for horizontal scalability of the input.
• [x] Add support for more content-types to the Azure Blob Storage input via content decoders :-
  • [x] JSON/NDJSON
  • [x] CSV
  • [ ] PARQUET
  • [ ] TEXT
• [ ] Add support for alternative state tracking via optional page marker parameter (user configurable with certain ordering limitations)

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)