elasticsearch history support

[jheld]

This looks to add elasticsearch as the task event history backend to flower.

Proposed Solution

Note this essentially still the original PR #821 (I don't recall why I closed it)

This is mainly 2 pieces:

• indexing events into elasticsearch in a somewhat efficient manner (for the process and for elasticsearch)
• searching/sorting/pagination of task history & task lookup by means of elasticsearch.

Done so far (working)

• indexing tasks into elasticsearch

  • I have a background thread that buffers up task events based on a queue and sends bulk index requests into elasticsearch

• Searching (moderate support for different fields), sorting on all fields. The sorting & pagination work but need more QA.

• Dashboard able to pull from elasticsearch (at startup)

Questions

• where will this logic live? flower subcommand? flower proper (w/ elasticsearch flag settings) @johnarnold

• originally I had the indexer outside of flower. It's now in flower, but configured in a hack-off standalone mode, based currently just on argv. There are a few --elasticsearch flags to control the behavior, just for draft/dev mode for now.

• I am using kombu's LRUCache to cache certain search_after queries for the task history pagination. It is my way around the elasticsearch pagination restrictions, and keeps deep pagination requests super performant in my testing.

• can we improve the elasticsearch indexing process?

[jheld]

@auvipy (as a repost from personal account) thank you for the fixes. How far do you think this PR is from being merged? Anything I can do to help? Docs, comments, screenshots, etc.

[auvipy]

I need some more time to properly review and test this....