Add support for Elastic Search as a store

[jeffmcaffer]

Elastic search seems like an interesting choice for store technology. Would give live access to the data in a queryable way.

[markwalkom]

Not sure what this might involve, but I would be interested in discussing it and seeing what sort of resources I might be able to provide (I work @ Elastic).

[jeffmcaffer]

Great @markwalkom. I should not be that bad. Basically the store API as about 5-10 methods like upsert, get, list, ... all basic point or list queries.

What are you doing with ghcrawler and perhaps I can help guide.

[markwalkom]

I was looking at it as part of ​the tool set that ​ Measure ​ uses - https://github.com/MeasureOSS/Measure - and thought if we could enable storage from this in Elasticsearch we can also leverage Kibana for analytics :)

That'd also tie nicely into other tools like https://github.com/morucci/repoxplorer and https://wiki.linuxfoundation.org/chaoss/software​

On 13 Mar 2018 2:30 p.m., "Jeff McAffer" [email protected] wrote:

Great @markwalkom https://github.com/markwalkom. I should not be that bad. Basically the store API as about 5-10 methods like upsert, get, list, ... all basic point or list queries.

What are you doing with ghcrawler and perhaps I can help guide.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Microsoft/ghcrawler/issues/98#issuecomment-372535117, or mute the thread https://github.com/notifications/unsubscribe-auth/ADCYTj7_p0qXG--fQEFx4VVqHIkIsTbXks5tdz1DgaJpZM4N2p7m .

[craigez]

Hi @markwalkom,

I have a basic implementation here: https://github.com/craigez/ghcrawler/commit/4540ca1283d30aa0483a9ca9adf398e0ba41772a (and here: https://source.codeaurora.org/external/qostg/ghcrawler/commit/?h=develop&id=4540ca1283d30aa0483a9ca9adf398e0ba41772a)

It's licensed under the MIT license, but I've been unable to get permission to sign the Microsoft CLA to get the contribution merged into the upstream 😢 . Additionally, we haven't really tested it much as we will probably go an alternate direction for our metrics, but we are still looking at options.

[markwalkom]

that's awesome, thanks @craigez! pity the licensing is causing problems though :(

[craigez]

It's not the licensing, MIT is fine, just the CLA to Microsoft.

[craigez]

I'll poke on this again and see if I can get it resolved soon.

[k----n]

It seems like https://github.com/yougov/mongo-connector can be useful to sync to Elasticsearch