Discussion: Dealing with ES 2.0

[MichaelXavier]

Just thought I'd start an issue for this since the topic has come up a few times. I personally don't have an immediate need for 2.0 since I don't run it but I think it will start being a default choice for self-hosted solutions since ES seem to be burying 1.x documentation.

One way to do it would be to split the module tree and share types and APIs where possible by moving them to a common internal module. Some of the APIs seem to be the ame according to @MHova's experiences. To avoid breaking everything the default Database.Bloodhound module could alias to Database.Bloodhound.V1.

Another alternative is to keep one module and have the BHEnv know what version of ES its targeting. Maybe it would be a GADT or phantom type or something. This could also probably apply to MonadBH, such that shared operations would run in MonadBH v m and v2-only operations in MonadBH V2 m.

I think it will end up being a massive amount of work though so it will probably get done mostly when some enterprise user needs it and can invest the time. I think serialization will be especially bad since JSON instances will likely vary between versions. ToJSON/FromJSON being globally scoped and all makes it hard to adapt to one version or another.

Lastly, I wonder if a good first step would be to create a branch and run the test suite with V2 installed. That will give us a good progress report. If we go down the split module tree route, we'll probably need to break up the test suite accordingly (which is fine by me since one large monadic operation puts SHM in a chokehold).

Thoughts?

[bitemyapp]

Opening statement: these are my preferences stated frankly. I'm going to lean hard based on what you think is best.

I think maintaining concurrent support when things conflict is probably a boondoggle. This library has attracted some excellent contributors, but you're the only long-term labor input it has had so far. Though @MHova has been great at coordinating and communicating on the issues.

My preference would be to support one-set-of-versions-at-a-time in one module, but something could be coordinated between a maintainer committing to support a V1 module who is using it in prod and a committed maintainer who is using the V2 module in prod, then that's fine. Without the handshake and mutual labor inputs, I'm apt to simply wait it out and see what people actually need. @MHova I think is using 2.0 and 2.1 and they haven't noticed any of the breakage (lol) that our unit tests did. I'll be more hesitant to do this without having a PIC for each version though as I don't trust my own ability to remember what's-what by myself.

If a larger org is willing pay for someone's time to handle the maintenance load, an internal employee or otherwise, then I'm open to something else. I think things have been done incredibly cheaply so far compared to the beast that is the .NET client for example, but there's only a handful of us and the book has annihilated my time for OSS coding.

What do you think? Feel free to pipe up @MHova @bermanjosh @centromere and any others that care.

(which is fine by me since one large monadic operation puts SHM in a chokehold).

Oh, could you expand "SHM" for me please? Don't think I understood.

[bermanjosh]

I'm still stuck on the 0.8 release; we're only (hopefully hopefully hopefully) going over to GHC 7.10 this month, and I've been focusing on non-searching issues for the last few months, so I haven't totally kept abreast regarding what's been changing with ES.

Are there any other libs facing similar backend-versioning issues which we could investigate? I'm fine with the V1 / V2 modules split, but I don't much experience with this sort of library maintenance, so don't weigh my opinion too heavily.

I'm happy to pitch in if someone else is able to coordinate, but as everyone's seen with how much my code has been (correctly) rewritten, I don't think I'm the right guy to maintain.

[bitemyapp]

@bermanjosh I think the limiting factor right now is someone that wants or cares about fully correct 2.0 support.

btw - Are you on Aeson 0.8/0.9 if you're using Bloodhound 0.8? Trying to get a read on what versions of things people are on.

[MichaelXavier]

Oh, could you expand "SHM" for me please? Don't think I understood.

Sorry, structured-haskell-mode. I think it keeps some kind of coprocess and repeatedly parses a file (or maybe even just a current scope) into the AST. For whatever reason it seems to be less performant for very large monadic blocks like in tests.hs. Obviously I don't advocate changing the actual code to accommodate my editor preferences, I just turn off structured-haskell-mode for the test buffer.

I mainly brought this ticket up because I was surprised to find someone somewhat successfully using Bloodhound for ES 2.0. They will undoubtedly start running into issues and will need guidance on how to proceed with it. The appropriate workflow may be a branch that people can vendor until we get to an acceptable level of coverage.

I do think we would need at least one person who has some skin in the game, I agree. The reason is that if you don't use some feature or version of ES in production, it becomes hard to be a good gatekeeper against issues in the features coming in. I'm going to be on 1.x for the forseeable future I think.

[MHova]

I think it'll be less of a maintenance nightmare to just split into V1 and V2 modules. I think this is what @bitemyapp and @MichaelXavier are leaning towards?

Welp, I just ran into my first >=2.0 breakage. I'm specifically on 2.1, by the way. The symptom is that _cache is no longer a valid field on Or Filters and ElasticSearch balks if you attempt to specify it. Currently OrFilter will stick a _cache in the JSON no matter what.

Some interesting notes on this problem:

1. Client-specified filter caching is going away altogether in 2.1. ElasticSearch will now decide on its own whether or not to cache a filter.
2. The above link claims that _cache will merely be ignored if present. This seems to be true for And Filters and Terms Filters, but for some freakin' reason ElasticSearch throws a fit if you specify it for an Or Filter (which I just started using).
3. Queries and Filters were merged in general in 2.0. And Filters and Or Filters are still allowed, but they're merely syntactic sugar for must and should bool queries underneath the hood.

My immediate solution is to change OrFilter.cache to a Maybe Cache in my vendor version. It could be argued that all the cache fields in all the Filter data constructors should be changed to Maybe Cache anyway, even in V1, to allow clients the option of not specifying it and fall back to ElasticSearch's default. Of course, the long term solution would be to not even have such a field in the V2 Query/Filter data types.

Edit: cache is of type Cache, not Bool

[bitemyapp]

I think it'll be less of a maintenance nightmare to just split into V1 and V2 modules. I think this is what @bitemyapp and @MichaelXavier are leaning towards?

No, I'm leaning towards "we support one set of versions in one module". Elasticsearch moves quickly enough and we don't have enough labor that maintaining concurrent V1 and V2 versions is worth it IMO. They can use an older version of Bloodhound if needs be, in my opinion.

I expressed an openness to being wrong but don't mistake my describing of how to handle V1/V2 split as a preference. I was describing what would need to happen to make me more comfortable with it.

Client-specified filter caching is going away altogether in 2.1. ElasticSearch will now decide on its own whether or not to cache a filter. The above link claims that _cache will merely be ignored if present. This seems to be true for And Filters and Terms Filters, but for some freakin' reason ElasticSearch throws a fit if you specify it for an Or Filter (which I just started using).

We can add a warning not to use it with >=2.0, which I'm not a fan of, but I don't think this issue alone is worth a module split. That is very annoying.

Queries and Filters were merged in general in 2.0. And Filters and Or Filters are still allowed, but they're merely syntactic sugar for must and should bool queries underneath the hood.

Costs us nothing to keep it for now.

If we can appoint a PIC for a V1 and a V2 (separate PICs, ideally that are using it in production at work and are paid to write Haskell in some capacity), then I'm warmer to the idea. Otherwise, I don't see any reason to privilege not-really-supported-any-longer Elasticsearch versions with doubling our support load.

Is there a reason we think it's necessary to have concurrent modules on an ongoing basis? Is it hard to migrate Elasticsearch clusters? Do people just not care what version they run? I'd solved this before by replicating documents, catching up the failover to the prod cluster, then doing an Indiana Jones bag swap for the upgrade. It's easier if you have a proxy you can program between you and the cluster.

[MHova]

What does PIC stand for?

Can you elaborate on what you mean by "we support one set of versions in one module"? Do you mean splitting the Bloodhound library into separate libraries entirely?

[MHova]

For what it's worth, I opened an issue with ES about the _cache problem: https://github.com/elastic/elasticsearch/issues/16441

[bitemyapp]

What does PIC stand for?

Person in charge.

Can you elaborate on what you mean by "we support one set of versions in one module"? Do you mean splitting the Bloodhound library into separate libraries entirely?

No, I'm saying we support one set of versions in one library. That's it. We don't have the labor that I'm aware of at the moment to support concurrent sets of versions, however they are organized. I am very wary of mission creep and I don't yet have a reason to believe people on 1.x couldn't just use an older version of Bloodhound or upgrade their ES cluster (which is wise anyway; this isn't PostgreSQL running on a FreeBSD box).

[MHova]

Ah so you're saying we should move to a new major version of Bloodhound that only supports 2.x?

[bitemyapp]

@MHova if and when someone says they need 2.0-only support. For now the overlap seems to be mostly okay. Some documentation tweaks might help prevent nasty surprises.

[MichaelXavier]

Thanks for clarifying @bitemyapp I think we may have been incorrectly in agreement. I think it may be a mistake to do a cutoff on elasticsearch 1.x and have all future dev happen on 2.0. I think for things that are relatively easy to change like haskell libraries, having a cutoff and having everyone suck it up and move to the next is a pretty fair proposition. I think with databases you run a greater risk of alienating users and when they discover bugs (or likely missing features) with the older versions, you'll either need to roll those into an old version tree or wish them luck with a fork.

I have a feeling like Elastic is trying hard not to pull a Python 3 by hiding documentation but I think a lot of people are on ElasticSearch 1.x. Most of the people I know who use it are. Amazon's brand new hosted ElasticSearch is only 1.x and I would be surprised if they added 2.0 support in the next few quarters or even this year. If someone did want to upgrade, I don't know how easy it is to even migrate.

To be clear, I completely understand the desire to have a PIC and think its crucial. Bloodhound has never claimed to be a complete API client, people add what they need when they come across it. I think the same should apply with major version numbers: stakeholders that really need 2.0 support should step up for maintainership. I just think we need to be in touch with what versions the community actually uses and try to support a reasonable subset of versions to be useful to most of them. I do think there will come a time where 1.x support can be sunset but I don't think Elastic is being realistic about when that should be ;)

[MHova]

As an ElasticSearch 2.x user, I would self-servingly love to see Bloodhound abandon 1.x and charge forward with 2.x, but I must agree with @MichaelXavier. Bloodhound is a great library and I'd hate to see it alienate its userbase. As @MichaelXavier pointed out, AWS ElasticSearch is still on 1.x (1.5.2 to be exact).

I think the most dead-simple and maintainable way to handle the 1.x/2.x split is to just give up trying to come up with something slick: stick a V1 in all existing Bloodhound files, make a V2 duplicate of ALL files, and tell maintainers to have at it. It pains the obsessive programmer inside me to suggest that, but I think it'll cause the fewest headaches. V1 maintainers and V2 maintainers (i.e. me :laughing:) can make changes to their desired versions without fear of breaking the other. We can nudge contributors in PR reviews to apply their changes to both versions if they are applicable and useful to both.

[bitemyapp]

Amazon's brand new hosted ElasticSearch is only 1.x and I would be surprised if they added 2.0 support in the next few quarters or even this year.

That's a strong argument for staying where we are in terms of official support.

To be clear, I completely understand the desire to have a PIC and think its crucial. Bloodhound has never claimed to be a complete API client, people add what they need when they come across it. I think the same should apply with major version numbers: stakeholders that really need 2.0 support should step up for maintainership. I just think we need to be in touch with what versions the community actually uses and try to support a reasonable subset of versions to be useful to most of them. I do think there will come a time where 1.x support can be sunset but I don't think Elastic is being realistic about when that should be ;)

All agreed.

As an ElasticSearch 2.x user, I would self-servingly love to see Bloodhound abandon 1.x and charge forward with 2.x, but I must agree with @MichaelXavier.

Fair enough.

I think the most dead-simple and maintainable way to handle the 1.x/2.x split is to just give up trying to come up with something slick: stick a V1 in all existing Bloodhound files, make a V2 duplicate of ALL files, and tell maintainers to have at it.

I think we can be smarter than that and have a mutually compatible intersection module. I'm open to this setup, but I prefer the status quo for now unless you are putting yourself forward as a maintainer for the V2 module.

If whomever volunteers to start the module split doesn't maintain the V2 module (which for our purposes is currently the red-headed stepchild), the module split will be axed if they can't keep up with the releases. Not that there are that many or that they are that frequent, but they still have to get done and I loathe pushing out code without passing tests. Technically I could say we support 2.0 and probably get away with it, but because not all of the tests pass, I haven't.

Current take-away:

Status quo stands. Lets try to document sharp corners for our 2.0 people.

I'd like to put off theorycrafting the module split until we have a PIC.

If it's all right with you both I consider this matter closed for now. We can reopen it when we either have a PIC volunteer (no longer requiring 2, I think Michael and I can handle 1.x), something changes, or you feel there's some unfinished business.

[MHova]

But finding a "mutually compatible intersection module" is haaaaard and I'm laaaaaazy

Though seriously, what is the benefit of a mutually compatible module? Yes, there are some things that are clearly shareable (BHEnv, some of the top level Client functions, etc), but it takes a non-trivial amount of upfront effort to identify these intersections (the Types in particular) and having intersections may complicate future development.

I'd rather see official support for v2 get off the ground sooner and unburden v1 with v2 and vice versa. Can we start with a minimal intersection module, go ahead with the mass code duplication, and move things to the intersection module over time?

I'm very tempted to volunteer to become the v2 PIC, but I'm not quite ready to take the plunge yet. On Feb 4, 2016 11:42 AM, "Chris Allen" [email protected] wrote:

Amazon's brand new hosted ElasticSearch is only 1.x and I would be surprised if they added 2.0 support in the next few quarters or even this year.

That's a strong argument for staying where we are in terms of official support.

To be clear, I completely understand the desire to have a PIC and think its crucial. Bloodhound has never claimed to be a complete API client, people add what they need when they come across it. I think the same should apply with major version numbers: stakeholders that really need 2.0 support should step up for maintainership. I just think we need to be in touch with what versions the community actually uses and try to support a reasonable subset of versions to be useful to most of them. I do think there will come a time where 1.x support can be sunset but I don't think Elastic is being realistic about when that should be ;)

All agreed.

I think the most dead-simple and maintainable way to handle the 1.x/2.x split is to just give up trying to come up with something slick: stick a V1 in all existing Bloodhound files, make a V2 duplicate of ALL files, and tell maintainers to have at it.

I think we can be smarter than that and have a mutually compatible intersection module. I'm open to this setup, but I prefer the status quo for now unless you are putting yourself forward as a maintainer for the V2 module.

If whomever volunteers to start the module split doesn't maintain the V2 module (which for our purposes is currently the red-headed stepchild), the module split will be axed if they can't keep up with the releases. Not that there are that many or that they are that frequent, but they still have to get done and I loathe pushing out code without passing tests. Technically I could say we support 2.0 and probably get away with it, but because not all of the tests pass, I haven't.

Current take-away for me:

Status quo stands. Lets try to document sharp corners for our 2.0 people.

I'd like to put off theorycrafting the module split until we have a PIC.

If it's all right with you both I consider this matter closed for now. We can reopen it when we either have a PIC volunteer (no longer requiring 2, I think Michael and I can handle 1.x), something changes, or you feel there's some unfinished business.

— Reply to this email directly or view it on GitHub https://github.com/bitemyapp/bloodhound/issues/101#issuecomment-180019123 .

[bitemyapp]

Though seriously, what is the benefit of a mutually compatible module?

Avoiding unnecessary duplication when the overlap is 95% anyway. Identical types will be equal to each other, types/functions specific to the module version will not be. Haskell doesn't have structural equality for types. Acts as a form of documentation for the differences between V1 and V2. I'm not gonna merge a copy-pasta job. The value is the knowledge, knowing what's V1 or V2 specific. That should be embodied by the code.

[MHova]

Then can we start from the other direction? Move all existing code into an intersection module and move specific things out to V1 and V2 over time? This avoids duplication and the structure and level of v1/v2 support wouldn't be any worse than what we have now.

[bitemyapp]

@MHova /cc @MichaelXavier I'm giving Xavier an opportunity to push back here. While that plan sounds good, this is the theorycrafting I said I wanted to avoid until we had a PIC.

The answer's no for now unless we have someone willing to take responsibility for it. Elasticsearch's API is too big and I work what is essentially two full time jobs on top of maintaining this and other things. Given the limitations involved, I am going to resist scope creep.

[MichaelXavier]

The intersecting modules thing sounds like its worth a shot. The best advice I can give is that if you're using bloodhound in anger for 2.0, keep an internal fork while you work out the ergonomics and kinks in significant additions. For pretty much any feature I've added to bloodhound I'd already had that branch going on my fork and had it running in production. I do see now that theorycrafting (gonna steal that word for my vocabulary BTW) may not be helpful because hard to use APIs are usually born out of good ideas to start. Better that some folks get their dogfooding in and take a first stab and tell us how it went.

[bermanjosh]

@MichaelXavier @bitemyapp @MHova

I just cut myself over to ghc 8, and I'm trying to now move to E.S. 2.4. So I want to take a stab at getting us real support. I think I have the bandwidth to push this forward over the next little while, but I'll need so guidance:

Based on what I could remember / grok from the above, I'm going to give a shot at the intersection module. How does this sound?

• New folder (Internal?) for the overlapping types and functions (which will start as a copy of what I have now).
• Setup v1 and v2 folders / modules importing the above.
• I'll start running test suites (do we have instructions about that?) and see where we need to specialz for v2. In those cases, I'll split the instance declarations / types into the version specific folder/modules.
• The new API will be to explicitly import the version you want. However, I'll keep & point the existing modules to their v1 counterparts for backwards compatibility? (Or are we calling this a breaking change?)

Anyone have any comments, specific things I should look out for, gotchas etc...?

[MichaelXavier]

@bitemyapp and I discussed this a bit today, but the only things I'll add:

• We've got version range helpers in the test suite that can be used to add 2.X specific tests.
• Reiterating my point from before, I think it seems most appropriate for you to maintain your fork with the split modules for 2.0 specific stuff and get some good production mileage on the fork before submitting a PR. Obviously if you think our feedback would help or if there are big decisions that may affect mergability, reach out. The case I'm trying to avoid is getting a MVP in as a PR, making a breaking version release and finding there's a much better way to do it that we hadn't considered carefully enough due to lack of time in production.

[bermanjosh]

@MichaelXavier @bitemyapp

Understood. So I'll just go off into a corner for a few days and do my thing on my fork. When I have something which I'm ready for my production environment, I'll reach out to you guys for a quick review, but I'm not going to to push on merging. I'll cut over my dependences to my fork, and you guys can decide if / when to push to merge it back in. Right?

[MichaelXavier]

@bermanjosh sounds good to me! Let us know how it goes in production. I think we should shoot for merging somewhat battle-tested code without waiting so long that the merge is going to be painful.

[bermanjosh]

What?!? Balance between perfect code and code that's functionally useable? I've never heard of such a crazy idea . :-)

[bitemyapp]

@bermanjosh speaking for myself, (don't know what Michael is cooking up), I have no changes planned at this time so you should be fairly clear to get this done and test it in prod before putting it up for review.

[bermanjosh]

@MichaelXavier @bitemyapp

As a 1st step, I'm working on getting the tests to pass, and it's raised a potential issue. A major breaking change is the unification of Filters and Querys. I think that practically speaking we could keep much of the current type structure, but:

• Not all of it. Specifically, I found the issue when dealing with the GeoBoundingBoxConstraint - it used to have a _cache field, but no longer. I can't separate just this into the V2.Types module, as that module imports the "original" Common.Types module. I need to split out a good chunk of the types into V1 and V1, basically all the way up to Search. I'm raising this because that could potentially be a lot of code duplication, but I don't really see an alternative.
• Another issue pushing me in this direction is that the current representation is semantically incorrect. A filter is now part of a query, and even if Elastic is providing a backwards-compatible API (which I'm not yet totally sure about). the V2.Types.Query should probably be updated to reflect this.

Thoughts?

[bermanjosh]

@bitemyapp Also, should we reopen this issue? (I keep having to dig for it)

[codedmart]

I am curious where this is going. I am just starting to get an elasticsearch instance running at work. It is v 2.4 and the first thing I just ran into is fuzzy_like_this has been removed and now use a fuzziness with the match or more_like_this queries. I am not feeling there is a consensus yet on how to split everything from what I have read. My knowledge of es is still low as this is my first time using it, but I would like to try and pitch in, especially for anything I run into I need.

[bermanjosh]

I hope to have a tests-passing fork by EOD tomorrow, so that can be useful as a baseline.

That being said, my gut says that most of my work on this attempt isnt going to work out. I'm ending up having to copy a significant amount of code to specialize for V1 / V2.

[codedmart]

Well I will just fork for now and play with what I need. Maybe after you have finished this stage others can take a look and provide some ideas and/or solutions.

[bermanjosh]

@MichaelXavier @bitemyapp @codedmart

Ok, please take a look at the following commit: https://github.com/bermanjosh/bloodhound/commit/a1801133c5a632fe5ec83b39f0d2dc171f9b8cb9 (ignore the state of the imports, ordering etc... - I pushed the above for the sake of this conversation)

Here's the situation: In order to deal with the new ES API, a lot, if not the majority, of the toJSON fromJSON instances are going to need to be changed (as some of the records will change to match). That means they're going to need to be defined separately for V1 and V2. Which means, a lot of code duplication, which is exactly what we were trying to avoid. (And this commit is just for one filter change; I'm sure there are many more).

At this point, I'm leaning towards pushing @MHova idea from Feb 4; Cutting over to ES-2. I'm open to ideas for different methods, approaches, etc... on the overlapping-approach, but I'm skeptical about the feasibility,

[bitemyapp]

@bermanjosh could you list the datatypes that need to change and provide a couple examples of before and after?

I'm not saying a pure cutover isn't the right answer, but I think you're taking a narrow view of what's possible.

I am inapt to charge headlong into any decisions, we need information recorded for posterity as well.

[codedmart]

So for the instance of my case yesterday. fuzzy_like_this has been removed (I am using ES 2.4) and the fuzziness option can be auto or a number(Double as is in the code now) and is accepted as part of match, multi_match, more_like_this, etc. I just added this for my sake to get me going:

data Fuzziness
  = Auto
  | Fuzziness Double
  deriving (Eq, Read, Show, Generic, Typeable)
instance ToJSON Fuzziness where
  toJSON Auto = "auto"
  toJSON (Fuzziness i) = toJSON i
instance FromJSON Fuzziness where
  parseJSON (Number i) = return $ Fuzziness (toRealFloat i)
  parseJSON _ = return Auto

data MultiMatchQuery =
  MultiMatchQuery { multiMatchQueryFields          :: [FieldName]
                  , multiMatchQueryString          :: QueryString
                  , multiMatchQueryOperator        :: BooleanOperator
                  , multiMatchQueryZeroTerms       :: ZeroTermsQuery
                  , multiMatchQueryFuzziness       :: Maybe Fuzziness
                  , multiMatchQueryTiebreaker      :: Maybe Tiebreaker
                  , multiMatchQueryType            :: Maybe MultiMatchQueryType
                  , multiMatchQueryCutoffFrequency :: Maybe CutoffFrequency
                  , multiMatchQueryAnalyzer        :: Maybe Analyzer
                  , multiMatchQueryMaxExpansions   :: Maybe MaxExpansions
                  , multiMatchQueryLenient         :: Maybe Lenient } deriving (Eq, Read, Show, Generic, Typeable)

mkMultiMatchQuery :: [FieldName] -> QueryString -> MultiMatchQuery
mkMultiMatchQuery matchFields query =
  MultiMatchQuery matchFields query
  Or ZeroTermsNone Nothing Nothing Nothing Nothing Nothing Nothing Nothing

So I see omitNulls is used in ToJSON instances so if this was added to bloodhound and we just documented the fuzziness(multiMatchQueryFuzziness) is a 2+ feature would that work? Not sure how many of these differences there are as I am just getting started.

[bermanjosh]

@bitemyapp Sorry about the drop, here. Had to deal with an emergency issue at work for a few days.

You can look at the exports lists of V1.Client and V1.Types for a sense of what I've had to split out so far: https://github.com/bermanjosh/bloodhound/commit/a1801133c5a632fe5ec83b39f0d2dc171f9b8cb9

As for what will need to be done, since ES 2.0 filters and queries are combined, all the filter types which have a _cache field are going to have to specialized; with the field for V1, without for V2.

That still leaves open the more fundamental question of: What is the semantically correct representation of the new filter / query relationship?