elephant-bird
elephant-bird copied to clipboard
twitter
lucene-queryparser compatibility?
In com.twitter.elephantbird.mapreduce.output.LuceneIndexOutputFormat.createIndexWriter, there is a call to the API org.apache.lucene.index.LogByteSizeMergePolicy.setUseCompoundFile, which was removed after version 4.0.0 of org.apache.lucene:lucene-queryparser. This prevents me from using any features of later versions of the query parser in my program (specifically, 4.7.2) if I want to continue to use elephant-bird-pig-lucene.
What are your plans for upgrading this dependency? Currently, the org.apache.lucene components are at version 6.0.1 (35 releases beyond 4.0.0, released in October 2012).
Hi Immo, Development on this sub package is mostly suspended (nothing to do with the code -- the internal Twitter need for which this was built is now served by a completely different system). We can look at pull requests.
Yeah sorry, this is pretty old and we haven't made changes to it in a long time. We may want to consider removing it if it's gotten so stale as to be obsolete. IIRC there are other hadoop-lucene integrations out there, probably with better support. I remember this one: http://www.cloudera.com/documentation/archive/search/1-3-0/Cloudera-Search-User-Guide/csug_introducing.html but I don't know the current state of that either.
Many thanks - I suggest closing this issue and leaving things exactly as they are. It is never a great idea to just withdraw a library that other people have built into their applications. I had enough trouble with the gephi library, whose original repository was closed down and the replacement only contained later versions that were not backwards compatible.
We won't ever un-publish the existing maven artifacts. But if we really do feel that this abandoned then I'd be in favor of removing it from future versions of elephant bird -- that way nobody will waste their time trying to use it. On the other hand, if people are using it and want to improve on it, PRs are always welcome.