Search, Full-Text indexing

[win32nipuh]

Hi Antonello, what about Search implementation? Full-Text Search?

Thanks. Regards, Oleg.

[tsutomi]

Hi Oleg! thanks for the interest and the idea. Actually, the full-text search is something that definitively I want to implement: I must see how. My idea is to use Lucene to create indexes for string columns, but I must change a little the architecture of indexes at the moment seen in version 1.x. In fact, explicit indexes were never active in the current version and will be introduced in the new version, with specific types.

Lets stay tuned for the developments, although at the moment this has a lower priority (I'm still stuck with the query model and optimization).

[tsutomi]

Implementation of custom search indices will start next week: I'm thinking of using Lucene as first engine: the new design of table indices is quite dynamic and allow resolving at runtime.

[win32nipuh]

great! good luck! If I can help somehow please let me know.

[tsutomi]

Spasibo! Help is always welcome! If you decide to help, we could have a chat and you could put your hands on the code: DeveelDB 2.0 is way much simpler to read and write rather than the old version. In general, there's a lot to do: for instance, for supporting full-text search:

• The SQL parser must support the CREATE INDEX statement (including implementation-specific properties, such as 'store', 'analyze', etc.)
• Implementation of the set Statement/PreparedStatement for supporting the resolution of the statement in code
• Implementing a TableIndex for Lucene
• An implementation of ColumnIndex class for Lucene
• Support for creating indexes (checking for user's privileges, checking for columns not participating to other indices, etc.)
• The AlterTable command support (indices information are stored in the headers of a table and the creation of an index involves the alteration of a table structure)
• Tests for covering the case and assessing work

This feature has been on my desk since version 1, but the quality of code was very poor and not possible to extend: now everything is much different and even collaboration should be easier (for those interested in collaborating).

By the way, I'm working on the ALTER TABLE feature while we speak.

[tsutomi]

The system logic was changed to resolve dynamically (using the new Service Resolver) external indices types: by default DeveelDB provides only two ColumnIndex implementations, but if at the moment of the definition of the table, or when creating an index (the statement CREATE INDEX not implemented yet) any unknown index type (and meta) is specified, this is resolved against any external bound index type.

I've created a place-holder (at the moment) external Lucene.NET index (in a project named deveeldb-lucene): the implementation of this is still not done, cause I must do some researches on how to store and query such columns.

Additionally, since the modular architecture of the system, further system functions can be defined, to activate more advanced search and comparison features.