macnibblet
Non-eager updates
Because of the nature of ElasticSearch, index updates should not be real-time with main-db updates for most cases, because that puts a write-load on the index database.
There are usually several ways to handle this (or combination of those):
- periodic updates
- bulk (buffer with size) updates
- stacked (timed buffer) updates
UC1: A wiki page with hundreds of write ops / minute on ORM entities. The index (used for searching) does not have to be updated after each small change, but each 5 or 10 minutes.
UC2: User database - permission updates need to be updated on the spot, however user information (i.e. name, address, description, profile) can be synced every few minutes or each X flushes.
@Thinkscape bulk operations shouldn't probably go through PHP anyways in my opinion... hydration would be too slow for bulk operations.
As for the writes, we could just push them to a queue instead of directly asking ES.
That would probably be simpler
For bulk operations: The proposed solution would be to implement a amqp and let the the user decide what ever messaging system he wants.
We should also add a property prioritisation with three different levels
1, IGNORE - not related to ES skip bulk 2, BULK (default) - should be placed in the amqp if it setup else sync with es 3, IMMIDIATE - sync directly with ES
As discussed on IRC, I think a simple "write" backend adapter would solve the problem and let anyone plug in any kind of logic.
It would also ease operations for mocking/testing.
That allows for various implementations:
- writing to
/dev/null(seriously, only for testing :-) - writing directly to ES as soon as data is flushed to the DB
- collecting data from a flush, package it and send it bulk to ES
- writing to a different adapter (a message queue or whatever) for later pickup