couchdb-lucene
couchdb-lucene copied to clipboard
rnewson
Search Timeout with BigCouch
Im using couchdb-lucene with couchdb with no issue. I do my queries on the jetty server through a nodejs server.
I want to use BigCouch but I can't make it work. I always get a timeout.
curl localhost:5985/local/data_96e130282817487a71d73b73fa1049f5/_design/js/all?q=hello
{"reason":"Search timed out.","code":500}
Logs:
bin:$ ./run
2013-07-15 16:28:26,203 INFO [Config] Index output goes to: /gitroot/couchdb- lucene/target/couchdb-lucene-0.9.0-SNAPSHOT/indexes
2013-07-15 16:28:26,221 INFO [Main] Accepting connections with [email protected]:5985
2013-07-15 16:28:29,573 INFO [data_96e130282817487a71d73b73fa1049f5] Indexing from update_seq start
2013-07-15 16:28:39,607 WARN [log] /local/data_96e130282817487a71d73b73fa1049f5/_design/js/all: java.io.IOException: Search timed out.
2013-07-15 16:28:44,641 INFO [data_96e130282817487a71d73b73fa1049f5] View[name=_design/js/all, digest=utxtbbct89n07w77054dx35b] now at update_seq [9,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5EZrsQErsQIrCRJAUgm2YNVMeE0KMkBpCoeSVVxTn45XJW5MURVAkhVPVgVA9wsQ5gqU4hZeSxAkqEBSAEVzkdWie42iMoFEJX7kd2H3cwDEJX3CZv5AKIS5M4sAGSoVtk"]
2013-07-15 16:28:47,834 WARN [log] /local/data_96e130282817487a71d73b73fa1049f5/_design/js/all: java.io.IOException: Search timed out.
2013-07-15 16:29:02,870 WARN [log] /local/data_96e130282817487a71d73b73fa1049f5/_design/js/all: java.io.IOException: Search timed out.
How can I find out what is wrong ?
I use the latest (master) couchdb-lucene with bigcouch 0.4. Timeout is 10000 and I have less then 10 docs in the database.
in fact, the log I put was from a 0.9 couchdb-lucene. But its the same thing with 0.10...
[root@server2 bin]# ./run
2013-07-15 12:40:20,184 INFO [Config] Index output goes to: /opt/couchdb-lucene/target/couchdb-lucene-0.10.0-SNAPSHOT/indexes
2013-07-15 12:40:20,205 INFO [Main] Accepting connections with [email protected]:5985
2013-07-15 12:40:47,274 INFO [data_96e130282817487a71d73b73fa1049f5] Indexing from update_seq start
2013-07-15 12:40:57,308 WARN [log] /local/data_96e130282817487a71d73b73fa1049f5/_design/js/all: java.io.IOException: Search timed out.
2013-07-15 12:41:02,358 INFO [data_96e130282817487a71d73b73fa1049f5] View[name=_design/js/all, digest=utxtbbct89n07w77054dx35b] now at update_seq [10,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5EFrsQIpsTUCKwkSQFIJtmDVTHhNCjJAaQqHklVcU5-OVyVuTFEVQJIVT1YFQPcLEO4jRCz8liAJEMDkAIqnI-s0gRuqxGSygUQlfvx2QxReQCi8j5hMx9AVILcmQUAa11W9A"]
I don't if its related... I have a bunch of these:
[Mon, 15 Jul 2013 17:25:29 GMT] [info] [<0.5766.0>] [--------] Shutting down group server <<"_design/native">>, db <0.274.0> closing w/ reason
killed
[Mon, 15 Jul 2013 17:25:29 GMT] [info] [<0.5771.0>] [--------] Shutting down group server <<"_design/native">>, db <0.275.0> closing w/ reason
killed
[Mon, 15 Jul 2013 17:25:29 GMT] [info] [<0.5776.0>] [--------] Shutting down group server <<"_design/native">>, db <0.276.0> closing w/ reason
killed
Grab the url that couchdb-lucene is trying from the log and try it independently on the command line?
I set the log level of bigcouch to debug...
I can see couchdb-lucene queries but no error ??
doing a cURL
curl 'admin:password@localhost:5984/data_96e130282817487a71d73b73fa11a82a/_changes?limit=0&descending=true' looks good.
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [b818c4ed] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/ 200
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [b818c4ed] 127.0.0.1 localhost:5984 GET /data_96e130282817487a71d73b73fa11a82a/ 200 ok 24
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [a63be149] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/_local%2Flucene 200
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [a63be149] 127.0.0.1 localhost:5984 GET /data_96e130282817487a71d73b73fa11a82a/_local%2Flucene 200 ok 3
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [da629301] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/ 200
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [da629301] 127.0.0.1 localhost:5984 GET /data_96e130282817487a71d73b73fa11a82a/ 200 ok 9
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [21082f9d] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/_all_docs?startkey=%22_design%22&endkey=%22_design0%22&include_docs=true 200
[Mon, 15 Jul 2013 17:50:40 GMT] [info] [<0.2122.0>] [21082f9d] 127.0.0.1 localhost:5984 GET /data_96e130282817487a71d73b73fa11a82a/_all_docs?startkey=%22_design%22&endkey=%22_design0%22&include_docs=true 200 ok 20
[Mon, 15 Jul 2013 17:50:41 GMT] [info] [<0.2122.0>] [fcc27315] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/_changes?limit=0&descending=true 200
[Mon, 15 Jul 2013 17:50:41 GMT] [info] [<0.2122.0>] [fcc27315] 127.0.0.1 localhost:5984 GET /data_96e130282817487a71d73b73fa11a82a/_changes?limit=0&descending=true 200 ok 18
[Mon, 15 Jul 2013 17:50:56 GMT] [info] [<0.2123.0>] [30795c51] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/_changes?feed=continuous&heartbeat=15000&include_docs=true&since=%5B21%2C%22g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5EFrsQEpsQMoiRJAUgm2YNVscFVGcNVGUJUOYBUxYNVMeE2KwGkqh5NFbpZeSxAkqEBSAEVzs9KZMVpHkTlAojK_VmJDATMPABReR_ZdkN4kCCrfABRCXQnQxYAWzRWzQ%22%5D 200
[Mon, 15 Jul 2013 17:51:17 GMT] [info] [<0.2122.0>] [492a69e7] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/_changes?limit=0&descending=true 200
[Mon, 15 Jul 2013 17:51:17 GMT] [info] [<0.2122.0>] [492a69e7] 127.0.0.1 localhost:5984 GET /data_96e130282817487a71d73b73fa11a82a/_changes?limit=0&descending=true 200 ok 14
[Mon, 15 Jul 2013 17:51:28 GMT] [info] [<0.2122.0>] [ae914982] undefined - - 'GET' /data_96e130282817487a71d73b73fa11a82a/_changes?limit=0&descending=true 200
[Mon, 15 Jul 2013 17:51:28 GMT] [info] [<0.2122.0>] [ae914982] 127.0.0.1 localhost:5984 GET /data_96e130282817487a71d73b73fa11a82a/_changes?limit=0&descending=true 200 ok 11
I did many more tests and its driving me crazy...
if I don't set the security on the database, it works fine. If I secure it, it timeout. But if i remove the security, then it still doesn't work.
curl 10.0.1.52:5985/local/data_96e130282817487a71d73b73fa1a7e43/_design/js/all?q=bar* {"limit":25,"etag":"24d65ff19390","fetch_duration":0,"q":"default:bar*","search_duration":10,"total_rows":1,"skip":0,"rows":[{"id":"test","score":1}]}bin:$
Then I change the security
curl 10.0.1.52:5985/local/data_96e130282817487a71d73b73fa1a7e43/_design/js/all?q=bar* {"reason":"Search timed out.","code":500}
broken...
Ok, then put it back to no security but I have the same time error...
The url contains the admin:password@localhost:5984 couchdb-lucene.ini... and the bigcouch log do not have 401 requests from couchdb-lucene after the db is secure.
The other issue (maybe related ?) is if a replicate the database (futon replicator), for some target database name, it can't copy the two design docs:
{"session_id":"84bf95b8d4690f7c2dfa3be91783ed95","start_time":"Mon, 15 Jul 2013 21:15:04 GMT","end_time":"Mon, 15 Jul 2013 21:15:04 GMT","start_last_seq":0,"end_last_seq":[2,"g1AAAAFDeJzLYWBg4MhgTmHgL87JLzd0MDTQM9Az1DM3zAFKMCUyJMn___8_K5EBrsQYrsQYrCRJAUgm2YNVMcJVGcFVGUFUOYBUxaOpQrcuKQGkqp6AjXksQJKhAUgBFc5HVoluK0TlAojK_cgq0W2GqDwAUXmfsO0PICpB7swCAM7JV3g"],"recorded_seq":[2,"g1AAAAFDeJzLYWBg4MhgTmHgL87JLzd0MDTQM9Az1DM3zAFKMCUyJMn___8_K5EBrsQYrsQYrCRJAUgm2YNVMcJVGcFVGUFUOYBUxaOpQrcuKQGkqp6AjXksQJKhAUgBFc5HVoluK0TlAojK_cgq0W2GqDwAUXmfsO0PICpB7swCAM7JV3g"],"missing_checked":0,"missing_found":2,"docs_read":2,"docs_written":0,"doc_write_failures":2}
For target database "foo", it can't copy. But for "foo_foo", its alright!
{"session_id":"c3ec8583dbf531138dc0c49e38bd3b76","start_time":"Mon, 15 Jul 2013 21:10:27 GMT","end_time":"Mon, 15 Jul 2013 21:10:27 GMT","start_last_seq":0,"end_last_seq":[2,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5EBrKQ4J78crsTcGKwkSQFIJtmDVTHiNCjJAaQqHk2VIUyVKVRVAkhVPZKN2MzKYwGSDA1ACqhwPrJKdPMgKhdAVO4nbOYBiMr7hM18AFEJcmcWAGIXVsk"],"recorded_seq":[2,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5EBrKQ4J78crsTcGKwkSQFIJtmDVTHiNCjJAaQqHk2VIUyVKVRVAkhVPZKN2MzKYwGSDA1ACqhwPrJKdPMgKhdAVO4nbOYBiMr7hM18AFEJcmcWAGIXVsk"],"missing_checked":0,"missing_found":2,"docs_read":2,"docs_written":2,"doc_write_failures":0}
Regular queries on the bigcouch cluster works as expected... It can GET, PUT, POST, docs without issues.
Any ideas to fix this would be very much welcome.
Thanks.
Getting closer. It seems couchdb-lucene does send the _changes call to index new changes but does not get a response from bigcouch 0.4. The timeout then fires (as expected).
In my test, couchdb-lucene is expecting a higher last_seq than the database sends in the changes feed. Quite strange.
Could you show the output of the following two commands to confirm this?
- "curl host:port/databasename" and report the value of "update_seq"
- "curl host:port/databasename/_changes" and report the value of "last_seq" right at the end.
This seems like a bug in bigcouch's _changes feed when descending=true, which is how bigcouch figures out what update sequence to wait for. Since it never appears, the search eventually times out.
Here it is: 1. curl 10.0.1.52:5984/data_b8008162df528b2008b72781dc007b94/ {"db_name":"data_b8008162df528b2008b72781dc007b94","update_seq":[13,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5EJrsQIrsQIrCRJAUgm2YNVMeM0KMkBpCqekFkJIFX1YFWMOM3KYwGSDA1ACqhwPrJKdPMgKhdAVO4nbOYBiMr7-NwIUfkAohLkziwARWZWsw"],"purge_seq":0,"other":{"data_size":11873},"doc_del_count":0,"doc_count":5,"disk_size":139940,"disk_format_version":5,"compact_running":false,"instance_start_time":"0"}
curl 10.0.1.52:5984/data_b8008162df528b2008b72781dc007b94/_changes ... "last_seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFxMDTQM9Az1DMzygFKMCUyJMn___8_K5ERrsQIpsQUoiRJAUgm2YNVMeE0KMkBpCqekFkJIFX1YFUMYFXFOfnlxjBV5sZgVXksQJKhAUgBFc6HqcRmHkTlAojK_cgq0d0HUXkAovI-PjdCVD6AqAS5MwsAXwhW1w"]
In the couchdb-lucene console, it output:
2013-07-16 03:29:14,023 INFO [data_b8008162df528b2008b72781dc007b94] Indexing from update_seq [5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5ERrKQ4J7_cGKbE3BisJEkBSCbZg1Ux4TQoyQGkKh7JLJAqI7gqI4iqBJCqerAqBpw25rEASYYGIAVUOB-mEpt5EJULICr3I6tEdx9E5QGIyvv43AhR-QCiEuTOLABwOlbj"] 2013-07-16 03:29:24,045 WARN [log] /local/data_b8008162df528b2008b72781dc007b94/_design/js/all: java.io.IOException: Search timed out.
No security set:
bin:$ curl 10.0.1.52:5984/data_b8008162df528b2008b72781dc02a4bf/ {"db_name":"data_b8008162df528b2008b72781dc02a4bf","update_seq":[3,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5EBrsQIrsQIrCRJAUgm2YNVMeI0KMkBpCoeTRWGWQkgVfVoNqKblccCJBkagBRQ4Xx85kFULoCo3E_YzAMQlffx-Rei8gFEJcidWQA_GFap"],"purge_seq":0,"other":{"data_size":11134},"doc_del_count":0,"doc_count":3,"disk_size":62114,"disk_format_version":5,"compact_running":false,"instance_start_time":"0"}
bin:$ curl 10.0.1.52:5984/data_b8008162df528b2008b72781dc02a4bf/_changes {"results":[ {"seq":[1,"g1AAAAHjeJzLYWBg4MlgTmHgL87JLzd2MDTQM9Az1DM3zgFKMCUyJMn___8_K5EBrCQtsbjECKbE1AisJEkBSCbZg1UxwlUZwlUZQlQ5gFTFo5llAlNlZoRDFchRcLPM8ZmF4a4EkKp6NFXo7spjAZIMDUAKqHA-PrdhVwlyH9xmc3wqsbkRonIBROV-ZDPRIwKi8gBE5X3CZj6AqAT5PQsAXYKBgw"],"id":"_design/native","changes":[{"rev":"1-14e8901b3f8d6c9179a58f684542cb15"}]}, {"seq":[2,"g1AAAAGPeJzLYWBg4MpgTmHgL87JLzd2MDTQM9Az1DM3zgFKMCUyJMn___8_K5EBrCQtsbjECKbE1AisJEkBSCbZg1UxwlUZwlUZQlQ5gFTFo5llAlNlZoRDFchRcLPM8ZmF4a4EkKp6NFXo7spjAZIMDUAKqHA-sg_QzYOoXABRuR_ZfeiBBlF5AKLyPj43QlQ-gKgEuTMLACq5a5I"],"id":"384f7c8141b84e7b37ba8fdbe100183b","changes":[{"rev":"1-4c6114c65e295552ab1019e2b046b10e"}]}, {"seq":[3,"g1AAAAFDeJzLYWBg4MhgTmHgL87JLzd2MDTQM9Az1DM3zgFKMCUyJMn___8_K5EBrCQtsbjECKbE1AisJEkBSCbZg1UxwlUZwlUZQlQ5gFTFo6nCMCsBpKoezUZ0s_JYgCRDA5ACKpyPzzyIygUQlfthZmLzKETlAYjK-_j8C1H5AKIS5M4sAGsJVuM"],"id":"_design/js","changes":[{"rev":"1-0c7e5e1b431b9624342bba75c24415f2"}]} ], "last_seq":[3,"g1AAAAFDeJzLYWBg4MhgTmHgL87JLzd2MDTQM9Az1DM3zgFKMCUyJMn___8_K5EBrCQtsbjECKbE1AisJEkBSCbZg1UxwlUZwlUZQlQ5gFTFo6nCMCsBpKoezUZ0s_JYgCRDA5ACKpyPzzyIygUQlfthZmLzKETlAYjK-_j8C1H5AKIS5M4sAGsJVuM"]}
=> Both last_seq and are update_seq are the same.
I set the security.
curl 10.0.1.52:5984/data_b8008162df528b2008b72781dc02a4bf/ {"db_name":"data_b8008162df528b2008b72781dc02a4bf","update_seq":[11,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5ERrsQIrsQIrCRJAUgm2YNVMeE0KMkBpCoeTRWGWQkgVfVoNqKblccCJBkagBRQ4Xx85kFULoCo3E_YzAMQlffx-Rei8gFEJcidWQBDqFax"],"purge_seq":0,"other":{"data_size":11134},"doc_del_count":0,"doc_count":3,"disk_size":94859,"disk_format_version":5,"compact_running":false,"instance_start_time":"0"} bin:$ curl 10.0.1.52:5984/data_b8008162df528b2008b72781dc02a4bf/_changes {"results":[ {"seq":[1,"g1AAAAIveJzLYWBg4MtgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5EBrsQIrsQIrCRJAUgm2aOpMoapMjPEoao4J78cbpY5PrPQHZXkAFIVj6bKBG6jEQ5VIBvhZpnjMwvDjwkgVfUE3JXHAiQZGoAUUOH8rERGnOZBVC6AqNxP2MwDEJX38bkRovIBRCXeuMCuElt8YKjMAgCM1Zb3"],"id":"384f7c8141b84e7b37ba8fdbe100183b","changes":[{"rev":"1-4c6114c65e295552ab1019e2b046b10e"}]}, {"seq":[2,"g1AAAAHjeJzLYWBg4MlgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5EBrsQIrsQIrCRJAUgm2aOpMoapMjPEoao4J78cbpY5PrPQHZXkAFIVD1bFiNtdCSBV9QTMymMBkgwNQAqocD4-8yAqF0BU7ids5gGIyvv4wg6i8gFEJd7ww64SWxhiqMwCAAx9gkg"],"id":"_design/js","changes":[{"rev":"1-0c7e5e1b431b9624342bba75c24415f2"}]}, {"seq":[3,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5EBrsQYpsQMoiRJAUgm2YNVMeI0KMkBpCoeTZURXJURRFUCSFU9mo3oZuWxAEmGBiAFVDgfn3kQlQsgKvcTNvMAROV9ZJXYzXwAUQlyZxYAQCtWqg"],"id":"_design/native","changes":[{"rev":"1-14e8901b3f8d6c9179a58f684542cb15"}]} ], "last_seq":[3,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF0MDTQM9Az1DM1zAFKMCUyJMn___8_K5EBrsQYpsQMoiRJAUgm2YNVMeI0KMkBpCoeTZURXJURRFUCSFU9mo3oZuWxAEmGBiAFVDgfn3kQlQsgKvcTNvMAROV9ZJXYzXwAUQlyZxYAQCtWqg"]}
=> last_seq stick to [3, ... but update_seq has bump to [11,...
Is that Ok ?
I don't think security is the problem here, just that the index is now caught up but the mechanism that couchdb-lucene uses to confirm them doesn't work with bigcouch. The issue, at base, is that bigcouch reports a higher update sequence for changes?descending=true than it reports at the end of changes?descending=false.
humm, here a database with no security.
bin:$ curl admin:[email protected]:5984/data_b8008162df528b2008b72781dc0e77f3/_changes?descending=false {"results":[ {"seq":[1,"g1AAAAMdeJzLYWBgEMlgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5EBrsQYpsTMEJuS4pz8crgp5jhNMYGbAlGSpAAkk-zBqhhxOifJAaQqnoCLMFWBHAVXZW6MxyxDuI1QsxJAquoJuR5DFbZgwG4Wuh_zWIAkQwOQAiqcj89tEJULICr343MfdpXYQgW3mdjdeQCi8j6--MCuElsI4TYTu98fQFTijRvsKkG2w800x2FmFgCPKNkS"],"id":"_design/native","changes":[{"rev":"1-14e8901b3f8d6c9179a58f684542cb15"}]}, {"seq":[2,"g1AAAAHdeJzLYWBg4MlgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5EBrsQYpsTMEJuS4pz8crgp5jhNMYGbAlGSpAAkk-zBqhjhBsHtMjeGqHIAqYpHUoXVrASQqno0G9G9lscCJBkagBRQ4Xx8boOoXABRuZ-wmQcgKu_jCzXsKrEFHm4zsbvzAUQlyO9ZAOFdgXU"],"id":"design/js","changes":[{"rev":"1-0c7e5e1b431b9624342bba75c24415f2"}]}, {"seq":[3,"g1AAAAGXeJzLYWBg4MpgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQEpsQMoiRJAUgm2SOpKs7JLzeGqTI3hqhyAKmKJ2RWAkhVPVgVA05H5bEASYYGIAVUOB9ZJbp5EJULICr3EzbzAETlfWSVcJ-YGeJRCfIz3ExzQmZid-cDiEqQ37MAT9ttyA"],"id":"design/-t~person-k~created_at-r~_count","changes":[{"rev":"1-2285b0fcebbd568d4a1d08e21e109563"}]}, {"seq":[4,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQEpsQMoiRJAUgm2SOpKs7JLzeGqTI3hqhyAKmKJ2RWAkhVPVgVA05H5bEASYYGIAVUOB9ZJbp5EJULICr3EzbzAETlfXxuhKh8AFEJcmcWAGYWVtk"],"id":"b8008162df528b2008b72781dc0e9b94","changes":[{"rev":"1-023cf0b23f13b539fc956ce004f06cd8"}]}, {"seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQEpsQMoiRJAUgm2YNVMYFVFefklxvDVJkbQ1Q5gFTFEzIrAaSqHqyKAaej8liAJEMDkAIqnI-sEt08iMoFEJX7CZt5AKLyPj43QlQ-gKgEuTMLAGcMVto"],"id":"b8008162df528b2008b72781dc0eaa79","changes":[{"rev":"1-3c8a04546ab48a2ae68d4d226646df8a"}]} ], "last_seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQEpsQMoiRJAUgm2YNVMYFVFefklxvDVJkbQ1Q5gFTFEzIrAaSqHqyKAaej8liAJEMDkAIqnI-sEt08iMoFEJX7CZt5AKLyPj43QlQ-gKgEuTMLAGcMVto"]}
descending bin:$ curl admin:[email protected]:5984/data_b8008162df528b2008b72781dc0e77f3/_changes?descending=true {"results":[ {"seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2YNVMeE0KMkBpCqekFkJIFX1YFUMOM3KYwGSDA1ACqhwPrJKdPMgKhdAVO4nbOYBiMr7-NwIUfkAohLkziwAQhBWqw"],"id":"design/js","changes":[{"rev":"1-0c7e5e1b431b9624342bba75c24415f2"}]}, {"seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2YNVMeE0KMkBpCqekFkJIFX1YFUMOM3KYwGSDA1ACqhwPrJKdPMgKhdAVO4nbOYBiMr7-NwIUfkAohLkziwAQhBWqw"],"id":"design/-t~person-k~created_at-r~_count","changes":[{"rev":"1-2285b0fcebbd568d4a1d08e21e109563"}]}, {"seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2YNVMeE0KMkBpCqekFkJIFX1YFUMOM3KYwGSDA1ACqhwPrJKdPMgKhdAVO4nbOYBiMr7-NwIUfkAohLkziwAQhBWqw"],"id":"b8008162df528b2008b72781dc0eaa79","changes":[{"rev":"1-3c8a04546ab48a2ae68d4d226646df8a"}]}, {"seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2YNVMeE0KMkBpCqekFkJIFX1YFUMOM3KYwGSDA1ACqhwPrJKdPMgKhdAVO4nbOYBiMr7-NwIUfkAohLkziwAQhBWqw"],"id":"b8008162df528b2008b72781dc0e9b94","changes":[{"rev":"1-023cf0b23f13b539fc956ce004f06cd8"}]}, {"seq":[4,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2aOpQjcoyQGkKp6QWQkgVfVgVQw4zcpjAZIMDUAKqHA-skp08yAqF0BU7ids5gGIyvv43AhR-QCiEuTOLABBGlaq"],"id":"_design/native","changes":[{"rev":"1-14e8901b3f8d6c9179a58f684542cb15"}]} ], "last_seq":[4,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2aOpQjcoyQGkKp6QWQkgVfVgVQw4zcpjAZIMDUAKqHA-skp08yAqF0BU7ids5gGIyvv43AhR-QCiEuTOLABBGlaq"]}
lucene query bin:$ curl 10.0.1.52:5985/local/data_b8008162df528b2008b72781dc0e77f3/_design/js/all?q=per* {"limit":25,"etag":"4ecca8618ddb","fetch_duration":0,"q":"default:per*","search_duration":0,"total_rows":2,"skip":0,"rows":[{"id":"b8008162df528b2008b72781dc0e9b94","score":1},{"id":"b8008162df528b2008b72781dc0eaa79","score":1}]}bin:$
I can Add more document and couchdb-lucene still works fine.
But when I modify the security:
bin:$ curl admin:[email protected]:5984/data_b8008162df528b2008b72781dc0e77f3/_changes?descending=false {"results":[ {"seq":[1,"g1AAAAMdeJzLYWBgEMlgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5EBrsQYpsTMEJuS4pz8crgp5jhNMYRbBDElSQFIJtmDVTHidE6SA0hVPAEXYaoCOQquytwYj1kY7koAqapHU2UCt9EIhypswYDdLHQ_5rEASYYGIAVUOB-f2yAqF0BU7sfnPuwqsYUKbjOxu_MAROV9fPGBXSW2EMJtJna_P4CoxBs32FWCbIebaY7DzCwAgf3ZDQ"],"id":"_design/native","changes":[{"rev":"1-14e8901b3f8d6c9179a58f684542cb15"}]}, {"seq":[2,"g1AAAAHjeJzLYWBg4MlgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5ERrsQQpsQUoiRJAUgm2aOpMoKrMoKocgCpigerYsBpHaaq4pz8crgqc2M8ZmG4KwGkqh5NFbq78liAJEMDkAIqnI_PPIjKBRCV-wmbeQCi8j4-2JXCfIz3ExzQmZid-cDiEqQ37MAo22CSw"],"id":"design/-t~person-k~created_at-r~_count","changes":[{"rev":"1-2285b0fcebbd568d4a1d08e21e109563"}]}, {"seq":[3,"g1AAAAGXeJzLYWBg4MpgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5ERrsQQpsQUoiRJAUgm2aOpQjcoyQGkKp6QWQkgVfVgVQxwVUZwVUZgVXksQJKhAUgBFc5HVoluHkTlAojK_YTNPABReR9ZJbpPsKsszskvh5tpTshM7O58AFEJ8nsWAB_ibZs"],"id":"_design/js","changes":[{"rev":"1-0c7e5e1b431b9624342bba75c24415f2"}]}, {"seq":[4,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5ERrsQQpsQUoiRJAUgm2aOpQjcoyQGkKp6QWQkgVfVgVQxwVUZwVUZgVXksQJKhAUgBFc5HVoluHkTlAojK_cgq0d0HUXkAovI-PjdCVD6AqAS5MwsAQ4pWrQ"],"id":"b8008162df528b2008b72781dc0e9b94","changes":[{"rev":"1-023cf0b23f13b539fc956ce004f06cd8"}]}, {"seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5ERrsQQpsQUoiRJAUgm2YNVMeE0KMkBpCqekFkJIFX1YFUMcFVGcFVGYFV5LECSoQFIARXOR1aJbh5E5QKIyv3IKtHdB1F5AKLyPj43QlQ-gKgEuTMLAESAVq4"],"id":"b8008162df528b2008b72781dc0eaa79","changes":[{"rev":"1-3c8a04546ab48a2ae68d4d226646df8a"}]} ], "last_seq":[5,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjF2MDTQM9Az1DMzzAFKMCUyJMn___8_K5ERrsQQpsQUoiRJAUgm2YNVMeE0KMkBpCqekFkJIFX1YFUMcFVGcFVGYFV5LECSoQFIARXOR1aJbh5E5QKIyv3IKtHdB1F5AKLyPj43QlQ-gKgEuTMLAESAVq4"]}
descending bin:$ curl admin:[email protected]:5984/data_b8008162df528b2008b72781dc0e77f3/_changes?descending=true {"results":[ {"seq":[12,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5EJrsQQrsQQrCRJAUgm2aOpQjcoyQGkKp6QWQkgVfVgVYw4zcpjAZIMDUAKqHA-skp08yAqF0BU7ids5gGIyvv43AhR-QCiEuTOLABFqlay"],"id":"b8008162df528b2008b72781dc0eaa79","changes":[{"rev":"1-3c8a04546ab48a2ae68d4d226646df8a"}]}, {"seq":[11,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5EJrsQQrsQQrCRJAUgm2aOpQjcoyQGkKh6sihG3WQkgVfVoqtDNymMBkgwNQAqocD4-8yAqF0BU7ids5gGIyvv4_AtR-QCiEuTOLABE2lax"],"id":"design/js","changes":[{"rev":"1-0c7e5e1b431b9624342bba75c24415f2"}]}, {"seq":[10,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2YNVMeE0KMkBpCqekFkJIFX1aKrQzcpjAZIMDUAKqHA-PvMgKhdAVO4nbOYBiMr7yD7BbuYDiEqQO7MAQ75WsA"],"id":"design/-t~person-k~created_at-r~_count","changes":[{"rev":"1-2285b0fcebbd568d4a1d08e21e109563"}]}, {"seq":[9,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2YNVMeE0KMkBpCqekFkJIFX1aKrQzcpjAZIMDUAKqHA-PvMgKhdAVO4nbOYBiMr7hM18AFEJcmcWAEOSVq8"],"id":"b8008162df528b2008b72781dc0e9b94","changes":[{"rev":"1-023cf0b23f13b539fc956ce004f06cd8"}]}, {"seq":[8,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2aOpQjcoyQGkKp6QWQkgVfUEzMpjAZIMDUAKqHA-PvMgKhdAVO4nbOYBiMr7hM18AFEJcmcWAEKcVq4"],"id":"_design/native","changes":[{"rev":"1-14e8901b3f8d6c9179a58f684542cb15"}]} ], "last_seq":[8,"g1AAAAFDeJzLYWBg4MhgTmHgT0ssLjFyMDTQM9Az1DM1ygFKMCUyJMn___8_K5ERrsQQrsQQrCRJAUgm2aOpQjcoyQGkKp6QWQkgVfUEzMpjAZIMDUAKqHA-PvMgKhdAVO4nbOYBiMr7hM18AFEJcmcWAEKcVq4"]}
lucene query bin:$ curl 10.0.1.52:5985/local/data_b8008162df528b2008b72781dc0e77f3/_design/js/all?q=per* {"reason":"Search timed out.","code":500} bin:$
The security change affect the _changes stream in BigCouch. Regular Couchdb doesn't do that.
Do you experience the same thing ? Maybe my cluster is wrong somehow...
Changing security settings bumps the update_seq of the database but doesn't add any rows to _changes (since no document changed).
Could it be the root of the problem ? I mean, couchdb-lucene with BigCouch works fine before I set the security...
Yes, I think that's the root cause. If your last database update is to _security rather than a document, you'll get stuck like this. I think that's a bug in the last_seq that BigCouch emits.
Unfortunately, even if i do a lot changes after I changed the security, it doesn't fix couchdb-lucene. If I revert the security settings and restart couchdb-lucene, its still stuck. The only way to make it works is to never touch the security. Its a big deal for a couchapp.
Do you think its possible to workaround the issue in your code ? I don't think there will anymore BigCouch release before the couchdb 2.0 merge.
well, I also code bigcouch and couchdb, so I'm sure I can figure something out. Not a lot of time free right now, though.
When a couchdb database's _security is updated, the update_seq of the db is updated. A BigCouch database consists of multiple couchdb databases ("q") each of which will receive the _security update. The sequence value is thus counting the _security bump multiple times, and I think that's what's confusing things. Not sure what can be done about it yet.
So here's the plan. I'll change couchdb-lucene. It's clearly relying on behavior that it shouldn't. In order to ensure an index is 'caught up' when you query it, I'll have to make an http request to _changes with the since value, process every item, then run the search. I can't just rely on knowing when I pass an epoch in the already running _changes?feed=continuous request.
If I understand well, each search query to the index will perform a request to _changes with since value to the last sequence it perform an update. No more feed=continuous request ?
That's exactly how I process my queries for my spatial cache. I use a PostgreSQL database to index the geomtries and front it by a nodejs sever that perform the _change call, update the database with the new changes, store the last sequence in a special PG table and generate the proper SQL. It works well with both Couchdb and Bigcouch so far.
Ah, no, not quite that. It'll be both. When you first query an index we'll do /_changes?since=
That or figure out how to reliably know when we've gone past "current". Using the first seq from _changes?descending=true isn't it (for bigcouch).
In a situation where there are thousands of databases (one per users), would couchdb-lucene keep one feed=continuous connection for each db that have been query at least once since couchdb-lucene run ? If yes, is there a configuration to disable the continuous feed ?
Currently it would and there's no configuration option to turn that off. But there should be. On 22 Jul 2013 01:30, "Jean-Felix" [email protected] wrote:
In a situation where there are thousands of databases (one per users), would couchdb-lucene keep one feed=continuous connection for each db that have been query at least once since couchdb-lucene run ? If yes, is there a configuration to disable the continuous feed ?
— Reply to this email directly or view it on GitHubhttps://github.com/rnewson/couchdb-lucene/issues/176#issuecomment-21320014 .
🤔 I'm facing pretty much the problem here #264 ... any updates on that? I know, 5 years... 🤣
sorry, nope. I might have time to review a pull request but not to write it myself any time soon.