Add newlines between docs in JSON response to match CouchDB?

[dahjelle]

I just learned that CouchDB separates docs in its response with newlines, which means that one can stream the JSON response and start parsing each line of JSON without waiting for the entire response to be sent or using a streaming JSON parser. (See pull-couch for an implementation of this.)

However, it appears that pouchdb-server is not doing this.

I have no idea if this is an "official" feature of CouchDB or if it has even already been removed in CouchDB. (I've tested on 1.6.1.)

Interestingly, while I've not done a in-depth test, a cursory test seems to indicate that a pull of 8.3k documents from _all_docs took 3.7 s in the naive implementation (i.e. ending up with an array of documents in Node.js), and using the newline approach took 2.4 s instead. It's an interesting—if rather fragile—optimization.

[nolanlawson]

In general if PouchDB Server does anything differently from CouchDB, it's a bug. This definitely seems like one of those cases.

OTOH PouchDB has no optimization for _all_docs (with bare options) to allow it to actually stream from the filesystem, meaning that it would need to buffer all docs in-memory and then stream them out to the client. Still, it would make sense to match CouchDB's behavior here.

[nolanlawson]

Test case:

curl -X POST \
  -d '{"source":"http://examples.cloudant.com/animaldb","target":"animaldb","create_target":true}' \
  -H 'Content-Type:application/json' \
  http://localhost:6984/_replicate
curl http://localhost:6984/animaldb/_all_docs

CouchDB gives a formatted result, PouchDB Server does not.

[metametadata]

Stumbled upon this using clojure-clutch/clutch. This client library expects newlines in the response.