Custom reduce functions don't work in partitioned DB

[silvpol]

Description

In a partitioned DB using custom reduce function doesn't work, all queries to that view fail with 500 error. Creating exact same view in non-partitioned db works. Built in reduce funtions work fine though.

Steps to Reproduce

1. Set up CouchDB cluster
2. Create partitioned and non-partitioned DBs
3. Apply the example view on both
4. Try to query in both databases
5. Swap custom reduce for _count built-in
6. Now works on both DBs

Expected Behaviour

Custom reduce functions should work fine on partitioned DB or should be documented non-working

Your Environment

Docker Compose setup with three nodes using Docker Hub couchdb:3 image

• CouchDB version used: 3.2.1
• Browser name and version: Postman 9.4.12
• Operating system and version: Docker container on MacOS 10.15.7

Additional Context

Design doc:

{
    "views": {
        "one": {
            "map": "mapfun = function (doc) {\n  if (doc.id !== \"\") {\n    emit(doc.id, 1);\n  }\n};\n",
            "reduce": "function(keys, values, rereduce) {\n    if (rereduce) {\n        return sum(values);\n    } else {\n        return values.length;\n    }\n}"
        }
    },
    "filters": {},
    "updates": {},
    "language": "javascript",
    "options": {}
}

Error returned:

{
    "error": "{{invalid_ejson,{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                   <<\"01d5a505-192f-44f7-bf99-2960797f1956\">>}},\n [{jiffy,encode,2,[{file,\"src/jiffy.erl\"},{line,99}]},\n  {couch_os_process,writejson,2,[{file,\"src/couch_os_process.erl\"},{line,97}]},\n  {couch_os_process,handle_call,3,\n                    [{file,\"src/couch_os_process.erl\"},{line,200}]},\n  {gen_server,try_handle_call,4,[{file,\"gen_server.erl\"},{line,636}]},\n  {gen_server,handle_msg,6,[{file,\"gen_server.erl\"},{line,665}]},\n  {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,247}]}]}",
    "reason": "{gen_server,call,\n            [<17837.15344.57>,\n             {prompt,[<<\"reduce\">>,\n                      [<<\"function(keys, values, rereduce) {\\n    if (rereduce) {\\n        return sum(values);\\n    } else {\\n        return values.length;\\n    }\\n}\">>],\n                      [[[{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                            <<\"01d5a505-192f-44f7-bf99-2960797f1956\">>},\n                         <<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba:01d5a505-192f-44f7-bf99-2960797f1956\">>],\n                        1],\n                       [[{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                            <<\"02d1378c-f30d-4148-b5fd-3b216f612463\">>},\n                         <<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba:02d1378c-f30d-4148-b5fd-3b216f612463\">>],\n                        1],\n                       [[{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                            <<\"037362ca-f266-49d2-ab19-4cc946613de6\">>},\n                         <<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba:037362ca-f266-49d2-ab19-4cc946613de6\">>],\n                        1],\n                       [[{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                            <<\"04a2bae3-cd74-42fd-9d5d-113c827d69de\">>},\n                         <<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba:04a2bae3-cd74-42fd-9d5d-113c827d69de\">>],\n                        1],\n                       [[{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                            <<\"061f26ec-36fb-41b5-a361-44bb759f3c5b\">>},\n                         <<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba:061f26ec-36fb-41b5-a361-44bb759f3c5b\">>],\n                        1],\n                       [[{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                            <<\"0850305e-0859-4fe2-b65b-d4f96aca0366\">>},\n                         <<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba:0850305e-0859-4fe2-b65b-d4f96aca0366\">>],\n                        1],\n                       [[{p,<<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba\">>,\n                            <<\"0f4146b5-6014-4907-931d-0dc82b0d365c\">>},\n                         <<\"8a1d99e3-db1d-4a8a-a61a-bac8dad0d3ba:0f4146b5-6014-4907-931d-0dc82b0d365c\">>],\n                        1]]]},\n             infinity]}",
    "ref": 157986948
}

Docker Compose setup:

docker setup.zip

[nickva]

Thanks for your report, @silvpol. Yeah it does look like a bug.

Does it fail with a single document or do we need to create a certain number of them?

[silvpol]

@nickva I have tested and it also happens in a db with just a single doc.

Inserting custom reduce funtion via API is not restricted and causes ALL views in the given design doc to fail with the same error message, which is rather scary.

I have also tried to update design doc via Fauxton and it shows warning that partitioned DB doesn't support custom reduce. This points to a known limitation but it would be good to add it somewhere in the API docs.

[nickva]

@silvpol That makes sense.

It is quite embarrassing, it seems we never intended to support partitioned db custom reduces but added a check in Fauxton only, but not the internal design doc updater validation logic.

The fix here it seems is to update the ddoc validation logic to conform to what we intended.

[nickva]

Added an integration tests which demonstrates it as well: https://github.com/apache/couchdb/pull/3957

[nickva]

Fauxton indicates we don't intend to support it

https://github.com/apache/couchdb-fauxton/blob/f9da884aaee6ce60779b924196f5bfa3308a02c0/app/addons/documents/index-editor/components/IndexEditor.js#L48-L54

[rizwan92]

Its very essential feature we need custom reducer in partition db , shifting from normal db to partitioneddb would break our few features from platform

[trickkiste]

@nickva I can't see the logic behind not supporting custom reduce functions in partitioned DBs? Why? Partitioned DBs make a whole lot of sense e.g. in accounting apps, so you can easily separate the various accounting periods. But not being able to use custom reduce functions kind of kills the purpose of going with couchDB map/reduce in the first place.

Will this feature be added with the next release?

[trickkiste]

In a partitioned DB using custom reduce function doesn't work, all queries to that view fail with 500 error. Creating exact same view in non-partitioned db works. Built in reduce funtions work fine though.

@silvpol , did you find any way around that limitation yet? Or does one have to move back to non-partitioned DB for real?

[silvpol]

@trickkiste I have changed data design a bit to make it work with built in functions, using an array type to emit intermediate calculations and _stats function to achive what I wanted. It's a bit clunky but works good enough.