lodex
lodex copied to clipboard
Inist-CNRS
General problem in exporters
After cancel and/or restart export, we got a crash and an report in the logs :
assert.js:81
throw new assert.AssertionError({
^
AssertionError: headers have already been sent
at Object.set status [as status] (/app/node_modules/koa/lib/response.js:85:5)
at Object.status (/app/node_modules/delegates/index.js:92:31)
at Object.onError (/app/src/api/index.js:37:5)
at Array.onerror (/app/node_modules/koa/lib/application.js:133:34)
at listener (/app/node_modules/on-finished/index.js:169:15)
at onFinish (/app/node_modules/on-finished/index.js:100:5)
at callback (/app/node_modules/ee-first/index.js:55:10)
at Socket.onevent (/app/node_modules/ee-first/index.js:93:5)
at emitOne (events.js:101:20)
at Socket.emit (events.js:191:7)
at emitErrorNT (net.js:1284:8)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)
Need investigation in : https://github.com/Inist-CNRS/lodex/blob/master/src/api/controller/api/export.js
I asked for an extendednquads (this lets time to cancel the download). I canceled it. I then asked for the same export. It began. With some warning/error messages in lodex's output:
api_1 | 2017-11-30T16:38:18.468Z - info: /api/export/extendednquads? method=GET, remoteIP=::ffff:172.20.0.1, userAgent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36, status=200
api_1 | 2017-11-30T16:38:18.522Z - warn: uri=https://api.istex.fr/document?q=categories.inist.raw:("vertebres: système cardiovasculaire")&scroll=30s&output=arkIstex,link,doi,fulltext&size=5000&sid=lodex, json=true, error=null, statusCode=200, statusMessage=OK, date=Thu, 30 Nov 2017 16:38:18 GMT, x-dns-prefetch-control=off, x-frame-options=SAMEORIGIN, strict-transport-security=max-age=15552000; includeSubDomains,x-download-options=noopen, x-xss-protection=1; mode=block, access-control-allow-origin=*, access-control-allow-headers=Authorization, access-control-allow-methods=GET, OPTIONS, content-type=application/json; charset=utf-8, content-length=30, connection=close, lodexMessage=No results
api_1 | 2017-11-30T16:38:18.524Z - error: uri=https://api.istex.fr/document/?q=categories.inist.raw:(%22terre,%20ocean,%20espace%22)&size=5000&output=arkIstex,link,doi,fulltext&defaultOperator=OR&scroll=30s&scrollId=DnF1ZXJ5VGhlbkZldGNoZAAAAAAcXLRVFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIF37qhZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABdiMWIWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgXfurFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF37rRZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABdiMWMWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAcXLRWFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIF37rBZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBd-7EWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgXfuuFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIFDB6hZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABdiMWQWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgXfuwFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAF2IxZRZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABxctFcWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAbsKWhFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAIF37sxZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBd-68WTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAcXLRhFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAHFy0WRZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABxctFgWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAXYjFpFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIF37shZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBd-7QWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAXYjFoFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAF2IxZhZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABxctFoWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAXYjFnFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIF37tRZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABuwpZ0WcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAcXLRdFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAG7ClohZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABuwpZ8WcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAXYjFqFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAHFy0aRZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBd-7oWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgUMHrFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAIF37thZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABuwpaMWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgXfu4Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAHFy0WxZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBd-7cWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgUMHtFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAIFDB7BZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABuwpaYWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAbsKWkFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAG7ClnhZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABdiMWsWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgUMHuFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAG7CloBZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBd-78WTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgXfu5Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF37wBZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABxctFwWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAXYjFsFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAHFy0YBZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBd-7sWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgUMHvFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAHFy0ZxZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABdiMW4WbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgUMHxFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAG7ClpRZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABxctF8Wb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAgXfu8Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIFDB8BZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAACBd-74WTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAcXLReFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAHFy0ZBZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBd-8MWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAXYjFtFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIF37vRZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQwfIWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAbsKWnFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAHFy0YhZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBQwfMWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAXYjFvFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAG7ClqRZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBQwfgWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAcXLRmFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIFDB9BZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABxctGMWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAgUMH1FlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAG7ClqxZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABdiMXAWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgXfvBFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIFDB9hZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABxctGgWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAcXLRqFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIFDB9xZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABdiMXEWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAcXLRlFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIFDB-RZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABuwpagWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgXfvCFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAF2IxchZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABuwpaoWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAXYjFzFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAF2IxdBZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABxctGsWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAXYjF1Fm1YY1dSejBRUmptOTJsT2E5SDc1dnc%3D, json=true, error=null
api_1 | 2017-11-30T16:38:20.751Z - warn: uri=https://api.istex.fr/document?q=categories.inist.raw:("vertebres: sang, organes hematopoietiques, système reticulo-endothelial")&scroll=30s&output=arkIstex,link,doi,fulltext&size=5000&sid=lodex, json=true, error=null, statusCode=200, statusMessage=OK, date=Thu, 30 Nov 2017 16:38:20 GMT, x-dns-prefetch-control=off, x-frame-options=SAMEORIGIN, strict-transport-security=max-age=15552000; includeSubDomains, x-download-options=noopen, x-xss-protection=1; mode=block, access-control-allow-origin=*, access-control-allow-headers=Authorization, access-control-allow-methods=GET, OPTIONS, content-type=application/json; charset=utf-8, content-length=30, connection=close, lodexMessage=No results
...
But it ended in:
api_1 | 2017-11-30T16:39:40.858Z - error: uri=https://api.istex.fr/document/?q=categories.inist.raw:(%22sciences%20medicales%22)&size=5000&output=arkIstex,link,doi,fulltext&defaultOperator=OR&scroll=30s&scrollId=DnF1ZXJ5VGhlbkZldGNoZAAAAAAgXhVxFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAF2JLOBZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABdiSzYWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgUNvBFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAIF4VchZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ28IWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgUNvAFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAIF4VeRZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABdiSzcWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAXYktAFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIFDbwxZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABdiSzoWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAXYks-Fm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIF4VdBZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABxcx44Wb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAgXhV1Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4VcxZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABuwv2YWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAXYktBFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAG7C_YxZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABxcx48Wb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAgXhV4Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4VdxZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ28kWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgXhV2Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAG7C_cRZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABuwv2gWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgUNvGFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAG7C_ZBZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBQ28QWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgUNvIFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAG7C_bxZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABxcx54Wb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAgXhWNFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4VexZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABxcx5AWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAXYktFFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIF4VehZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABdiSzsWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgXhWIFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4VgxZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABuwv2UWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgUNvOFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAF2JLORZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAACBQ28cWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgXhV8Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAHFzHmRZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBQ28UWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAXYks_Fm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIF4VfxZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ29AWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgXhV-Fk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4VfRZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ28oWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAbsL9pFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAHFzHlBZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABuwv2cWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAXYktCFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAG7C_chZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBQ28sWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAcXMeRFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAF2JLRhZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAACBeFYAWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAXYks8Fm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIF4VjhZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ28wWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgXhWBFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAG7C_cxZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBeFYUWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgUNvSFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAIF4VghZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABxcx5IWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAcXMeWFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAF2JLPRZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABuwv2oWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAXYktEFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAIFDbzRZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAACBeFYwWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAcXMeTFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAHFzHlRZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABxcx5oWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAbsL9sFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAIF4VhBZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ288WVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAcXMedFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAF2JLQxZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABuwv20WcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgXhWHFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4VhhZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABuwv2sWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAcXMeYFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAG7C_cBZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABxcx5sWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAgUNvRFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAG7C_bhZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBeFYkWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgXhWKFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4VixZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABxcx5wWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAcXMeXFm9jeDZhdG9BVHppRHJyemF4dUgwOUE%3D, json=true, error=null
api_1 | 2017-11-30T16:39:40.886Z - warn: uri=https://api.istex.fr/document?q=categories.inist.raw:("techniques et industries diverses")&scroll=30s&output=arkIstex,link,doi,fulltext&size=5000&sid=lodex, json=true, error=null, statusCode=200, statusMessage=OK, date=Thu, 30 Nov 2017 16:39:41 GMT, x-dns-prefetch-control=off, x-frame-options=SAMEORIGIN, strict-transport-security=max-age=15552000; includeSubDomains, x-download-options=noopen, x-xss-protection=1; mode=block, access-control-allow-origin=*, access-control-allow-headers=Authorization, access-control-allow-methods=GET, OPTIONS, content-type=application/json; charset=utf-8, content-length=30, connection=close, lodexMessage=No results
api_1 | 2017-11-30T16:39:54.836Z - warn: uri=https://api.istex.fr/document?q=categories.inist.raw:("domaines interdisciplinaires: science des materiaux; rheologie")&scroll=30s&output=arkIstex,link,doi,fulltext&size=5000&sid=lodex, json=true, error=null, statusCode=200, statusMessage=OK, date=Thu, 30 Nov 2017 16:39:54 GMT, x-dns-prefetch-control=off, x-frame-options=SAMEORIGIN, strict-transport-security=max-age=15552000; includeSubDomains, x-download-options=noopen, x-xss-protection=1; mode=block, access-control-allow-origin=*, access-control-allow-headers=Authorization, access-control-allow-methods=GET, OPTIONS, content-type=application/json; charset=utf-8, content-length=30, connection=close, lodexMessage=No results
api_1 | 2017-11-30T16:40:02.232Z - error: uri=https://api.istex.fr/document/?q=categories.inist.raw:(%22sciences%20exactes%20et%20technologie%22)&size=5000&output=arkIstex,link,doi,fulltext&defaultOperator=OR&scroll=30s&scrollId=DnF1ZXJ5VGhlbkZldGNoZAAAAAAbsMTMFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAIF4a3xZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABuwxM0WcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAcXMt6Fm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIFDhGBZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABdiUJAWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgXhriFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAG7DE0hZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABuwxM8WcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgXhrgFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIF4a4RZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ4RkWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAbsMTOFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAG7DE0BZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABuwxNEWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAcXMt7Fm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAHFzLfBZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABdiUJEWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgXhrlFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAG7DE0xZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBQ4R0WVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAcXMt9Fm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIF4a5BZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ4RoWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgXhrjFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAHFzLfhZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBQ4RsWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgUOEcFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAIFDhHhZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABdiUJYWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAXYlCXFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAF2JQlBZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABxcy38Wb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAXYlCTFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAF2JQkhZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAABuwxNQWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgXhrnFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAIFDhIRZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABdiUJUWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAXYlCcFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAF2JQmhZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAACBQ4R8WVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAgUOEgFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAIFDhIhZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABuwxNcWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgXhrmFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAF2JQmRZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAACBeGugWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAXYlCYFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAG7DE1RZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABdiUJ8WbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAXYlCbFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAHFzLhhZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAACBeGukWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAbsMTWFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAHFzLgBZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABxcy4EWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAbsMTYFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAHFzLghZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABdiUJ0WbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAcXMuIFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIF4a7BZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ4SUWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAbsMTZFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAIFDhIxZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAACBeGusWTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAgXhrqFk12SFNUcEg3UkI2aU9jZ1lLaHFiYlEAAAAAHFzLixZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABdiUJ4WbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAXYlCgFm1YY1dSejBRUmptOTJsT2E5SDc1dncAAAAAG7DE2hZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAABuwxNsWcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgUOEkFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAHFzLhBZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABxcy4cWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAbsMTcFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAHFzLihZvY3g2YXRvQVR6aURycnpheHVIMDlBAAAAABuwxN0WcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAgUOEmFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAG7DE3hZwQlI1c1hpeVNHZUY3R28zVDNCR3B3AAAAACBeGu0WTXZIU1RwSDdSQjZpT2NnWUtocWJiUQAAAAAcXMuFFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIFDhJxZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAACBQ4SsWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAcXMuJFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIFDhKBZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABxcy4MWb2N4NmF0b0FUemlEcnJ6YXh1SDA5QQAAAAAbsMTgFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAIF4a7hZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAACBQ4SkWVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAbsMThFnBCUjVzWGl5U0dlRjdHbzNUM0JHcHcAAAAAIFDhLBZVRTR6Y2NvSlFES19VMGpFc05VOUhBAAAAABuwxN8WcEJSNXNYaXlTR2VGN0dvM1QzQkdwdwAAAAAcXMuMFm9jeDZhdG9BVHppRHJyemF4dUgwOUEAAAAAIF4a7xZNdkhTVHBIN1JCNmlPY2dZS2hxYmJRAAAAABdiUKIWbVhjV1J6MFFSam05MmxPYTlINzV2dwAAAAAgUOEqFlVFNHpjY29KUURLX1UwakVzTlU5SEEAAAAAF2JQoRZtWGNXUnowUVJqbTkybE9hOUg3NXZ3AAAAACBQ4S0WVUU0emNjb0pRREtfVTBqRXNOVTlIQQAAAAAcXMuNFm9jeDZhdG9BVHppRHJyemF4dUgwOUE%3D, json=true, error=null
api_1 | 2017-11-30T16:40:02.268Z - warn: uri=https://api.istex.fr/document?q=categories.inist.raw:("vertebres: système respiratoire")&scroll=30s&output=arkIstex,link,doi,fulltext&size=5000&sid=lodex, json=true, error=null, statusCode=200, statusMessage=OK, date=Thu, 30 Nov 2017 16:40:02 GMT, x-dns-prefetch-control=off, x-frame-options=SAMEORIGIN, strict-transport-security=max-age=15552000; includeSubDomains, x-download-options=noopen, x-xss-protection=1; mode=block, access-control-allow-origin=*, access-control-allow-headers=Authorization, access-control-allow-methods=GET, OPTIONS, content-type=application/json; charset=utf-8, content-length=30, connection=close, lodexMessage=No results
api_1 | 2017-11-30T16:40:10.859Z - warn: uri=https://api.istex.fr/document?q=categories.inist.raw:("manutention, levage. stockage. emballage")&scroll=30s&output=arkIstex,link,doi,fulltext&size=5000&sid=lodex, json=true, error=null, statusCode=200, statusMessage=OK, date=Thu, 30 Nov 2017 16:40:10 GMT, x-dns-prefetch-control=off, x-frame-options=SAMEORIGIN, strict-transport-security=max-age=15552000; includeSubDomains, x-download-options=noopen, x-xss-protection=1; mode=block, access-control-allow-origin=*, access-control-allow-headers=Authorization, access-control-allow-methods=GET, OPTIONS, content-type=application/json; charset=utf-8, content-length=30, connection=close, lodexMessage=No results
mongo_1 | 2017-11-30T16:40:14.223+0000 I NETWORK [conn223] end connection 172.20.0.3:48536 (9 connections now open)
without crashing application.
But, the number of Document returned is ridiculously low:
$ grep Document ~/Téléchargements/export\ \(44\).nq | wc -l
141629
I was waiting for millions of documents.
The https://api.istex.fr/document/?q=categories.inist.raw:(%22terre,%20ocean,%20espace%22) only should return 15810 docs. https://api.istex.fr/document/?q=categories.inist.raw:(%22sciences%20medicales%22): 115317 https://api.istex.fr/document/?q=categories.inist.raw:(%22sciences%20exactes%20et%20technologie%22): 22169
Those 3 queries alone totalize 153296 documents (which total is above the 141629 documents contained in the exported file).
Explanation: a bug in the ISTEX API. Any query with more than 5000 results yields only the first 5000 documents. It's diagnosed, and will be fixed next week.
Sorry @QuanticPotatoes , but I have to close this issue because too old. Hoping that this was resolved at the time.
