Allow user to add their own data to job and batch documents

[lukasz1234567890]

Is your feature request related to a problem? Please describe. No

What is the user-facing scenario that prompted this gap in functionality? Sometimes business requirements requires me to produce multiple final documents from one source document. For example (not a real one): go through <some-list> and for each list element create final document, skip elements with attribte expired. Often I have to answer "why there is no data? we should get two records". The reason for that is the business logic implemented. At that point I don't want to go back to feature description in our issue log or read the source code. I'd prefer to execute search query in my jobs database.

Describe the solution you'd like I want to have an option to add additional information to a jobOutput. I would put there information what has happened during any particular job (where there any errors? warnings? additional statuses etc).

Describe alternatives you've considered I'm adding a log message with xdmp:log to know if a final document is not being created. I don't want to go through logs to explain business cases. I'd like to keep MarkLogic logs for technical stuff.

Additional context Add any other context or screenshots about the feature request here.

[rjrudin]

Pinging @yasht2010 - this is a valid request and something we could certainly support.

@lukasz1234567890 - how would you want to achieve this? One approach I have in mind is to allow for a user to specify a module path that would be invoked each time a Job and/or Batch document is written, where the module path could add whatever it likes to the Job/Batch document. That module path would be specified on a step or a flow.

[lukasz1234567890]

@rjrudin I'd like to add to the job output some information about harmonizations of documents. Like:

• exception caught while calling external service, document not processed {$uri}
• illegal argument (dynamic, business logic)
• etc

[rjrudin]

Thanks @lukasz1234567890 - does a new step option that allows you to specify a module path for customizing job/batch documents sound like it would work for you? e.g. (very tentative option name here)

"batchInterceptorPath": "/path/to/myModule.sjs"

Where "myModule.sjs" would give you access to the Job and Batch documents so you can fiddle with them before they're persisted.

[briantang]

@lukasz1234567890 Do you still have need for this enhancement?