n8n icon indicating copy to clipboard operation
n8n copied to clipboard

Support ndjson for application/json requests

Open pinkasey opened this issue 2 years ago • 4 comments

Backstory -

I have a workflows that starts with a webhook. This webhook is called with a request with Content-Type: application/json, and ndjson body, like this:

{ "id": 1, "message": "hello" }
{ "id": 2, "message": "I'm a JSON inside a NDJSON" }

I am completely unable to make the webhook work, because the parsing of the json is happening in Server.ts, so fiddling with the workflow settings (e.g, setting rawBody to true) - has no effect. I get an error in the log, and the workflow is not started.

What I propose in this PR -

If a request is made with Content-Type: application/json, and parsing the body as json fails - try to parse it as ndjson.

  • If that works - great! return an array of all the jsons in the body (1 line = 1 item)
  • if not - log an error, and pass the original answer.

Note: In a perfect world, the API I'm consuming would send the request to the webhook with application/x-ndjson. But I have no control over that - it's a 3rd party. And that would be the experience of most users of n8n. I also tried to see what would've happened if the Content-Type was right - I got an empty body.

This PR is related to this feature-request from body-parser: https://github.com/expressjs/body-parser/issues/478 (If that feature-request is handled, then we could use it here instead of this PR. But I don't know if they'd answer)

pinkasey avatar Dec 19 '22 13:12 pinkasey

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Dec 19 '22 13:12 CLAassistant

General questions to this approach:

  • what if you cant determine the type of the request body based on the Content-Type header (for example, incorrect header? or if the type is generic)

whoiskatrin avatar Jan 05 '23 16:01 whoiskatrin

General questions to this approach:

  • what if you cant determine the type of the request body based on the Content-Type header (for example, incorrect header? or if the type is generic)

If we can't determine the type of the request body based on the Content-Type header, then we continue as usual, no change there. I believe the default body type is plain text. If that causes an error - very well. An incorrect Content-Type is a serious error by the client.

pinkasey avatar Mar 16 '23 11:03 pinkasey

I've accepted CR comments, and rebased this branch.

I've since stopped using n8n, so I will not make further changes to this PR. I hope this PR can be of help to others.

pinkasey avatar Mar 16 '23 11:03 pinkasey

Sorry that this never got reviewed or merged. Unfortunately this can't be rebased anymore, since Server.ts has changed quite a lot.

If anyone needs ndjson support on webhooks, you can enable the "Binary Data" option on the webhook, and then parse this data in a "Code" node.

If you think this should actually be a feature in n8n, then please create a Feature request, so that this can first be discussed with the team before anything is implemented 🙏🏽

netroy avatar Jun 20 '24 15:06 netroy