How can I use multiple targets and pipelines?

[bmaupin]

I'd like to be able to send logs to syslog and to stdout.

I know I need a pipeline for syslog (pino-syslog+pino-socket), and that works fine, e.g.

const pino = require('pino');
const logger = pino({
  transport: {
    pipeline: [
      {
        target: 'pino-syslog',
      },
      {
        target: 'pino-socket',
        options: {
          port: 5140,
          mode: 'udp',
        },
      },
    ],
  },
});

My guess would be that I'd need to combine pipeline with targets so that I can send the logs to multiple targets; one target for stdout (I'm testing with pino-pretty but the format doesn't really matter) and the other target for the syslog pipeline.

I tried this:

const logger = pino({
  transport: {
    targets: [
      {
        target: 'pino-pretty',
      },
      {
        pipeline: [
          {
            target: 'pino-syslog',
          },
          {
            target: 'pino-socket',
            options: {
              port: 5140,
              mode: 'udp',
            },
          },
        ],
      },
    ],
  },
});

But it throws this error: TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined

Thanks!

[mcollina]

This is not currently possible. You can't mix & match. However it would be amazing to support that, it's would be a really nice feature to add. However I don't have time to work on it, so it would be great to have a PR.

[dbacarel]

I'll give it a try - I could see this as a nice to have in preparation to the v8 release

[Starkrights]

Heya! I was just thinking the same thing and came across this issue. It looks like @dbacarel's fork hasn't had any commits in relation to this particular feature, so I started poking around myself to see what a potential PR might look like.

From what I understand, the transport function checks for the fullOptions.targets and fullOptions.pipeline properties, and sets the target passed to buildStream as either worker.js or worker-pipeline.js, which return a stream that executes the expected functionality (worker: input->multistream to all targets | worker-pipeline: input->pipeline(first stream -> last stream)`

With the way those workers are written, I see a couple of potential solutions.

Solution 1: Abuse the existing functionality of worker.js

Essentially instead of setting the top-level 'target' to worker-pipeline.js if we see a pipeline property, we can just parse the pipeline obj into its own "target" for the worker.js module, so that it spins up a worker-pipeline.js transport to include in its multistream.

Implementation summary

In lib/transport.js, if options.pipeline exists, instead of mapping it onto options.targets, we could map the pipeline targets as such:

User input options

Identical to as they exist now

{
  targets: [
    /* Targets as mapped from the "targets" property */
    {target: 'target1', /*(...)*/},
    {target: 'target2', /*(...)*/},
    /*(...)*/
  },
  pipeline: [
    {target: 'pipelineTarg1', /*(...)*/},
    {target: 'pipelineTarg2', /*(...)*/},
  ]
}   

Resultant options structure

/*options.targets*/ {
  targets: [
    /* Targets as mapped from the "targets" property */
    {target: 'target1', /*(...)*/},
    {target: 'target2', /*(...)*/},
    /*(...) rest of options.targets targets*/
    
    // pipeline target
    {
      target: bundlerOverrides['pino-pipeline-worker'] || join(__dirname, 'worker-pipeline.js'), 
      options: {
        targets: [    
          {target: 'pipelineTarg1', /*(...)*/},
          {target: 'pipelineTarg2', /*(...)*/}
        ]
      }
    }
}

In essence, it just utilizes worker-pipeline.js as any other transport target, where the "options" property for the target is just the target array for the pipeline.

When worker.js creates all the transports, it gets to the pipeline target, gets a stream with loadTransportStreamBuilder(t.target), calls the default export of worker-pipeline.js with fn(t.options) which equates to fn( {targets: [(...)]} ), or exactly what worker-pipeline.js expects

Pros:

• Least amount of new and rewritten code
• No API changes... ish
  • Currently, if there is a targets and a pipeline property in the options, the pipeline just gets skipped/ignored. That could be a breaking change in cases where both are defined. It would be a pretty dumb breaking change, but a breaking change nonetheless.

Cons:

• Hacky
• Isn't very expandable- only allows for one pipeline.

Solution 2: Sol 1, but with modified user input

Similar to solution 1, but involves some logic rewrites and an addition to the API. Instead of using the top-level pipeline property, we can take a boolean property within each obj in the targets array, eg targets:[ {pipeline: true, targets: [] }, (...) ], so that multiple pipelines can be included. This could be combined with solution 1 to allow for backwards compatibility with the top-level pipeline property.

TL;DR: Different input, same output as sol 1. Also allows for multiple pipeline outputs.

Implementation summary

User input options

{
  targets: [
    /* Targets as mapped from the "targets" property */
    {target: 'target1', /*(...)*/},
    {target: 'target2', /*(...)*/},
    /*(...)*/
    {
      pipeline: true,
      targets: [
        {target: 'pipelineTarg1', /*(...)*/},
        {target: 'pipelineTarg2', /*(...)*/},
      ]
  }
}   

Code example

// lib/transport.js

if (targets) {
    target = bundlerOverrides['pino-worker'] || join(__dirname, 'worker.js')
    options.targets = targets.map((dest) => {
    
      if(!dest.pipeline)    
        return {
          ...dest,
          target: fixTarget(dest.target)
        }
      else {
        return {
          options: {...dest.targets},
          target: bundlerOverrides['pino-pipeline-worker'] || join(__dirname, 'worker-pipeline.js')
        }
    })
  }

Solution 3: New worker transport

We could create a new worker file, mixed-worker.js, that handles the creation of individual worker and worker-pipeline transports. That is, the pipeline transports would not be included under the worker worker.js multistream, and would rather receive stream data from the new worker file. The mixed-worker.js transport could have its own multistream, which just forwards the log data to both (all? if multiple pipelines are to be desired?) the worker and worker-pipeline transports.

I'm gonna skip over the implementation here since the input/output options are the same as prior, and the new transport would essentially just be a small transport.

Considerations

I've got no idea what the performance impact of one, let alone multiple pipelines running under the worker.js transport's multistream would be. If it's not great, would it be within the realm of sanity to spawn a second worker-thread for running pipelines?

I'm sure there are other things I'm missing too so thoughts and criticism are of course welcome.

@mcollina, would you mind giving this a skim at some point? I'd love to have a bit of a sanity check, as well as a confirmation that this is still within y'alls scope before starting work on a real PR haha.

[mcollina]

What seems the most interesting to me is the creation of a mixed-worker.js that can handle multiple targets and pipelines.

[Starkrights]

Gotcha, I'll start chipping away at a PR then!

[ouya99]

how is progress with this draft? @mcollina I wonder if it is possible to use transport with pipeline together with a pino.multistream as a workaround?

[ouya99]

@bmaupin facing some problem, did you find workaround to your issue?

[bmaupin]

@ouya99 No, being able to log to syslog and stdout at the same time is a requirement for us so we used winston and winston-syslog instead.

[3hson]

Can we have this feature anytime soon?

[mcollina]

If someone contributes it (or is willing to fund its development).

[weiyin]

@mcollina It would be immensely helpful if this could be developed. Would you be able to get the PR from @Starkrights merged? If necessary, I may be able to put some work into it.

[3hson]

@mcollina i'm able to contribute too

[mcollina]

That PR still needs work (tests, docs) to be finished.

[dbacarel]

I'll give it a try - I could see this as a nice to have in preparation to the v8 release

second time's the charm, meantime I missed also the v9 release deadline 🥲 . I pushed a draft PR (at the moment is a prototype) to propose the introduction of named pipelines.

Will continue polishing it but wanted to taste the waters first regarding the idea. What do you think about them? Any useful?

[dbacarel]

@bmaupin

Targets and (multiple) pipelines can now be defined within targets.

Example of usage is shown in documentation page: https://getpino.io/#/docs/api?id=pinotransportoptions-gt-threadstream

In addition, levels can be also specified for each pipeline too.

The feature is available in the v9.1.0 released yesterday https://github.com/pinojs/pino/releases/tag/v9.1.0

Thanks @mcollina for the review and feedbacks.