conduit icon indicating copy to clipboard operation
conduit copied to clipboard

Error with structured records with timestamps from standalone plugin

Open hariso opened this issue 2 years ago • 2 comments

Bug description

In a pipeline with a source using a standalone connector and with records with timestamp fields, an error occurs and causes the pipeline to stop. Here's what the logs says:

2023-02-08T11:07:30+00:00 ERR node stopped error="node pipeline1:source1 stopped with error: source stream was stopped unexpectedly: error reading from source: read stream error: error converting payload: error converting after: proto: invalid type: time.Time" component=pipeline.Service node_id=pipeline1:source1 stack=[{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/lifecycle.go","func":"github.com/conduitio/conduit/pkg/pipeline.(*Service).runPipeline.func1","line":515},{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run","line":124},{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run.func2","line":100},{"file":"/home/haris/projects/conduitio/conduit/pkg/plugin/standalone/v1/client.go","func":"github.com/conduitio/conduit/pkg/plugin/standalone/v1.unwrapGRPCError","line":160}]

This does not happen with built-in connectors. This has originally been found here.

Steps to reproduce

  1. Build the generator plugin
  2. Copy conduit-connector-generator into Conduit's connectors directory (./connectors by default)
  3. Start Conduit with the following pipeline:
---
version: 1.0
pipelines:
  pipeline1:
    status: running
    name: pipeline1
    description: 'Standalone generator source, file destination'
    connectors:
      source1:
        type: source
        plugin: standalone:generator
        name: source1
        settings:
          recordCount: "-1"
          readTime: "1s"
          format.options: "testField:time"
          format.type: "structured"
      destination1:
        type: destination
        plugin: "builtin:file"
        name: destination1
        settings:
          path: /tmp/file-destination.txt

Expected results: Pipeline starts successfully. Records are generated and written to the file destination.

Actual results: Pipeline fails with described error.

Version

{"version":"v0.6.0-nightly.20230207-4-g69f4265", "os":"linux", "arch":"amd64"}

hariso avatar Feb 08 '23 10:02 hariso

Issue is being discussed here.

hariso avatar Mar 06 '23 15:03 hariso

goal here:

  • document the types we do support
  • If we encounter an unsupported type we should inform the user

simonl2002 avatar Jul 24 '23 16:07 simonl2002