conduit
conduit copied to clipboard
Error with structured records with timestamps from standalone plugin
Bug description
In a pipeline with a source using a standalone connector and with records with timestamp fields, an error occurs and causes the pipeline to stop. Here's what the logs says:
2023-02-08T11:07:30+00:00 ERR node stopped error="node pipeline1:source1 stopped with error: source stream was stopped unexpectedly: error reading from source: read stream error: error converting payload: error converting after: proto: invalid type: time.Time" component=pipeline.Service node_id=pipeline1:source1 stack=[{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/lifecycle.go","func":"github.com/conduitio/conduit/pkg/pipeline.(*Service).runPipeline.func1","line":515},{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run","line":124},{"file":"/home/haris/projects/conduitio/conduit/pkg/pipeline/stream/source.go","func":"github.com/conduitio/conduit/pkg/pipeline/stream.(*SourceNode).Run.func2","line":100},{"file":"/home/haris/projects/conduitio/conduit/pkg/plugin/standalone/v1/client.go","func":"github.com/conduitio/conduit/pkg/plugin/standalone/v1.unwrapGRPCError","line":160}]
This does not happen with built-in connectors. This has originally been found here.
Steps to reproduce
- Build the generator plugin
- Copy
conduit-connector-generator
into Conduit's connectors directory (./connectors
by default) - Start Conduit with the following pipeline:
---
version: 1.0
pipelines:
pipeline1:
status: running
name: pipeline1
description: 'Standalone generator source, file destination'
connectors:
source1:
type: source
plugin: standalone:generator
name: source1
settings:
recordCount: "-1"
readTime: "1s"
format.options: "testField:time"
format.type: "structured"
destination1:
type: destination
plugin: "builtin:file"
name: destination1
settings:
path: /tmp/file-destination.txt
Expected results: Pipeline starts successfully. Records are generated and written to the file destination.
Actual results: Pipeline fails with described error.
Version
{"version":"v0.6.0-nightly.20230207-4-g69f4265", "os":"linux", "arch":"amd64"}
Issue is being discussed here.
goal here:
- document the types we do support
- If we encounter an unsupported type we should inform the user