create custom writer doesn't seem to work

[alba054]

hello @gosom, It's been a week since I clone your google-maps-scraper and I want to implement my own writer and have explored your scrapemate repo as well but I realize that it doesn't seem to work. Below is my custom writer maybe I can get some hint or clue where I do wrong thank you

func NewResultWriter(writer *kafka.Writer) scrapemate.ResultWriter {
	return &resultWriter{writer: writer}
}

type resultWriter struct {
	writer *kafka.Writer
}

func (r *resultWriter) Run(ctx context.Context, in <-chan scrapemate.Result) error {
	for result := range in {
		entry, ok := result.Data.(*gmaps.Entry)

		if !ok {
			return errors.New("invalid data type")
		}

		// defer writer.Close()

		// The message you want to send
		message, err := json.Marshal(entry)

		if err != nil {
			fmt.Println(err)
		}

		// Write a message to the Kafka topic
		err = r.writer.WriteMessages(ctx, kafka.Message{
			Key:   []byte(fmt.Sprintf("Key-%d", time.Now().Unix())), // Optional: Key for partitioning
			Value: message,
		})
		if err != nil {
			log.Fatalf("Failed to write message to Kafka: %s", err)
		}

		fmt.Println("Message successfully written to Kafka topic:", "sample-data-batch-01")

		// if err := r.saveEntry(ctx, entry); err != nil {
		// 	return err
		// }
	}

	return nil
}

[gosom]

Hi @alba054 ,

entry, ok := result.Data.(*gmaps.Entry)

		if !ok {
			return errors.New("invalid data type")
		}

I guess it returns an error here.

In the latest version added support to write custom writers via Go's plugin system.

please see the updated README and the example and let me know if this solves your problem.

[gosom]

@alba054 closing this issue since there is support to add a custom writer now.