Go Dcp Couchbase

Go Dcp Couchbase streams documents from Couchbase Database Change Protocol (DCP) and writes to Couchbase bucket in near real-time.

Features

• Less resource usage and higher throughput.
• Update multiple documents for a DCP event(see Example).
• Handling different DCP events such as expiration, deletion and mutation(see Example).
• Managing batch configurations such as maximum batch size, batch bytes, batch ticker durations.
• Scale up and down by custom membership algorithms(Couchbase, KubernetesHa, Kubernetes StatefulSet or Static, see examples).
• Easily manageable configurations.

Concepts

General Concept

Merge at target bucket

Example

Struct Config

package main

import (
  "github.com/Trendyol/go-dcp-couchbase"
  "time"

  "github.com/Trendyol/go-dcp-couchbase/config"
  dcpConfig "github.com/Trendyol/go-dcp/config"
)

func main() {
  c, err := dcpcouchbase.NewConnectorBuilder(&config.Config{
    Dcp: dcpConfig.Dcp{
      Hosts:      []string{"localhost:8091"},
      Username:   "user",
      Password:   "password",
      BucketName: "dcp-test",
      Dcp: dcpConfig.ExternalDcp{
        Group: dcpConfig.DCPGroup{
          Name: "groupName",
          Membership: dcpConfig.DCPGroupMembership{
            RebalanceDelay: 3 * time.Second,
          },
        },
      },
      Metadata: dcpConfig.Metadata{
        Config: map[string]string{
          "bucket":     "dcp-test-meta",
          "scope":      "_default",
          "collection": "_default",
        },
        Type: "couchbase",
      },
      Debug: true,
    },
    Couchbase: config.Couchbase{
      Hosts:          []string{"localhost:8091"},
      Username:       "user",
      Password:       "password",
      BucketName:     "dcp-test-backup",
      BatchSizeLimit: 10,
      RequestTimeout: 10 * time.Second,
    },
  }).SetMapper(dcpcouchbase.DefaultMapper).Build()
  if err != nil {
    panic(err)
  }

  defer c.Close()
  c.Start()
}

Configuration

Dcp Configuration

Check out on go-dcp

Couchbase Specific Configuration

Exposed metrics

For DCP related metrics see also.

Breaking Changes

Contributing

Go Dcp Couchbase is always open for direct contributions. For more information please check our Contribution Guideline document.

License

Released under the MIT License.