go-data-transfer
go-data-transfer copied to clipboard
Data Transfer Shared Component for go-filecoin & go-lotus
go-data-transfer
A go module to perform data transfers over ipfs/go-graphsync
Description
This module encapsulates protocols for exchanging piece data between storage clients and miners, both when consummating a storage deal and when retrieving the piece later.
Table of Contents
Usage
Requires go 1.13
Install the module in your package or app with go get "github.com/filecoin-project/go-data-transfer/v2"
Initialize a data transfer module
-
Set up imports. You need, minimally, the following imports:
package mypackage import ( gsimpl "github.com/ipfs/go-graphsync/impl" datatransfer "github.com/filecoin-project/go-data-transfer/v2/impl" gstransport "github.com/filecoin-project/go-data-transfer/v2/transport/graphsync" "github.com/libp2p/go-libp2p/core/host" )
-
Provide or create a libp2p host.Host
-
You will need a transport protocol. The current default transport is graphsync. go-graphsync GraphExchange
-
Create a data transfer by building a transport interface and then initializing a new data transfer instance
func NewGraphsyncDataTransfer(h host.Host, gs graphsync.GraphExchange) { tp := gstransport.NewTransport(h.ID(), gs) dt := impl.NewDataTransfer(h, tp) }
-
If needed, build out your voucher struct and its validator.
A push or pull request must include a voucher. The voucher's type must have been registered with the node receiving the request before it's sent, otherwise the request will be rejected.
datatransfer.Voucher and datatransfer.Validator are the interfaces used for validation of graphsync datatransfer messages. Voucher types plus a Validator for them must be registered with the peer to whom requests will be sent.
Example Toy Voucher and Validator
type myVoucher struct {
data string
}
func (v *myVoucher) ToBytes() ([]byte, error) {
return []byte(v.data), nil
}
func (v *myVoucher) FromBytes(data []byte) error {
v.data = string(data)
return nil
}
func (v *myVoucher) Type() string {
return "FakeDTType"
}
type myValidator struct {
ctx context.Context
ValidationsReceived chan receivedValidation
}
func (vl *myValidator) ValidatePush(
sender peer.ID,
voucher datatransfer.Voucher,
baseCid cid.Cid,
selector datamodel.Node) error {
v := voucher.(*myVoucher)
if v.data == "" || v.data != "validpush" {
return errors.New("invalid")
}
return nil
}
func (vl *myValidator) ValidatePull(
receiver peer.ID,
voucher datatransfer.Voucher,
baseCid cid.Cid,
selector datamodel.Node) error {
v := voucher.(*myVoucher)
if v.data == "" || v.data != "validpull" {
return errors.New("invalid")
}
return nil
}
Please see go-data-transfer/blob/master/types.go for more detail.
Register a validator
Before sending push or pull requests, you must register a datatransfer.Voucher
by its reflect.Type
and dataTransfer.RequestValidator
for vouchers that
must be sent with the request. Using the trivial examples above:
func NewGraphsyncDatatransfer(h host.Host, gs graphsync.GraphExchange) {
tp := gstransport.NewTransport(h.ID(), gs)
dt := impl.NewDataTransfer(h, tp)
vouch := &myVoucher{}
mv := &myValidator{}
dt.RegisterVoucherType(reflect.TypeOf(vouch), mv)
}
For more detail, please see the unit tests.
Open a Push or Pull Request
For a push or pull request, provide a context, a datatransfer.Voucher
, a host recipient peer.ID
, a baseCID cid.CID
and a selector datamodel.Node
. These
calls return a datatransfer.ChannelID
and any error:
channelID, err := dtm.OpenPullDataChannel(ctx, recipient, voucher, baseCid, selector)
// OR
channelID, err := dtm.OpenPushDataChannel(ctx, recipient, voucher, baseCid, selector)
Subscribe to Events
The module allows the consumer to be notified when a graphsync Request is sent or a datatransfer push or pull request response is received:
func ToySubscriberFunc (event Event, channelState ChannelState) {
if event.Code == datatransfer.Error {
// log error, flail about helplessly
return
}
//
if channelState.Recipient() == our.PeerID && channelState.Received() > 0 {
// log some stuff, update some state somewhere, send data to a channel, etc.
}
}
dtm := SetupDataTransferManager(ctx, h, gs, baseCid, snode)
unsubFunc := dtm.SubscribeToEvents(ToySubscriberFunc)
// . . . later, when you don't need to know about events any more:
unsubFunc()
Contributing
PRs are welcome! Please first read the design docs and look over the current code. PRs against master require approval of at least two maintainers. For the rest, please see our CONTRIBUTING guide.
License
This repository is dual-licensed under Apache 2.0 and MIT terms.
Copyright 2019. Protocol Labs, Inc.