ipfs-lite Add examples

This project looks promising :) It would be nice to add some examples to how to use it

Mar 22 '19 13:03 koalalorenzo

I'm not sure I wanted to commit this but since I did, it might be useful until I write a better example: https://github.com/hsanjuan/ipfs-lite/blob/d2191f217f81879baf5e5d09d51d5233628ac430/test/ipfs-lite.go

Apr 16 '19 08:04 hsanjuan

@hsanjuan The example is giving me the following output

02:12:02.368 ERROR   ipfslite: dial attempt failed: no good addresses ipfs.go:122
02:12:02.368 ERROR   ipfslite: dial attempt failed: no good addresses ipfs.go:122
02:12:02.369 ERROR   ipfslite: dial attempt failed: no good addresses ipfs.go:122
02:12:02.369 ERROR   ipfslite: dial attempt failed: no good addresses ipfs.go:122
Hello World

What's the error message being caused by?

Apr 25 '19 09:04 bonedaddy

hey @postables that's probably because some of the default bootstrap peers are offline or don't work when it tries to connect to them. Maybe worth reducing this to a warning... however it does connect correctly to other bootstrap peers and as you see it was able to fetch the content from somewhere.

Apr 25 '19 09:04 hsanjuan

ah okay that makes sense, thanks!

Apr 25 '19 10:04 bonedaddy

Getting

me@host:~$ go run /home/me/go/src/github.com/hsanjuan/ipfs-lite/examples/litepeer/litepeer.go 
2019-10-27T20:03:00.795+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip6/2a03:b0c0:0:1010::23:1001/tcp/4001] failed to negotiate security protocol: rsa keys must be >= 2048 bits to be useful
  * [/ip4/178.62.158.247/tcp/4001] failed to negotiate security protocol: rsa keys must be >= 2048 bits to be useful
2019-10-27T20:03:00.943+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip6/2604:a880:800:10::4a:5001/tcp/4001] dial tcp6 [2604:a880:800:10::4a:5001]:4001: connect: connection refused
  * [/ip4/104.236.76.40/tcp/4001] dial tcp4 104.236.76.40:4001: connect: connection refused
2019-10-27T20:03:01.074+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip6/2604:a880:1:20::203:d001/tcp/4001] failed to negotiate security protocol: rsa keys must be >= 2048 bits to be useful
  * [/ip4/104.236.179.241/tcp/4001] failed to negotiate security protocol: rsa keys must be >= 2048 bits to be useful
2019-10-27T20:03:01.079+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip4/104.131.131.82/tcp/4001] dial tcp4 104.131.131.82:4001: connect: connection refused
2019-10-27T20:03:01.121+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip4/128.199.219.111/tcp/4001] failed to negotiate security protocol: rsa keys must be >= 2048 bits to be useful
  * [/ip6/2400:6180:0:d0::151:6001/tcp/4001] failed to negotiate security protocol: rsa keys must be >= 2048 bits to be useful
2019-10-27T20:03:05.798+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip6/2604:1380:2000:7a00::1/tcp/4001] dial tcp6 [2604:1380:2000:7a00::1]:4001: connect: connection refused
  * [/ip4/147.75.83.83/tcp/4001] dial tcp4 147.75.83.83:4001: connect: connection refused
  * [/ip4/139.178.70.249/tcp/4001] dial tcp4 139.178.70.249:4001: connect: connection refused
  * [/ip6/2604:1380:1000:6000::7/tcp/4001] dial tcp6 [2604:1380:1000:6000::7]:4001: i/o timeout
2019-10-27T20:03:05.798+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip6/2604:1380:1000:6000::1/tcp/4001] dial tcp6 [2604:1380:1000:6000::1]:4001: connect: connection refused
  * [/ip4/139.178.68.71/tcp/4001] dial tcp4 139.178.68.71:4001: connect: connection refused
  * [/ip6/2604:1380:1000:6000::3/tcp/4001] dial tcp6 [2604:1380:1000:6000::3]:4001: i/o timeout
2019-10-27T20:03:05.799+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip6/2604:1380:0:c100::1/tcp/4001] dial tcp6 [2604:1380:0:c100::1]:4001: connect: connection refused
  * [/ip4/147.75.77.187/tcp/4001] dial tcp4 147.75.77.187:4001: connect: connection refused
  * [/ip4/139.178.70.181/tcp/4001] dial tcp4 139.178.70.181:4001: connect: connection refused
  * [/ip6/2604:1380:1000:6000::5/tcp/4001] dial tcp6 [2604:1380:1000:6000::5]:4001: i/o timeout
2019-10-27T20:03:05.800+0100	WARN	ipfslite	go-log/log.go:187	failed to dial : all dials failed
  * [/ip6/2604:1380:3000:1f00::1/tcp/4001] dial tcp6 [2604:1380:3000:1f00::1]:4001: connect: connection refused
  * [/ip4/147.75.94.115/tcp/4001] dial tcp4 147.75.94.115:4001: connect: connection refused
  * [/ip4/139.178.88.21/tcp/4001] dial tcp4 0.0.0.0:4005->139.178.88.21:4001: i/o timeout
  * [/ip6/2604:1380:1000:6000::9/tcp/4001] dial tcp6 [2604:1380:1000:6000::9]:4001: i/o timeout
2019-10-27T20:03:05.800+0100	WARN	ipfslite	go-log/log.go:192	only connected to 0 bootstrap peers out of 9
2019-10-27T20:03:05.800+0100	WARN	dht	go-log/log.go:192	self walk: error: failed to find any peer in table

What is going wrong here? Can the code be changed so that the default bootstrap peer list from https://docs.ipfs.io/guides/examples/bootstrap/ is used?

Oct 27 '19 19:10 probonopd

Hi, the problem is libp2p was upgraded to only allow 2048 rsa keys and the default bootstrappers use 1024 ones.

You can either:

set crypto.MinRsaKeyBits (https://github.com/libp2p/go-libp2p-core/blob/master/crypto/rsa_common.go#L13) to 1024
Use any other bootstrappers
Wait, the team is working to resolve this by launching more bootstrappers that were already in the list: https://github.com/ipfs/infra/issues/378

I'm going to update the example with the first fix though...

Oct 28 '19 09:10 hsanjuan

Maybe it would be a good idea to extend the example to also write (add) something to IPFS. But wait, why are we getting a different CID for the "Hello World"?

package main

// This example launches an IPFS-Lite peer, adds a hello-world to
// and fetches a hello-world hash from the IPFS network.

import (
	"context"
	"fmt"
	"io/ioutil"
	"strings"

	ipfslite "github.com/hsanjuan/ipfs-lite"
	"github.com/ipfs/go-cid"
	corecrypto "github.com/libp2p/go-libp2p-core/crypto"
	crypto "github.com/libp2p/go-libp2p-crypto"
	"github.com/multiformats/go-multiaddr"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// Bootstrappers are using 1024 keys. See:
	// https://github.com/ipfs/infra/issues/378
	corecrypto.MinRsaKeyBits = 1024

	ds, err := ipfslite.BadgerDatastore("test")
	if err != nil {
		panic(err)
	}
	priv, _, err := crypto.GenerateKeyPair(crypto.RSA, 2048)
	if err != nil {
		panic(err)
	}

	listen, _ := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/4005")

	h, dht, err := ipfslite.SetupLibp2p(
		ctx,
		priv,
		nil,
		[]multiaddr.Multiaddr{listen},
	)

	if err != nil {
		panic(err)
	}

	lite, err := ipfslite.New(ctx, ds, h, dht, nil)
	if err != nil {
		panic(err)
	}

	lite.Bootstrap(ipfslite.DefaultBootstrapPeers())

	fmt.Println("Adding file")
	n, err := lite.AddFile(ctx, strings.NewReader("Hello World"), nil)
	if err != nil {
		panic(err)
	}

	fmt.Println(n.Cid())
	// Why don't we get "QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u"?

	fmt.Println("")
	cc := "QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u"
	fmt.Println("Getting", cc)

	c, _ := cid.Decode(cc)
	rsc, err := lite.GetFile(ctx, c)
	if err != nil {
		panic(err)
	}
	defer rsc.Close()
	content, err := ioutil.ReadAll(rsc)
	if err != nil {
		panic(err)
	}

	fmt.Println(string(content))
}

Oct 28 '19 20:10 probonopd

Line-breaks?

-------- Original Message -------- On Oct 28, 2019, 21:13, probonopd wrote:

Maybe it would be a good idea to extend the example to also write (add) something to IPFS. But wait, why are we getting a different CID for the "Hello World"?

package main

// This example launches an IPFS-Lite peer, adds a hello-world to // and fetches a hello-world hash from the IPFS network.

import ( "context" "fmt" "io/ioutil" "strings"

ipfslite "github.com/hsanjuan/ipfs-lite" "github.com/ipfs/go-cid" corecrypto "github.com/libp2p/go-libp2p-core/crypto" crypto "github.com/libp2p/go-libp2p-crypto" "github.com/multiformats/go-multiaddr" )

func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel()

// Bootstrappers are using 1024 keys. See: // https://github.com/ipfs/infra/issues/378 corecrypto.MinRsaKeyBits = 1024

ds, err := ipfslite.BadgerDatastore("test") if err != nil { panic(err) } priv, _, err := crypto.GenerateKeyPair(crypto.RSA, 2048) if err != nil { panic(err) }

listen, _ := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/4005")

h, dht, err := ipfslite.SetupLibp2p( ctx, priv, nil, []multiaddr.Multiaddr{listen}, )

if err != nil { panic(err) }

lite, err := ipfslite.New(ctx, ds, h, dht, nil) if err != nil { panic(err) }

lite.Bootstrap(ipfslite.DefaultBootstrapPeers())

fmt.Println("Adding file") n, err := lite.AddFile(ctx, strings.NewReader("Hello World"), nil) if err != nil { panic(err) }

fmt.Println(n.Cid()) // Why don't we get "QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u"?

fmt.Println("") cc := "QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u" fmt.Println("Getting", cc)

c, _ := cid.Decode(cc) rsc, err := lite.GetFile(ctx, c) if err != nil { panic(err) } defer rsc.Close() content, err := ioutil.ReadAll(rsc) if err != nil { panic(err) }

fmt.Println(string(content)) }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Oct 29 '19 09:10 hsanjuan

To answer my own question, we don't get a "Qm..." hash because there is apparently a new has format now that does not begin with "Qm...".

Follow-up question: How can the example be made to actually serve the file? Right now the example just exits, and peers cannot download the added file.

Oct 29 '19 17:10 probonopd

@probonopd ah, yes I guess it uses CIDv1 by default, instead of v0.

How can the example be made to actually serve the file? Right now the example just exits, and peers cannot download the added file.

You need to keep the peer (the example) running then.. I think adding select{ } at the end might make it.

Oct 29 '19 18:10 hsanjuan

Can you please post a working example? This does not seem to work for me:

package main

// This example launches an IPFS-Lite peer, adds a hello-world to
// and fetches a hello-world hash from the IPFS network.

import (
	"context"
	"fmt"
	"io/ioutil"
	"os"

	ipfslite "github.com/hsanjuan/ipfs-lite"
	"github.com/ipfs/go-cid"
	corecrypto "github.com/libp2p/go-libp2p-core/crypto"
	crypto "github.com/libp2p/go-libp2p-crypto"
	"github.com/multiformats/go-multiaddr"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// Bootstrappers are using 1024 keys. See:
	// https://github.com/ipfs/infra/issues/378
	corecrypto.MinRsaKeyBits = 1024

	fmt.Println("Generating datastore")

	ds, err := ipfslite.BadgerDatastore("test")
	if err != nil {
		panic(err)
	}
	priv, _, err := crypto.GenerateKeyPair(crypto.RSA, 2048)
	if err != nil {
		panic(err)
	}

	fmt.Println("Listening")

	listen, _ := multiaddr.NewMultiaddr("/ip4/0.0.0.0/tcp/4005")

	h, dht, err := ipfslite.SetupLibp2p(
		ctx,
		priv,
		nil,
		[]multiaddr.Multiaddr{listen},
		ipfslite.Libp2pOptionsExtra...,
	)

	if err != nil {
		panic(err)
	}

	lite, err := ipfslite.New(ctx, ds, h, dht, nil)
	if err != nil {
		panic(err)
	}

        fmt.Println("Peer address", h.ID())
	fmt.Println("Bootstrapping")

	lite.Bootstrap(ipfslite.DefaultBootstrapPeers())

	fmt.Println("Adding file")

	f, err := os.Open("/etc/os-release")
	if err != nil {
		fmt.Println(err)
	} else {

		n, err := lite.AddFile(ctx, f, nil)
		if err != nil {
			fmt.Println(err)
		}

		f.Close()

		fmt.Println("Added, check it at https://explore.ipld.io/#/explore/" + n.String())

	}
	fmt.Println("")
	cc := "bafybeiduiecxoeiqs3gyc6r7v3lymmhserldnpw62qjnhmqsulqjxjmtzi"
	fmt.Println("Getting", cc)

	c, _ := cid.Decode(cc)
	rsc, err := lite.GetFile(ctx, c)
	if err != nil {
		fmt.Println(err)
	}
	defer rsc.Close()
	content, err := ioutil.ReadAll(rsc)
	if err != nil {
		fmt.Println(err)
	}

	fmt.Println(string(content))

	// Run forever. Is this sufficient to serve the files?

	ch := make(chan struct{})
	<-ch
}

Oct 29 '19 18:10 probonopd

@probonopd that seems enough, but mind that your nodes needs to be online for a while to become well connected and should be reachable by others etc.

Oct 29 '19 19:10 hsanjuan

Unfortunately it did not seem to work for me. I also cannot access the file from the browser on localhost.

Oct 29 '19 19:10 probonopd

what if you manually connect your local ipfs daemon to /ip4/127.0.0.1/tcp/4005/ipfs/<the generated peer id-you will have to print h.ID() to get it> ?

The example does not run the mDNS discovery service so it can only be reached from dht and that may not work very well for an spontaneous peer (and the ports should be open, maybe even extra addresses should be added to the peerstore etc).

Oct 29 '19 19:10 hsanjuan

Do I need to have a local ipfs daemon running in addition to my Go program? (I want to embed p2p file serving into my Go program, without the need for additional software to be installed on the user's machine.)

Oct 29 '19 19:10 probonopd

When you said you cannot access the file from the browser on localhost I assumed you were using the local gateway provided by a running ipfs daemon. You mean you were using the ipfs.io gateways?

You don't need to have the local daemon, but it's a way to test, for example, what the Go program peer is reporting as addresses, whether it is reachable from an external IP etc.

Oct 29 '19 20:10 hsanjuan

I was trying 127.0.0.1:4005/ipfs/<hash> under the assumption that would work. Nevermind if it doesn't. What I am more concerned with is that the file I tried to share never showed up on the ipfs.io gateway. Is there a way to somehow trigger the ipfs.io gateway to connect to my machine and fetch/cache the file as fast as possible?

And is there a way to see the "health" of my program's connection to the ipfs network?

Oct 29 '19 20:10 probonopd

I was trying 127.0.0.1:4005/ipfs/ under the assumption that would work

Ah no, that port is not an http endpoint, it's a libp2p endpoint. What you were trying to access is a gateway, which ipfs-lite does not provide.

Is there a way to somehow trigger the ipfs.io gateway to connect to my machine and fetch/cache the file as fast as possible?

No.. you need to keep your node running, and when the gateway looks up for the hash in the DHT it will eventually find that your node is a provider and try to connect to it on the addresses that the node is reporting.

A good question here is what addresses is the example node reporting and I'm not sure. It may not be reporting any external address. Normally, if you want to use your host properly, you enable a bunch of things like NAT hole punching and autodiscovery.

You can peek through here for an example on how initialization of a host with autonat, circuit etc is done: https://github.com/ipfs/ipfs-cluster/blob/e713969a806f6a60171e00a405060b827ba8c84c/clusterhost.go

I will try to improve the example (or the helper function) to do all this, but I'll need a couple of days for that...

Oct 29 '19 20:10 hsanjuan

Normally, if you want to use your host properly, you enable a bunch of things like NAT hole punching and autodiscovery.

Ah, I was hoping that this would be done automagically by IPFS, like ipfs-desktop seemingly does (because using it I was able to host files successfully in no time).

I will try to improve the example (or the helper function) to do all this, but I'll need a couple of days for that...

That will be much appreciated. Thanks for the great work you are doing :+1:

Oct 29 '19 20:10 probonopd

Yes, so the difference here is that ipfs-lite is the bare minimum and lets you provide your own preconfigured libp2p host, which you may use for a bunch of other things. I'd say that how the libp2p host is setup is a bit out of the scope of this library, but you're right that it should help a bit more (since it does provide helpers and tries to be useful).

Note that you could also run a full go-ipfs, with ALL the things the daemon running behind ipfs-desktop would provide. In fact, instructions were added recently: https://github.com/ipfs/go-ipfs/tree/master/docs/examples/go-ipfs-as-a-library

Oct 29 '19 20:10 hsanjuan

@probonopd I added a variable Libp2pOptionsExtra. You can pass that to SetupLibp2p (I updated the example) and get a peer with nat hole punching etc. Might improve your issue.

Nov 04 '19 17:11 hsanjuan

Thank you. However it is still not working for me; is there a way to increase verbosity? I can download but seemingly not upload/seed.

Nov 04 '19 18:11 probonopd

Thank you. However it is still not working for me; is there a way to increase verbosity? I can download but seemingly not upload/seed.

Can you share your peer address, and the hash of the content you are seeding?

Nov 04 '19 18:11 hsanjuan

How can I know my peer address?

Nov 04 '19 18:11 probonopd

The address should be something like /ip4//tcp/4005/p2p/<whatever h.ID() prints>

Nov 05 '19 09:11 hsanjuan

Updated https://github.com/hsanjuan/ipfs-lite/issues/1#issuecomment-547579318, this is the output:

Generating datastore
Listening
Peer address QmcAtotNSxpbVuQpQ1sNun17Q8xVWTRQE5SrGVN2htPddT
Bootstrapping
Adding file

Nov 05 '19 20:11 probonopd

What is the public ip?

Nov 06 '19 09:11 hsanjuan

ipfs-lite ipfs-lite copied to clipboard

Add examples

ipfs-lite
ipfs-lite copied to clipboard