Roadmap

[kumavis]

CLIENT

• [x] metamask ipfs zero client (trusted block head, p2p data lookups) https://github.com/kumavis/eth-ipfs-client
• [ ] browser ipfs LRU storage
• [ ] full light client

BEACONS

• [ ] trusted block head publishing (ipns? pubsub?)
• [ ] canonical hash table (merklization and publishing)
• [ ] publish new signed txs over pubsub
• [ ] advertise blockchain data availability (ipns? dht? topic into dht?)

BUCKETS

• [ ] eth data on ipfs https://github.com/ipfs/go-ipld-eth/issues/1#issuecomment-334163237
  • [X] eth-block
  • [ ] eth-block-list (ommers)
  • [ ] eth-tx / eth-tx-trie
  • [ ] eth-tx-receipt / eth-tx-receipt-trie
  • [X] eth-state-trie eth / eth-account-snapshot
  • [X] eth-storage-trie
  • [X] evm-code
• [ ] bridge boxes (musteka.la)
  • [x] phase 1: parity-ipfs + ipfs
    • [x] repository https://github.com/kumavis/ipfs-eth-bridge
    • [X] deploy a initial network of nodes running these bridges
  • [ ] phase 2: devp2p -> libp2p bridge https://github.com/MetaMask/eth-ipfs-browser-client/issues/1#issuecomment-338726046

OTHER

indexing / lookup

• [ ] eth json rpc -> ipfs lookup
  • [ ] filters middleware
  • [ ] getLogs support (ugh)
  • [ ] CHT (block number -> block hash)
  • [ ] block tracker / block head syncer
  • [ ] tx block references (tx -> block)
  • [ ] block expansion (block -> txs)

messaging

• [ ] whisper/swarm over libp2p

testing

• [ ] metamask libp2p test network

monitoring

• [ ] bridge network monitoring (serverless web application)

products and devices

• [ ] serverless block explorer (helps to "pull" IPLD links)
• [ ] daemons to pull / import data into IPFS nodes (kind of a synchronizer)

[ghost]

Moving up ^^

[danfinlay]

I like what I see happening over here. Keep up the good work, gentlemen!

[kumavis]

updated rpc->ipfs TODOs

[ghost]

@kumavis what is the plan for the smart contract code? Currently, the accounts in the state trie only store the codeHash, which is the keccak256 hash of the EVM Code (as defined in YP 4.1.), leaving open to the implementator how to store it.

Geth at least, just uses this codeHash as key in their levelDB. See here

Are we missing the IPLD codec for this element?

[kumavis]

@herman correct, we dont have a codec for it -- we should consider making one

i've been using the raw binary codec, but turns out that was actually not supported in js-ipfs and so I created a PR for it -- for sure about go-ipfs

[kumavis]

got stalled, i need to get back on it https://github.com/ipld/js-ipld-resolver/pull/90

[ghost]

@kumavis Will do the PR in go-cid as well. What is the link of the PR in js? Which code byte are we going to be using?

[kumavis]

@hermanjunge bin (raw binary) 0x55

[ghost]

Asked by @diasdavid to make a write up on my status to-date. Did it here https://github.com/ipfs/notes/issues/261

[kumavis]

some blockers:

• [x] https://github.com/multiformats/js-multicodec/pull/18
• [x] https://github.com/multiformats/js-multicodec/pull/19
• [x] https://github.com/ipld/js-ipld-resolver/pull/98

optional:

• [x] https://github.com/ipld/js-ipld-raw/issues/2

[daviddias]

@kumavis just missing https://github.com/ipld/js-ipld-resolver/pull/90, need your input there. All the others have been shipped.

@hermanjunge thank you for the write up! ❤️ https://github.com/ipfs/notes/issues/261

[kumavis]

thanks/ here's the remainders

• [x] https://github.com/multiformats/js-multicodec/pull/18/
• [x] https://github.com/ipld/js-ipld-resolver/pull/90

[kumavis]

• [ ] cache cht results!
• [ ] add cht middleware for block lookups
• [x] add all missing middleware (e.g. id mgmt)
• [ ] debug frequent chrome crashes https://support.google.com/chrome/a/answer/6271282
• [x] verify handling of empty code: 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 zb34WAxFQD4oNVkG9PRFauGV6cu51KgpQbBXARSbVDD5d9Viw
• [ ] verify handling of missing account / storage entry
• [ ] verify handling of empty storageRoot: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421 z46gvXALNuXdCn6ts67LS6JkPUkZb7zNrH6fMayQM7U9HNLDtWt (aka "Honolulu dittwitt")

[kumavis]

udapp-ipfs is live! uses infura for block-by-number lookups and logs, everything else is clientside (vm) or ipfs (state)

[kumavis]

published:

• eth-rpc-client
• eth-ipfs-client

[kumavis]

@diasdavid

• [x] js-multicodec needs a release

[kumavis]

exists, but super crashy and not fully QA'd https://github.com/MetaMask/metamask-extension/tree/ipfs-client

[ghost]

Did some editing to the first comment. Divided the goals into CLIENT, BEACONS, BUCKETS and OTHER.

[ghost]

devp2p -> libp2p bridge in Go Language (a.k.a. "Hot Importer")

• [X] Instance a node and connect to devp2p.
• [ ] Get Ethereum Data:
  • [ ] eth-block
  • [ ] eth-block-list (ommers)
  • [ ] eth-tx / eth-tx-trie
  • [ ] eth-tx-receipt / eth-tx-receipt-trie
  • [ ] eth-state-trie eth / eth-account-snapshot
  • [ ] eth-storage-trie
  • [ ] evm-code
• [ ] Setup a libp2p node with a localstore
• [ ] Logic to ask to the client for an element
  • [ ] Ask in the local storage of the client
  • [ ] Then in libp2p
  • [ ] Then in devp2p
  • [ ] If the latter happened, send the result to the local storage (ipfs dag put).
• [ ] Synchronizing daemons i.e. "I want the whole data for block B"
• [ ] Metrics API
• [ ] Deploy to musteka.la

[ghost]

From https://github.com/ipfs/go-ipld-eth-import/issues/9#issuecomment-338327558

seems actually rlpx is the transport/peer layer and devp2p is some sort of wire (sub)protocol multiplexer

So, I've reading code starting from here and here, and I agree. Also, I learned about transports here. seemed that I was having all nomenclatures wrong in my head: We didn't want to do a different wire protocol, only a transport.

I may then want to build a transport library called go-libp2p-rlpx. For that, My plan should be:

• Write down the golang APIs needed for a peer in libp2p,
• Learn how do I send messages to a peer,
• Learn how do I handle them (so the devp2p peers don´t cancel our connections for being bad neighbors, protocol-wise)

That should suffice to have an example like this running, getting peers and their ethereum data.

What are your thoughts @jwasinger?

[jwasinger]

This looks good @hermanjunge . As we discussed, I will be working on writing the javascript implementation of the rlpx transport js-libp2p-rlpx.

[dryajov]

@hermanjunge that's the approach I was suggesting earlier, it seems tho that according to @whyrusleeping, this might end up being more involved than necessary. An alternative seems to be running both devp2p and libp2p stacks in the client and doing the bridging that way, IIUC. Just leaving this here, for future reference:

whyrusleeping [11:39 AM] 
@dryajov you shouldnt mount devp2p on libp2p
[11:39] 
you want to just run both network stacks separately
[11:39] 
and have logic that ties between them
[11:39] 
a bridge, basically

[AFDudley]

@hermanjunge you were looking into adding postgres support to go-ethereum via https://github.com/ethereum/go-ethereum/tree/master/ethdb

https://github.com/ethereum/go-ethereum/issues/2690

Why was that strategy abandoned and this one adopted? It seems like support ipfs dag put from within go-ethereum would get the job done as well?

Thanks.

[ghost]

Hi @AFDudley

Why was that strategy abandoned and this one adopted? It seems like support ipfs dag put from within go-ethereum would get the job done as well?

The strategy of implementing RedisDB or PostgreSQL as storages in https://github.com/ethereum/go-ethereum/tree/master/ethdb was abandoned for lack of resources at that time. Now we have the capacity to commit in the area of blockchain storage.

There are several ways to skin a cat. The approach you mention is correct: Instead of making a put into levelDB, we just ipfs dag put. It is very likely to happen that when we are able to reach a good health of nodes running eth-ipld data, we should just implement something that talks to libp2p in the ethdb package.

The devp2p<->libp2p bridge being built is importing the github.com/ethereum/go-ethereum/p2p (and log and common). The idea behind this approach is to enable the user of this program to extract the data from the devp2p network without the need of running an ETH node. Generally speaking, from my experience in Chile, even an Ethereum fast synchronization is a huge pain, and it's getting worse. That's the motivation.

[kumavis]

break down of rpc methods and node requirements (updated)

client
  eth_protocolVersion
  eth_syncing
  eth_coinbase
  eth_mining
  eth_hashrate
  eth_gasPrice
  eth_accounts
  eth_sign
  eth_sendTransaction
  eth_sendRawTransaction
  eth_newFilter
  eth_newBlockFilter
  eth_newPendingTransactionFilter
  eth_uninstallFilter
  eth_getFilterChanges
  eth_getFilterLogs

client broadcast
  eth_sendRawTransaction

ipfs
  eth_getBlockByHash
  eth_getUncleByBlockHashAndIndex

ipns/block syncing (head tracking)
  eth_blockNumber

ipld:selectors (selectors/transforms)
  eth_getBlockTransactionCountByHash
  eth_getUncleCountByBlockHash

index:txToBlock (coselector)
  eth_getTransactionReceipt
  eth_getTransactionByHash
  eth_getTransactionByBlockHashAndIndex

log query?? / geth bloomFilterTrie / index:logToTx
  eth_getLogs (+index:logToTx)

lazy vm (slow) / remote vm + proofs (faster)
  eth_call
  eth_estimateGas

index:CHT
  eth_call
  eth_estimateGas
  eth_getBalance
  eth_getStorageAt
  eth_getTransactionCount
  eth_getCode
  eth_getBlockByNumber
  eth_getTransactionByBlockNumberAndIndex (+index:txToBlock)
  eth_getBlockTransactionCountByNumber (+ipld:selectors)
  eth_getUncleCountByBlockNumber (+ipld:selectors)
  eth_getUncleByBlockNumberAndIndex
  eth_getLogs (+log query)

[ghost]

Yet another writeup, older, but wiser. (they say)

https://github.com/ipld/ipld/issues/28#issuecomment-366832794

[ghost]

A more polished writeup:

Moving to a problem of maintaining a distributed ethereum snapshot database on js-ipfs browser peers

https://github.com/ipld/ipld/issues/29

[ghost]

Moving everything here https://github.com/MetaMask/mustekala/issues/2