radicle-dev
Branches sometimes not replicated on fetch
radicle-link @ e640fed9770de476b3983c30b231fd4dcd3cf7a7 When connecting two identical peers (the radicle org node) on the localhost (same machine), one of which has 4 projects and the other being fresh (empty monorepo), after replication of the 4 projects, I see that only one of them has had the default branch replicated, while the others only have certain refs replicated. I've included a diff of the trees.
On the left is the "server" that has all the data, and on the right is the "client" that is fetching from the server.
(logs: https://gist.github.com/cloudhead/02c469951f10c2ccc4d55a97d4adf362)
monorepo1/git/refs/namespaces | monorepo2/git/refs/namespaces/
├── hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto ├── hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto
│ └── refs │ └── refs
│ ├── rad │ ├── rad
│ │ ├── id │ │ ├── id
│ │ ├── ids │ │ ├── ids
│ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ │ └── signed_refs │ │ └── signed_refs
│ └── remotes │ └── remotes
│ ├── hydgnjorosxetwr3xrqsfazuornz7m8wrx7brdyk5nxc4j9j9jhjxg <
│ │ └── rad <
│ │ └── ids <
│ ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe │ ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
│ │ ├── heads <
│ │ │ ├── cloudhead <
│ │ │ │ └── master <
│ │ │ └── master <
│ │ └── rad │ │ └── rad
│ │ ├── id │ │ ├── id
│ │ ├── ids │ │ ├── ids
│ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ │ ├── self │ │ ├── self
│ │ └── signed_refs │ │ └── signed_refs
│ ├── hynkyndc6w3p8urucakobzna7sxwgcqny7xxtw88dtx3pkf7m3nrzc | │ └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
│ │ └── rad <
│ │ └── ids <
│ └── hyyui4i75gkh14um3bhc3f67bm6n3hhof61584ukw6zbwnkyzhuj1o <
│ └── rad │ └── rad
│ └── ids | │ ├── id
> │ ├── ids
> │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
> │ └── signed_refs
├── hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy ├── hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy
│ └── refs │ └── refs
│ ├── rad │ ├── rad
│ │ ├── id │ │ ├── id
│ │ ├── ids │ │ ├── ids
│ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ │ └── signed_refs │ │ └── signed_refs
│ └── remotes │ └── remotes
│ ├── hybz9gfgtd9d4pd14a6r66j5hz6f77fed4jdu7pana4fxaxbt369kg <
│ │ └── rad <
│ │ └── ids <
│ ├── hyd4ge6w1ig37cjg31zzi5kdgd91xtwezkzx91ypzwnh4dn4m5qm7h <
│ │ └── rad <
│ │ └── ids <
│ ├── hyd7cb39wws6jz5wmifzwtjn8u57siw6knipqttk6nhc657r1fozaw <
│ │ └── rad <
│ │ └── ids <
│ ├── hydgnjorosxetwr3xrqsfazuornz7m8wrx7brdyk5nxc4j9j9jhjxg <
│ │ └── rad <
│ │ └── ids <
│ ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe │ ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
│ │ ├── heads │ │ ├── heads
│ │ │ └── master │ │ │ └── master
│ │ └── rad │ │ └── rad
│ │ ├── id │ │ ├── id
│ │ ├── ids │ │ ├── ids
│ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ │ ├── self │ │ ├── self
│ │ └── signed_refs │ │ └── signed_refs
│ ├── hynkyndc6w3p8urucakobzna7sxwgcqny7xxtw88dtx3pkf7m3nrzc | │ └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
│ │ └── rad <
│ │ └── ids <
│ ├── hynpd6jij8ifug9zkxojj9q5k9mcfxu1pak1t71qbc3b1od1yrmp7h <
│ │ └── rad <
│ │ └── ids <
│ ├── hynqybwjzwzfcrydm1iar535qog77fst8jyk8w3yny9nz6d94cwq81 <
│ │ └── rad <
│ │ └── ids <
│ ├── hyy4ebh8amk3ypyawgpjnfpbgkxqwdo5n1iif5s17yghwmewrf4d8a <
│ │ └── rad <
│ │ └── ids <
│ ├── hyyao9irufccjs8pjjkijasgdt1ts4toiynkceg3scg58th95mmo5h <
│ │ └── rad <
│ │ └── ids <
│ ├── hyycogqwxy5bg9m9oyqydyf5ychz7qt311fossax5qbngt393b1o91 <
│ │ └── rad <
│ │ └── ids <
│ ├── hyyf6mr6awmj7f4ug9fyfibp9zcidqsz5bjsc5a49on4wrm6kxmztq <
│ │ └── rad <
│ │ └── ids <
│ ├── hyyrxmbfwe33p54cc41je7fhq7mccg56jiy4szeyehhk4pj9uxwhzs <
│ │ └── rad <
│ │ └── ids <
│ ├── hyyui4i75gkh14um3bhc3f67bm6n3hhof61584ukw6zbwnkyzhuj1o <
│ │ └── rad <
│ │ └── ids <
│ └── hyyz8tw1yp134qprtczndhr43ki8rwjqz13etitpxs8qpc69mjjqxh <
│ └── rad │ └── rad
│ └── ids | │ ├── id
> │ ├── ids
> │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
> │ └── signed_refs
├── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio ├── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ └── refs │ └── refs
│ └── rad │ └── rad
│ └── id │ └── id
├── hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo ├── hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo
│ └── refs │ └── refs
│ ├── rad │ ├── rad
│ │ ├── id │ │ ├── id
│ │ ├── ids │ │ ├── ids
│ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ │ └── signed_refs │ │ └── signed_refs
│ └── remotes │ └── remotes
│ └── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe | │ ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
│ ├── heads | │ │ └── rad
│ │ ├── cloudhead | │ │ ├── id
│ │ │ └── ci | │ │ ├── ids
│ │ └── master | │ │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
> │ │ ├── self
> │ │ └── signed_refs
> │ └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
│ └── rad │ └── rad
│ ├── id │ ├── id
│ ├── ids │ ├── ids
│ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ ├── self <
│ └── signed_refs │ └── signed_refs
└── hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo └── hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo
└── refs └── refs
├── rad ├── rad
│ ├── id │ ├── id
│ ├── ids │ ├── ids
│ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ └── signed_refs │ └── signed_refs
└── remotes └── remotes
├── hyddqac7smapegotydiuny3db1i3u16egp4krqaujor6ch5819tadr <
│ └── rad <
│ └── ids <
├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe ├── hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
│ ├── heads <
│ │ └── master <
│ └── rad │ └── rad
│ ├── id │ ├── id
│ ├── ids │ ├── ids
│ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio │ │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
│ ├── self │ ├── self
│ └── signed_refs │ └── signed_refs
└── hynkyndc6w3p8urucakobzna7sxwgcqny7xxtw88dtx3pkf7m3nrzc | └── hyniionkx6ycg7ncth8z3htbhmgt4jcwtzoxx68ptuj3qinfede56n
└── rad └── rad
└── ids | ├── id
> ├── ids
> │ └── hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
> └── signed_refs
98 directories, 35 files | 48 directories, 42 files
Some additional notes:
- It's always the same project that successfully replicates its branches, I've run this 3-4 times with this result
- Since the state of the "server" is also replicated (from upstream), I'm not 100% sure that it is in perfect state. But there are no obvious signs that something is wrong with it, and I'm able to read blobs from the repo, through surf.
Could you maybe compare the contents of the signed_refs to the refs which are
actually present?
I narrowed some logs down to the first namespace hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto here log.txt.gz.
When I grep'd /hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto.*heads: the heads of the signed refs were always empty. So I think the mystery here is why those heads never get entered into the signed_refs.
When I grep'd /hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto.*heads: the heads of the signed refs were always empty. So I think the mystery here is why those heads never get entered into the signed_refs.
Just realising that this might be the signed_refs of the seed and not the peer with the heads.
Still seeing this issue with 2df248a8e077d02cbb075da073dbe7ab9e282a70 FYI
Could you maybe compare the contents of the
signed_refsto the refs which are actually present?
Here's the first error that I'm getting from attempting to replicate the same repo:
Nov 12 15:16:11.771 DEBUG org-node: Replication of rad:git:hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto succeeded: ReplicateResult { updated_tips: {RefLike("refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master"): Oid(acce81a6568ec4342586a71a
Nov 12 15:16:11.773 DEBUG org-node: Setting repository head for rad:git:hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto to "refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master"
Nov 12 15:16:11.773 ERROR org-node: Error setting head in track_peers: failed to set project head: path "/Users/adaszko/tmp/2nd-node/git/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master" does not exist
Nov 12 15:16:11.773 ERROR org-node: Error tracking peers: failed to set project head: path "/Users/adaszko/tmp/2nd-node/git/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master" does not exist
As you're suggesting @kim, I don't see anything wrong with replication up until the contents of signed_refs:
SIGREFS: {
hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq: Refs {
heads: {
OneLevel(
"master",
): Oid(
acce81a6568ec4342586a71a314485408c264068,
),
},
rad: {
OneLevel(
"id",
): Oid(
c99d99f55cd277bba2187f136051b377b17b56f8,
),
},
tags: {},
notes: {},
remotes: Remotes(
{
hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo: Remotes(
{
hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe: Remotes(
{},
),
},
),
hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe: Remotes(
{},
),
},
),
},
}
When it comes to the file system, it looks like follows after the error (I'm filtering by the relevant identities):
> hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto
16/184 (0)
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/signed_refs
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/id
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo/rad/signed_refs
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyyqpngdoe4x4oto3emfdppbw7sj1pfaghbpmmhz5rqiuqg8uofmeo/rad/id
> refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/signed_refs
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/self
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/id
logs/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/id
logs/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
As can be seen, unsuprisingly, refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master is missing.
OTOH, when we filter by the identity that was present in signed_refs, we find that its heads replicated properly:
> hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq
20/184 (0)
> refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
refs/namespaces/hnrkk9c4zt9thuxhwi1ukxqcrs5tmhbtcsony/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
refs/namespaces/hnrkyghsrokxzxpy9pww69xr11dr9q7edbxfo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
refs/namespaces/hnrkyzfpih4pqsw3cp1donkmwsgh9w5fwfdwo/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/heads/master
refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/ids/hnrkqdpm9ub19oc8dccx44echy76hzfsezyio
refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/signed_refs
refs/namespaces/hnrkjajuucc6zp5eknt3s9xykqsrus44cjimy/refs/remotes/hybju4ci46qn4nj844ogabz45y5o98nqrdxs3pmksj5fok9wgwa1bq/rad/id
Full logs (with my changes to the logging code unfortunately) are available in a gist. Those are logs from the replicating peer, not the bootstrap peer, BTW.
Am I reading this correctly that you're expecting heads/master of ...zo645pe to be in the remotes of ...9wgwa1b, but
- either
...9wgwa1bdoesn't have it in the first place (although it knows about...zo645pe) - or the fetching node doesn't consider the
signed_refsof...zo645pe?
Sorry if that was a little bit unclear.
Basically, I'm going by the error message: I'm expecting refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master to exist on the file system.
IOW, I'm expecting hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master to be in remotes of hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto.
Yes yes. What I'm trying to parse is: there seem to be three nodes involved in your setup:
- the fetching node
- the remote node
- the node whose data 1. is interested in
So the question is: does 2. have the data of 3.? If not, we need to look into why. If yes, then 1. fails to discover that the data is there.
There are only two nodes :). I'm trying to keep the bug reproduction as small as possible. I'm also using the same repository data as @cloudhead in his examples, so the identities and hashes should mostly match with his comments (except for the replicating node identity, which is different in my setup).
The network topology couldn't be simpler: 1st node: let's call it the bootstrap node 2nd node: let's call it the replicating node
bootstrap node
|
|
|
v
replicating node
Test scenario:
Everything is happening on a single machine. The bootstrap node serves test data to the replicating node. It is assumed the test data got extracted into ~/.radicle/root.
In a local checkout of the client-services/org-node directory:
Bootstrap node is run like so: cargo run -- --subgraph 'https://api.thegraph.com/subgraphs/name/radicle-dev/radicle-orgs' --rpc-url REDACTED --identity ~/.radicle/identity --root ~/.radicle/root --orgs 0x8152237402e0f194176154c3a6ea1eb99b611482 --listen 127.0.0.1:8776
~/tmp/2nd-node/ is empty except for containing a manually generated identity file:
The replicating node is run like so: cargo run -- --subgraph 'https://api.thegraph.com/subgraphs/name/radicle-dev/radicle-orgs' --rpc-url REDACTED --identity ~/tmp/2nd-node/identity --root ~/tmp/2nd-node --orgs 0x8152237402e0f194176154c3a6ea1eb99b611482 --bootstrap [email protected]:8776 --listen 127.0.0.1:8777
Now, on to answer your question: ~/.radicle/root/git/refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/heads/master, i.e. the bootstrap repository file does exist on the file system but isn't being replicated to the replicating node repo.
Aha, thanks.
The signed_refs of the remote you're trying to fetch looks like this:
commit e562431913af9fa8dad3aaa6cea3a57631af100d (refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/signed_refs)
Author: cloudhead <cloudhead@hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe>
Date: Fri Aug 27 15:30:33 2021 +0200
Update rad/signed_refs for rad:git:hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto
diff --git a/refs b/refs
index 56e2f01..8a78704 100644
--- a/refs
+++ b/refs
@@ -1 +1 @@
-{"refs":{"heads":{"master":"0bad586e7f0749c70ee33b06eb6882a2523b0b99"},"rad":{"id":"c99d99f55cd277bba2187f136051b377b17b56f8"},"tags":{},"notes":{},"remotes":{}},"signature":"hybrp6ceians1jffsrhaoeu4aayzq4bc1ozh9enqomof3sur5kqh7ke4jamgoqk66rjdtcarww3sgamtq48jpaxxhkgmf84rodsrtx5ob"}
\ No newline at end of file
+{"refs":{"heads":{"cloudhead/master":"0bad586e7f0749c70ee33b06eb6882a2523b0b99","master":"0bad586e7f0749c70ee33b06eb6882a2523b0b99"},"rad":{"id":"c99d99f55cd277bba2187f136051b377b17b56f8"},"tags":{},"notes":{},"remotes":{}},"signature":"hyn37z39mbp3qbs645nxcma5iub4qpk6bozyarw7i31q6ty1ozc8xessaqqnw8qp1sjpxbr1dpqnw66no49j37tqu56fdqiepdoe6oeyd"}
For some reason, it's not heads/master but heads/cloudhead/master. Does the replicating node receive this?
Nope, heads/cloudhead/master is missing:
Filtering the file system tree on the replicating node side by hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe
gives:
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/signed_refs
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/self
refs/namespaces/hnrkbtw9t1of4ykjy6er4qqwxtc54k9943eto/refs/remotes/hyn9diwfnytahjq8u3iw63h9jte1ydcatxax3saymwdxqu1zo645pe/rad/id
Hm ok. Something seems a little off in the logs vs. the repository state you posted, so it's hard to make sense of it. Could you share the key that was used for the bootstrap node so I could run this scenario myself?
@kim I've made the reproduction smaller by removing two repos and I've created an end-to-end test case using Python at https://github.com/radicle-dev/radicle-client-services/pull/71. Instruction on how to use are in the README file. I hope it makes it easier to debug this issue and also serves as a regression suite for the future.