start-os icon indicating copy to clipboard operation
start-os copied to clipboard

Published 20 hours ago • verified publisher icon Start9Labs

[bug]: .local access sometimes breaks on all services.

Open chrisguida opened this issue 3 years ago • 21 comments

Not sure how to reproduce. I was testing spark and the .local address worked briefly, then stopped. I went and tried to visit other services but they don't work either. I tried on both mac and linux and that also doesn't work. All I get is this, on all services:

Screen Shot 2022-01-06 at 11 39 36 AM

Nothing that suspicious in the logs, though maybe I missed something:

Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.087 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Loading services for mDNS
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.091  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "embassy-30297ba3.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.092 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for pjzz3s232yrelgbrcjt2im74hnfe3t7crjlw55smrqmxojp5tbiehoad.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.095  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "pjzz3s232yrelgbrcjt2im74hnfe3t7crjlw55smrqmxojp5tbiehoad.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.096 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for r7l7qgzlp3o7iciz53upx4t2hu7l7hj4fpuqiqksxhvirosodhlcg5id.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.098  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "r7l7qgzlp3o7iciz53upx4t2hu7l7hj4fpuqiqksxhvirosodhlcg5id.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.100 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for skgq5mnyd6sxchfpglxw2qre7dlauxfujnkrvplzuqd5rbv5y7bxfbid.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.103  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "skgq5mnyd6sxchfpglxw2qre7dlauxfujnkrvplzuqd5rbv5y7bxfbid.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.104 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for 5bfhafo4jze47wkxbbxsm5s6hilfjqcbd34wuteeds4mqc6tq4ftqmid.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.106  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "5bfhafo4jze47wkxbbxsm5s6hilfjqcbd34wuteeds4mqc6tq4ftqmid.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.107 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for tryniygn3ireoqx2umxypwnfc4gtswhgmgtulcdvjdu6nftkba7xqhqd.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.110  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "tryniygn3ireoqx2umxypwnfc4gtswhgmgtulcdvjdu6nftkba7xqhqd.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.110 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for 76lx5hfsfaxhkrztapw52grgol62qra73bd57yhc6jvyzfzsio2do5id.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.112  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "76lx5hfsfaxhkrztapw52grgol62qra73bd57yhc6jvyzfzsio2do5id.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.112 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for 7tnq3yaeci6lrw7v2ova74kl3ttcrqks2j26rdo5il46owzj5lxvkiyd.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.114  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "7tnq3yaeci6lrw7v2ova74kl3ttcrqks2j26rdo5il46owzj5lxvkiyd.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.114 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for z57zpgc6yzzkbl2yqcchoao5gmk4frcsk65xqm2acwzg4owz73njqpyd.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.116  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "z57zpgc6yzzkbl2yqcchoao5gmk4frcsk65xqm2acwzg4owz73njqpyd.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.116 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for yliha5qbq54onrpnmzylsgvgoacnhbwxxadqazlno43bw7mzz3z6qcyd.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.117  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "yliha5qbq54onrpnmzylsgvgoacnhbwxxadqazlno43bw7mzz3z6qcyd.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.118 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for q6z5xc5zc2m4gvv4zcrk6cyipw3uzc4jlake6bwst44ap3h3oaqhz6id.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.119  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "q6z5xc5zc2m4gvv4zcrk6cyipw3uzc4jlake6bwst44ap3h3oaqhz6id.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.119 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for 77cmjrq5qioeyplo5njaiu44zorv5xbgjaumkzdo54qn4m2r3ihcs6yd.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.129  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "77cmjrq5qioeyplo5njaiu44zorv5xbgjaumkzdo54qn4m2r3ihcs6yd.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.130 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for ghymfs422rvwqmoi3wgv4qddjlvhw5vgr65zkshimdpdobq3kfwcozid.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.132  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "ghymfs422rvwqmoi3wgv4qddjlvhw5vgr65zkshimdpdobq3kfwcozid.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.132 DEBUG run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Adding mdns CNAME entry for gfoim2ptc2h7y24yphg4vifoiadiosrp7iv6y36n6362g7dtdwkhqkad.local
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.135  INFO run_main:remove{pkg_id=PackageId(Id("spark-wallet"))}: embassy::net::mdns: Published "gfoim2ptc2h7y24yphg4vifoiadiosrp7iv6y36n6362g7dtdwkhqkad.local"
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.684 ERROR embassy::manager: service crashed: 1: uncaughtException, stopping process
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Error: connect ENOENT /mnt/c-lightning/shared/lightning-rpc
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]:     at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1157:16)
Jan 06 17:30:51 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:51.684  INFO run_main: embassy::net: Generating SSL Certificate mountpoints for spark-wallet
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.769 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Loading services for mDNS
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.771  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "embassy-30297ba3.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.772 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for pjzz3s232yrelgbrcjt2im74hnfe3t7crjlw55smrqmxojp5tbiehoad.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.774  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "pjzz3s232yrelgbrcjt2im74hnfe3t7crjlw55smrqmxojp5tbiehoad.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.774 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for r7l7qgzlp3o7iciz53upx4t2hu7l7hj4fpuqiqksxhvirosodhlcg5id.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.778  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "r7l7qgzlp3o7iciz53upx4t2hu7l7hj4fpuqiqksxhvirosodhlcg5id.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.778 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for skgq5mnyd6sxchfpglxw2qre7dlauxfujnkrvplzuqd5rbv5y7bxfbid.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.780  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "skgq5mnyd6sxchfpglxw2qre7dlauxfujnkrvplzuqd5rbv5y7bxfbid.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.780 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for 5bfhafo4jze47wkxbbxsm5s6hilfjqcbd34wuteeds4mqc6tq4ftqmid.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.782  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "5bfhafo4jze47wkxbbxsm5s6hilfjqcbd34wuteeds4mqc6tq4ftqmid.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.783 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for tryniygn3ireoqx2umxypwnfc4gtswhgmgtulcdvjdu6nftkba7xqhqd.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.785  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "tryniygn3ireoqx2umxypwnfc4gtswhgmgtulcdvjdu6nftkba7xqhqd.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.785 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for 76lx5hfsfaxhkrztapw52grgol62qra73bd57yhc6jvyzfzsio2do5id.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.791  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "76lx5hfsfaxhkrztapw52grgol62qra73bd57yhc6jvyzfzsio2do5id.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.792 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for 7tnq3yaeci6lrw7v2ova74kl3ttcrqks2j26rdo5il46owzj5lxvkiyd.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.794  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "7tnq3yaeci6lrw7v2ova74kl3ttcrqks2j26rdo5il46owzj5lxvkiyd.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.795 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for z57zpgc6yzzkbl2yqcchoao5gmk4frcsk65xqm2acwzg4owz73njqpyd.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.797  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "z57zpgc6yzzkbl2yqcchoao5gmk4frcsk65xqm2acwzg4owz73njqpyd.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.797 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for yliha5qbq54onrpnmzylsgvgoacnhbwxxadqazlno43bw7mzz3z6qcyd.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.800  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "yliha5qbq54onrpnmzylsgvgoacnhbwxxadqazlno43bw7mzz3z6qcyd.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.801 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for q6z5xc5zc2m4gvv4zcrk6cyipw3uzc4jlake6bwst44ap3h3oaqhz6id.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.805  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "q6z5xc5zc2m4gvv4zcrk6cyipw3uzc4jlake6bwst44ap3h3oaqhz6id.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.805 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for 77cmjrq5qioeyplo5njaiu44zorv5xbgjaumkzdo54qn4m2r3ihcs6yd.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.807  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "77cmjrq5qioeyplo5njaiu44zorv5xbgjaumkzdo54qn4m2r3ihcs6yd.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.807 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for ghymfs422rvwqmoi3wgv4qddjlvhw5vgr65zkshimdpdobq3kfwcozid.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.809  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "ghymfs422rvwqmoi3wgv4qddjlvhw5vgr65zkshimdpdobq3kfwcozid.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.809 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for tbsrlmmrfjgcuca2gqrnjdcvrin775hxubccupq4c56qiae4eh4vyhqd.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.814  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "tbsrlmmrfjgcuca2gqrnjdcvrin775hxubccupq4c56qiae4eh4vyhqd.local"
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.814 DEBUG run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Adding mdns CNAME entry for gfoim2ptc2h7y24yphg4vifoiadiosrp7iv6y36n6362g7dtdwkhqkad.local
Jan 06 17:30:53 embassy-30297ba3 embassyd[4493]: Jan 06 17:30:53.817  INFO run_main:add{pkg_id=PackageId(Id("spark-wallet")) ip=172.18.0.7}: embassy::net::mdns: Published "gfoim2ptc2h7y24yphg4vifoiadiosrp7iv6y36n6362g7dtdwkhqkad.local"
Jan 06 17:34:08 embassy-30297ba3 embassyd[4493]: Jan 06 17:34:08.595 DEBUG embassy::net::tor: Attempting to self-check tor address
Jan 06 17:34:09 embassy-30297ba3 embassyd[4493]: Jan 06 17:34:09.934 DEBUG embassy::net::tor: Successfully verified main tor address liveness at na63or7ygqykbthyuoe2lsxihqmspbzciwn5vhnsyo3xmkfgjjuy56yd.onion
Jan 06 17:39:09 embassy-30297ba3 embassyd[4493]: Jan 06 17:39:09.936 DEBUG embassy::net::tor: Attempting to self-check tor address
Jan 06 17:39:15 embassy-30297ba3 embassyd[4493]: Jan 06 17:39:15.091 DEBUG embassy::net::tor: Successfully verified main tor address liveness at na63or7ygqykbthyuoe2lsxihqmspbzciwn5vhnsyo3xmkfgjjuy56yd.onion

Note: system UI works on .local, just not the services.

Also, .onion access still works.

chrisguida avatar Jan 06 '22 17:01 chrisguida

Restarting embassyd and the avahi daemon have no effect. A possibly unrelated error was emitted when I restarted embassyd:

Screen Shot 2022-01-06 at 12 06 48 PM

chrisguida avatar Jan 06 '22 18:01 chrisguida

are you able to ping the .local's?

ProofOfKeags avatar Jan 06 '22 18:01 ProofOfKeags

Sorry, my box got locked up trying to create a backup. Will let you know if it happens again

chrisguida avatar Jan 06 '22 22:01 chrisguida

Ok just recreated again. No, ping does not work

% ping 3dftml7i6rpz5iwbgjgowrthmtoah6xcxrohlubnfunm5dzhrixnllyd.local 
ping: cannot resolve 3dftml7i6rpz5iwbgjgowrthmtoah6xcxrohlubnfunm5dzhrixnllyd.local: Unknown host

chrisguida avatar Jan 07 '22 00:01 chrisguida

Without repro steps idk how I could possibly approach this.

ProofOfKeags avatar Jan 07 '22 19:01 ProofOfKeags

If it's nondeterministic, there won't be exact repro steps. Just wanted to document the issue in case someone else encounters it. I have seen this happen multiple times, but it's unpredictable.

chrisguida avatar Jan 07 '22 19:01 chrisguida

I'll start doing a battery of UAT today and see if it comes up then. That said, leaving this as a P1 when we can't reliably reproduce it may not be tenable. I'll punt on attending to this immediately and keep a passive eye out.

ProofOfKeags avatar Jan 07 '22 19:01 ProofOfKeags

Attempted fix in #1108 is gonna be DOA. New approach we are going to take is refactoring the net controller to have install, uninstall, start and stop methods rather than a simple add/remove dynamic. This should reduce the thrash in mdns and should keep it stable in the presence of service crash loops.

ProofOfKeags avatar Jan 21 '22 22:01 ProofOfKeags

From diagnostics that @chrisguida took. This should point the way to how to deal with this.

Feb 10 19:16:35 embassy-2ef08502 embassyd[1900]: Feb 10 19:16:35.652  WARN init{embassyd_addr=127.0.0.1:80 embassyd_tor_key=TorSecretKey(****) tor_control=127.0.0.1:9051 import_root_ca=None}: embassy::net::mdns: AvahiCallback: ClientState = AVAHI_CLIENT_S_RUNNING
Feb 10 19:16:35 embassy-2ef08502 embassyd[1900]: Feb 10 19:16:35.653  WARN init{embassyd_addr=127.0.0.1:80 embassyd_tor_key=TorSecretKey(****) tor_control=127.0.0.1:9051 import_root_ca=None}: embassy::net::mdns: AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_UNCOMMITED
Feb 10 19:28:13 embassy-2ef08502 embassyd[1900]: Feb 10 19:28:13.400  WARN shutdown: embassy::context::rpc: 2 RPC Context(s) are still being held somewhere. This is likely a mistake.
Feb 11 18:12:45 embassy-2ef08502 embassyd[2059]: Feb 11 18:12:45.707  WARN init{embassyd_addr=127.0.0.1:80 embassyd_tor_key=TorSecretKey(****) tor_control=127.0.0.1:9051 import_root_ca=None}: embassy::net::mdns: AvahiCallback: ClientState = AVAHI_CLIENT_S_REGISTERING
Feb 11 18:12:45 embassy-2ef08502 embassyd[2059]: Feb 11 18:12:45.708  WARN init{embassyd_addr=127.0.0.1:80 embassyd_tor_key=TorSecretKey(****) tor_control=127.0.0.1:9051 import_root_ca=None}: embassy::net::mdns: AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_UNCOMMITED
Feb 11 18:40:40 embassy-2ef08502 embassyd[2059]: Feb 11 18:40:40.688  WARN shutdown: embassy::context::rpc: 2 RPC Context(s) are still being held somewhere. This is likely a mistake.
Feb 11 19:41:20 embassy-2ef08502 embassyd[1796]: Feb 11 19:41:20.788  WARN init{embassyd_addr=127.0.0.1:80 embassyd_tor_key=TorSecretKey(****) tor_control=127.0.0.1:9051 import_root_ca=None}: embassy::net::mdns: AvahiCallback: ClientState = AVAHI_CLIENT_S_REGISTERING
Feb 11 19:41:20 embassy-2ef08502 embassyd[1796]: Feb 11 19:41:20.789  WARN init{embassyd_addr=127.0.0.1:80 embassyd_tor_key=TorSecretKey(****) tor_control=127.0.0.1:9051 import_root_ca=None}: embassy::net::mdns: AvahiCallback: EntryGroupState = AVAHI_ENTRY_GROUP_UNCOMMITED

ProofOfKeags avatar Feb 11 '22 19:02 ProofOfKeags

@chrisguida have you seen this at all recently?

ProofOfKeags avatar Jun 07 '22 01:06 ProofOfKeags

Nope, .local is very reliable now

chrisguida avatar Jun 07 '22 22:06 chrisguida

I've seen this issue crop up more frequently recently. Let's leave it open in case people can provide more data to help us track it down.

chrisguida avatar Jul 07 '22 14:07 chrisguida

If on Windows, this is almost certainly Bonjour. This was the case in 2 recent support calls. If on other OSes, I have no idea

kn0wmad avatar Jul 08 '22 03:07 kn0wmad

This is for the server side issue, not client side.

chrisguida avatar Jul 08 '22 14:07 chrisguida

I've been able to reliably recreate this with an alpha version of btcpay.

  1. turn on btcpay
  2. it goes into a crash loop (because it's broken)
  3. all .locals go down
  4. stop btcpay
  5. all .locals are still down
  6. only system restart embassyd fixes.

chrisguida avatar Jul 29 '22 15:07 chrisguida

Maybe we can detect when system .locals have been down for a while and restart the avahi processes?

chrisguida avatar Jul 29 '22 15:07 chrisguida

Did Tor interfaces go down as well?

MattDHill avatar Jul 29 '22 18:07 MattDHill

@MattDHill I don't remember, I believe I tried one .onion and it was also down

chrisguida avatar Jul 30 '22 01:07 chrisguida

I ask because it seems likely that the .locals are going down for the same reason we've seen .onions go down: network thrash resulting from a service caught in a rapid crash loop.

MattDHill avatar Jul 30 '22 02:07 MattDHill

Right, the difference is that the .locals don't come back up after the crash loop is resolved.

chrisguida avatar Jul 30 '22 12:07 chrisguida

I was able to reproduce the same on latest BTCPayServer from Alpha (witch fails in the crash loop). All services are not accessible over .local (.onion addresses work just fine) Your embassy-q1w2e3.local address is accessible as well.

This is just a sample peace of embassyd log (is flooded massively)

Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.608780Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "rhnkvbvsmmlskcvs3mx>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.608983Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for tz>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.614531Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "tzctpa7lyk35vzuqcco>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.614733Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for pl>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.619701Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "plrtwednvxssjbdeer3>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.619900Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for wz>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.624832Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "wzlkvhwfe7pro3qqkzx>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.625030Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for a2>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.630051Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "a232ncr7jexk2chvuba>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.630243Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for v2>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.635133Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "v27km6l67yadp477jju>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.635329Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for ne>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.640666Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "ne4ljwgc5xjyyf6yzwy>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.640866Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for yx>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.645607Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "yxtgpbjhafvrrf3jsks>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.645833Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for ft>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.650749Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "ftbsykrc5zhnnt73okb>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.650949Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for vc>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.656218Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "vc4bdmvu4k3na3ylb7w>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.656411Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for y2>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.661169Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "y2ttxnjzput3zraudrl>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.661364Z DEBUG run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for ty>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.666353Z  INFO run_main:remove{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "tykhhxfctwj4hacc6ff>
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.807636Z  INFO run_main: embassy::net: Generating SSL Certificate mountpoints for btcpayserver
Aug 01 23:12:08 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:08.813912Z  WARN execute{self=Docker(DockerProcedure { image: ImageId(Id("main")), system: false, entrypoint: "docker_entrypoint.sh", arg>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.941724Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Loading services for mDNS
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.951540Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "embassy-ababe988.local"
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.951726Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for zrvxb>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.956407Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "zrvxbplfqfxtuusyk6yqjd>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.956593Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for 7bvqg>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.961208Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "7bvqge7usek5cbzsc44cwh>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.961389Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for t25td>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.966865Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "t25td3eyj6mo25wl65kvbd>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.967057Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for m6sgp>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.971567Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "m6sgp2nmo64b36eyzwswbu>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.971752Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for x6doe>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.976587Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "x6doec4q3ghqbipceyfdth>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.976783Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for jpqoz>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.981538Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "jpqoz3bz2ok27yw3yojelc>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.981726Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for jhexu>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.986542Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "jhexuh6u34uwbaj6c46fp4>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.986725Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for uzp7p>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.992069Z  INFO run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Published "uzp7pjmzrh2yr3z3j3klhf>
Aug 01 23:12:09 embassy-ababe988 embassyd[1342]: 2022-08-01T23:12:09.992283Z DEBUG run_main:add{pkg_id=PackageId(Id("btcpayserver")) ip=172.18.0.19}: embassy::net::mdns: Adding mdns CNAME entry for 2zcxr>

k0gen avatar Aug 02 '22 18:08 k0gen

probably fixed by #1729

dr-bonez avatar Sep 08 '22 16:09 dr-bonez