feat: zeronsd support

[mcondarelli]

Feature Request

Describe the Feature Request

Add to the integrated Docker image (actually a compose set of images) also a zeronsd image to handle name serving on networks.

This would allow creation of self-contained server with everything needed to run a private network solution.

Describe Preferred Solution

In general I would like better a really integrated solution (a single image to run, not multiple images) and in this case more so because having separate images would mean run multiple instances of zerosnd container as it serves a single network.

Having a single container breaks a little Docker concept, makes it easier to deploy (e.g.: on "smart" NAS) and also makes possible to run multiple instances of a single process (i.e.: zeronsd) without associated overhead of running multiple containers.

The main user interface should (possibly under a checkbox control) launch a zeronsd instance each time a network is created.

Describe Alternatives

It is also possible to keep current structure and have a single container for zerosnd launching multiple instances of daemon under zero-ui conrol, but it seems harder to implement properly.

If the feature request is approved, would you be willing to submit a PR? Yes, but my time is currently very limited and I will surely need assistance with the UI part as I'm not proficient in javascript.

[dec0dOS]

Hello, @mcondarelli, thanks for detail description of the feature request.

I've been already considering about DNS support in ZeroUI, but was focused on the core part, there are a lot of features to do and bugs to fix. It would be great if you could help with zeronds integration.

[mcondarelli]

I started building an all-in-one Docker image (I'll do a proper clone ASAP). First comment is ZeroUI does not seem very robust on restart. I don't really know how relevant this is since Controller is not supposed to be restarted often, but:

1. Restarting controller without a previous "logout" puts controller in a strange state where it does not ask for login, and apparently UI is working, but it does not show anything; an explicit logout cures problem.
2. If clients are up-and running they continue to remain connected even while Controller is down but they will show as "OFFLINE" once controller is restarted; I did not find a way to solve this without explicit reboot (or leave/join) on clients.

I'll keep you posted on progress.

[mcondarelli]

Apparently ZeroNSd relies on ZeroTier Central RESTful API to get info about the network. At first glance it seems it uses just one call: GET /api/network/{networkId}. Is there any easy way to implement such a interface in ZeroUI? I would rather not touch ZeroNSd at all, if at all possible. Making it believe it's talking to a ZT-Central would make things much easier now and also in the future as ZeroNSd is still under heavy development.

[dec0dOS]

ZeroUI has ZeroTier Central compatible API (but not for user management, for now). Please check the ZeroUI README.md and ZeroTier Central API for more info.

[mcondarelli]

I've seen that, but I didn't find a way to get (and check against) ZeroTier Central API token (the one needed in "Token authentication is accomplished by sending the following header: Authorization: bearer <API token>"). Is such feature implemented? Should I just generate a random token to keep ZeroNSd happy? Sorry being so dense, but I'm really not a web expert :(

[dec0dOS]

Please refer to this discussion. No problem, keep asking and thanks for your contribution!

[mcondarelli]

I'm having problems so I tried to install and build on my devel machine (instead of building in a Docker container). Unfortunately something is very wrong on my setup:

mcon@cinderella:/tmp$ git clone https://github.com/dec0dOS/zero-ui.git
Cloning into 'zero-ui'...
remote: Enumerating objects: 686, done.
remote: Counting objects: 100% (686/686), done.
remote: Compressing objects: 100% (391/391), done.
remote: Total 686 (delta 326), reused 589 (delta 239), pack-reused 0
Receiving objects: 100% (686/686), 2.35 MiB | 4.25 MiB/s, done.
Resolving deltas: 100% (326/326), done.
mcon@cinderella:/tmp$ cd zero-ui/
mcon@cinderella:/tmp/zero-ui$ yarn install
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module 'worker_threads'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:692:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at /tmp/zero-ui/.yarn/releases/yarn-berry.cjs:289:2658
    at Object.<anonymous> (/tmp/zero-ui/.yarn/releases/yarn-berry.cjs:586:2636)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
mcon@cinderella:/tmp/zero-ui$ 

What am I doing so wrong?

[dec0dOS]

Maybe your node.js version is too old. Please try to upgrade to the latest LTS version.

[mcondarelli]

I am kind of trapped in an chicken-and-egg situation. I am trying to produce an all-in-one container with ZeroTier-One, ZeroUI and ZeroNSd.

Unfortunately ZeroNSd seems not to like alpine/musl (segfault; i reported it, but I'm unsure if/hen it will be fixed), so I switched to debian-bullseye/glibc, but here ZeroUI doesn't seem to display anything (blank page, content is a single line: <!doctype html><html lang="en"><head><meta charset="utf-8"/><base href="/app/"><link rel="icon" href="/app/favicon.ico"/><meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1,user-scalable=no"/><meta name="description" content="ZeroUI"/><link rel="manifest" href="/app/manifest.json"/><title>ZeroUI</title><link href="/app/static/css/8.bf512090.chunk.css" rel="stylesheet"><link href="/app/static/css/main.52e70a0d.chunk.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script src="/app/static/js/runtime-main.1e458087.js"></script><script src="/app/static/js/8.dd99f694.chunk.js"></script><script src="/app/static/js/main.a25a804b.chunk.js"></script></body></html> so I guess something is missing, but I'm out of my depth here).

You can find my current fork here. Any hint appreciated.

[dec0dOS]

Please check the browser console error log and the ZU_SECURE_HEADERS flag.

[mcondarelli]

Error log (Firefox) say:

Content Security Policy: Couldn’t process unknown directive ‘script-src-attr’
Loading failed for the <script> with source “http://172.17.0.2:4000/app/static/js/runtime-main.1e458087.js”. app:1:1
Loading failed for the <script> with source “http://172.17.0.2:4000/app/static/js/8.dd99f694.chunk.js”. app:1:1
Loading failed for the <script> with source “http://172.17.0.2:4000/app/static/js/main.a25a804b.chunk.js”. app:1:1

Chrome is slightly more informative:

Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR
main.52e70a0d.chunk.css:1 Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR
runtime-main.1e458087.js:1 Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR
8.dd99f694.chunk.js:1 Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR
main.a25a804b.chunk.js:1 Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR
:4000/app/favicon.ico:1 Failed to load resource: net::ERR_SSL_PROTOCOL_ERROR

I have:

# ---- copy Zero-UI
WORKDIR /app/frontend/build
COPY --from=builder /app/frontend/build /app/frontend/build/

WORKDIR /app/backend
COPY ./backend/package*.json /app/backend
COPY ./backend/yarn.lock /app/backend
RUN yarn install

COPY ./backend /app/backend

EXPOSE 4000
ENV NODE_ENV=production
ENV ZU_SECURE_HEADERS=true
ENV ZU_SERVE_FRONTEND=true

in Dockerfile.

[dec0dOS]

Set the ZU_SECURE_HEADERS to false. It may not work without HTTPS

[mcondarelli]

I did preliminary work and started to actually try to use ZeroNSd redirecting it to zero-ui backend.

First incompatibility arises from different path prefix: ZeroTier Central uses ".../api/v1/" while in your code it is just ".../api/". Would it break something to add this "v1" to path prefix?

Second and potentially blocking incompatibility is missing config/dns stanza in ".../network/" response. I assume I will need to implement it, but I would like some guidance there as I have no previous knowledge of Node.js and I would like to avoid "stupid mistakes". I already implemented the UI part (incomplete, but sufficient); I need to modify backend to include it in the answer.

[dec0dOS]

Hmm, it seems that ZeroTier has changed their API endpoint from /api/ to /api/v1. The old one is presented in the outdated documentation, and the new one introduces the v1 prefix. It shouldn't break anything in ZeroUI, but the frontend API base url should be changed as well. It may be the problem with custom scripts that uses the ZeroUI API.

Feel free to ask for help with implementation details.

[mcondarelli]

Sorry for the late comeback. I've been very busy in the meantime (and I'm still short on time). I need to add the "config": { "dns": {"domain": "some.domain", "servers": [ "10.0.0.3" ] } } stanza (and possibly others) to the GET https://my.zerotier.com/api/v1/network/{networkID} answer. I already added the relevant widgets, but I'm unsure how to structure code in backend/services/network.js to provide what's needed. Any help would be welcome to shorten time needed to understand the code. TiA

[dec0dOS]

Hello, @mcondarelli. I've found your issue in the zeronds repo and could help you to understand the architecture of ZeroUI.

ZeroUI splits into two parts: the backend and the frontend. The backend task is to "emulate" the ZeroTier Central API, so the tools like zeronds or any other that uses the ZeroTier Central API could communicate with the self-hosted ZeroTier controller. The frontend is made to follow the ZeroTier Central API spec, so you could even use the ZeroTier Central backend with a custom frontend from ZeroUI.

Some configuration on the backend should be written in the database for persistence because the controller itself does not store the information about the node names, for example.

[mcondarelli]

Thanks. I had divined that much. I seem to understand the background task essentially receives a data stream from foreground and updates some internal data. Background also listens for ZTC-like requests and fulfills them from internal data. What it's unclear to me is how the actual reply (JSON) formatting is done; apparently most of it is done just inserting things in "internal data" in the "right place"; can you clarify this, please? If you want I can publish my current code so you can se what I'm trying to do.

[dec0dOS]

The backend/services/network.js file is used to handle additional data on the backend. So you could add the DNS entry here. Sorry, I couldn't dig deeper on how to better implement the DNS feature for now.

I've checked your current changes in the repo, thanks for your contributions! By the way, I think that the primary solution to deploy the ZeroUI should be the multi-image one. It provides greater flexibility, so the controller itself could run outside the docker and rebuilding the image for multiple architectures after every release would take too much time, I don't even think that GitHub Actions could handle it. So I think there is a way to figure out how to run zeronds as a separate image/container.
The all-in-one container could be the alternative way to deploy ZeroUI.

[mcondarelli]

Sorry for the late comeback. I just pushed a NOT-WORKING copy of my current development.

Please check the throwaway branch.

Problem is this implements response to "http://localhost:3000/api/network/..." as required (I think, to be tested), but breaks frontend ("Domain", at least, disappears).

As said this is first nodejs program I see and I don't know which is the "Right Way" to handle this. I suspect problem lies in "delete" statements and attempted an object copy, but failed.

A bit of help would be appreciated ;)

[dec0dOS]

Hey, @mcondarelli! Thanks for your commitment! It may require some time to dig into zeronsd and your code. At first sight, it better to start from the backend and test the API from the command line (using curl for example). Does the backend part works properly? I couldn't figure out what hardcode pippo means in the servers key.

If I understand correctly, the ZeroTier controller node should join the created network itself to make the DNS server accessible over ZT network.

[mcondarelli]

Thanks for the fast answer.

The string "pippo" (Italian equivalent of "foo") is a leftover to see if I was formatting the right things; I already removed it in my current code.

I am testing backend via curl, exactly as you suggested. That part seems to be correct (i.e.: it formats things how I wanted, I'll have to see if ZeroNSd is happy with it).

I also started modifying frontend to be able to input needed info and it seemed to be working before last commit (the one where ZeroNSd complains about missing dns: {...} stanza). With latest mods typing something in "Domain" is correctly reported by backend, but it disappears in frontend upon refresh.

AFAIK ZeroNSd should run on a node joined to the Network it controls. It is not important ZeroUI, Controller and ZeroNSd run on the same host; I merely did that for ease of debugging. I am currently developing on a Debian Sid Virtual Machine (VirtualBox) because I had several problems developing on my host (Linux Mint 20.2).

[dec0dOS]

I've been thinking for a while about https://github.com/zerotier/zeronsd/issues/107. It seems that the ZeroTier team does not want to provide a custom URL argument to support enterprise ZeroTier Central self-hosted edition or made by the community self-hosted one like ZeroUI. The URL to my.zerotier.com is hardcoded in ZeroTier Central Rust API. Supporting the fork is not a great option, I think it's better to create the entry in the /etc/hosts to make the API think that it speaks with ZeroTier Central. Something like 127.0.0.1 my.zerotier.com is more than enough, I suppose.

[mcondarelli]

Actually my solution was quite simple: since I recompile ZeroNSd from source in Dockerfile I can add a patch:

diff --git a/src/utils.rs b/src/utils.rs
index 0f62d52..e3c523f 100644
--- a/src/utils.rs
+++ b/src/utils.rs
@@ -21,6 +21,7 @@ pub(crate) fn central_config(token: String) -> Configuration {
     let mut config = Configuration::default();
     config.user_agent = Some(version());
     config.bearer_access_token = Some(token);
+    config.base_path = "http://localhost:3000/api".to_string();
     return config;
 }
 

[mcondarelli]

I did some more work on the same branch as before. This seems to be almost working. Feedback to allow clients to use ZeroNSd for name resolution is still missing but direct query via dig seems to work. I won't be able to work on this till next weekend, but I would like to have some feedback.

[dec0dOS]

Actually my solution was quite simple: since I recompile ZeroNSd from source in Dockerfile I can add a patch:

diff --git a/src/utils.rs b/src/utils.rs
index 0f62d52..e3c523f 100644
--- a/src/utils.rs
+++ b/src/utils.rs
@@ -21,6 +21,7 @@ pub(crate) fn central_config(token: String) -> Configuration {
     let mut config = Configuration::default();
     config.user_agent = Some(version());
     config.bearer_access_token = Some(token);
+    config.base_path = "http://localhost:3000/api".to_string();
     return config;
 }
 

That's a great solution! Nevertheless, I think it's better to add the entry in /etc/hosts file in the Docker image, because it is much more resilient to any changes and refactoring in the ZeroTier Central Rust API, or the ZeroNSd itself.

I did some more work on the same branch as before. This seems to be almost working. Feedback to allow clients to use ZeroNSd for name resolution is still missing but direct query via dig seems to work. I won't be able to work on this till next weekend, but I would like to have some feedback.

I could not find new changes. Did you push them?

[mcondarelli]

I could not find new changes. Did you push them?

Oops! Sorry. It should be on github now.

[mcondarelli]

That's a great solution! Nevertheless, I think it's better to add the entry in /etc/hosts file in the Docker image, because it is much more resilient to any changes and refactoring in the ZeroTier Central Rust API, or the ZeroNSd itself.

I opted for the patch to be able to modify also the path part of the URL. If you plan to move from /api/ to /api/v1/ then adding a stanza to /etc/hosts is better.

[mcondarelli]

Did you have any chance to check my work? I would like to have your comments before proceeding. TiA!

[dec0dOS]

Hello, @mcondarelli! Thanks again for your work, I'm really busy now, but I'd try to review your commits ASAP. Please open the pull request, It would be easier to propose and review the changes.

[gjpin]

I've been thinking for a while about zerotier/zeronsd#107. It seems that the ZeroTier team does not want to provide a custom URL argument to support enterprise ZeroTier Central self-hosted edition or made by the community self-hosted one like ZeroUI. The URL to my.zerotier.com is hardcoded in ZeroTier Central Rust API. Supporting the fork is not a great option, I think it's better to create the entry in the /etc/hosts to make the API think that it speaks with ZeroTier Central. Something like 127.0.0.1 my.zerotier.com is more than enough, I suppose.

It seems that since v0.2.5 it's possible to provide a custom URL argument through the env var ZEROTIER_CENTRAL_INSTANCE