Update tdex to v1.0.0

[altafan]

This updates TDEX from v0.9.1 to v1.0.0.

The compose file defines a brand new oceand service required by the new version of tdexd v1.0.0. Updated also the dashboard to v1.0.1.

[altafan]

Thanks for this update @altafan! I have left some suggestions below for you to address before I test.

Also, are you okay if I push a commit directly to your update-tdex branch? We need to add a pre-start hook to make sure that users who already have tdex installed receive the new ocean-data directory with appropriate ownership when they update.

Thank you @nmfretz for the review!

I guess I have to give you access to my repo in order to directly push changes? Can you wait until all required changes are fixed?

[nmfretz]

I guess I have to give you access to my repo in order to directly push changes? Can you wait until all required changes are fixed?

Actually, here's what I think will be needed. You can add this above the Tor Hidden Service logic in the existing pre-start.

set -euo pipefail

APP_DATA_DIR="$(readlink -f $(dirname "${BASH_SOURCE[0]}")/..)"
OCEAN_DATA_DIR="${APP_DATA_DIR}/ocean-data"

[ ! -d "${OCEAN_DATA_DIR}" ] && mkdir -p "${OCEAN_DATA_DIR}" && chown 1000:1000 "${OCEAN_DATA_DIR}"

[nmfretz]

Hey @altafan can you please update the exposed daemon port to 9002: https://github.com/getumbrel/umbrel-apps/pull/754#discussion_r1335746250

Also, what would be the best way for us to test external connection in the tdex ecosystem?

[altafan]

Also, what would be the best way for us to test external connection in the tdex ecosystem?

@nmfretz I made a test on a remote host, and once I installed the tdex app, I just went to <ip>:9090 to see a page similar to the following one (the url might be different):

[tiero]

Any news? @nmfretz

[nmfretz]

oh sorry, @tiero and @altafan can you guys push the quick fix noted here: https://github.com/getumbrel/umbrel-apps/pull/754#discussion_r1326207517. Port 9090 clashes with an existing app. 9092 is free for example.

Also, since this is an update, we can now add release notes to the umbrel-app.yml file. See Lightning app as an example: https://github.com/getumbrel/umbrel-apps/blob/776587169cba9e345fcbc9e54f82675fd10e9eb2/lightning/umbrel-app.yml#L24-L28

[altafan]

@nmfretz i pushed the changes. Now the port is 9092. I also followed your suggestion and added some release notes, thanks 🙏.

[nmfretz]

Thanks @altafan. I have made a few changes and we are nearly there:

• in your recent commit 19e9840, you changed the port in the manifest to 9092. This port is actually for your dashboard UI, so should stay as 9094. I have changed that here a5bac52
• updated the Caddyfile so that new installs start with the correct tdexd port (9092) 2f61879
• added logic to the pre-start hook so that existing installs that are updating to this new version receive the change to the caddyfile f566598

I tested on a new install and it works well 👌. However, something breaks in the tdexd container when updating the app from the previous version. Here is the error I get from the tdexd container

$ sudo docker logs tdex_tdexd_1
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] Deleting empty file: /home/tdex/.tdex-daemon/db/feeder/000007.vlog
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] Deleting empty file: /home/tdex/.tdex-daemon/db/markets/000007.vlog
FATA[0000] failed to initialize grpc service             error="invalid opts: invalid app config: opening prices db: manifest has unsupported version: 7 (we support 8).\nPlease see https://github.com/dgraph-io/badger/blob/master/README.md#i-see-manifest-has-unsupported-version-x-we-support-y-error on how to fix this."

Could you please take a look into this? Once this is fixed we can go live!

[altafan]

@nmfretz sorry for the late reply

However, something breaks in the tdexd container when updating the app from the previous version. Here is the error I get from the tdexd container

The reason is that the database of the prev version of tdexd must be migrated in order to be compatible with the new version. The migration tool is packed within the official docker image, do you think it's possible to setup a migration script that runs it if needed to solve the issue?

Otherwise, it could just be enough to delete the existing datadir, the daemon will create a new one at first startup and the user will restore the wallet, which seems an easier solution.

[altafan]

@nmfretz let's go for the easiest way and eventually remove the datadir if existing. Can you tell me if I can add this conditional operation to some hook script?

[tiero]

@smolgrrr any inputs here on what @altafan mentioned?

[nmfretz]

@tiero @altafan sorry for the delay.

Otherwise, it could just be enough to delete the existing datadir, the daemon will create a new one at first startup and the user will restore the wallet, which seems an easier solution.

For this option we'd run the risk of users clicking update without reading the release notes, so there could be a subset of users who update and don't have (or have lost) their recovery phrase, which would be pretty bad I think.

Additionally we'd have to make sure that this database deletion logic doesn't run on subsequent app starts/restarts/updates, which is totally possible but adds complexity.

The migration tool is packed within the official docker image, do you think it's possible to setup a migration script that runs it if needed to solve the issue?

Yes, we can do this with a pre-start script. The benefit here is that we users won't accidently lose access to their funds.

We could do something like this:

• before starting the app, check if tdex-data dir has anything in it. If it doesn't then this is a new app install. We create a file in the data directory called something like HAS_MIGRATED and then exit the script and allow the app to start.
• If tdex-data has data then check if the file called HAS_MIGRATED exists:
  • If it HAS_MIGRATED exists then we exit the script and allow the app to start normally. This will ensure that subsequent app restarts/updates don't nuke the database again.
  • If HAS_MIGRATED doesn't exist then we know that we need to migrate. We start the required container(s) for the migration tool and run the necessary commands to migrate. Then we create the HAS_MIGRATED file and exit the script so that the app starts up.

I can write this script on priority for you guys and test it out, I would just need guidance on the commands to run for migration. If I don't answer here, please feel free to DM me on X, Nostr, or Telegram!

[altafan]

@nmfretz the migration is an interactive process that requires the user to enter the password to unlock the data stored in the db. Can this be a problem for the workflow?

[tiero]

Guys can we merge this? People are still using the old deprecated version.

Let's override the old data directory with bash?

[tiero]

Would maybe make an additional App ie TDEX V1 and deprecate this one be better? So we assume user do not expect a migration being a full restore

[nmfretz]

Guys can we merge this? People are still using the old deprecated version.

Let's override the old data directory with bash?

Would maybe make an additional App ie TDEX V1 and deprecate this one be better? So we assume user do not expect a migration being a full restore

Hey @tiero. Alrighty, let's get this done. Without an automatic migration process on update, let's go with the following:

1. A warning at the top of the release notes explaining to users that this update will require them to restore the wallet from their seed phrase.

2. For existing installs we will keep the old data directory around so that in a worst-case scenario they can use this old database to restore using an older version of tdex. The way we can do this is just by simply changing the data directory bind mount on the host from say:

${APP_DATA_DIR}/tdex-data:/home/tdex/.tdex-daemon

to:

${APP_DATA_DIR}/tdexd:/home/tdex/.tdex-daemon

Let me make those changes and push them here so we can all test.

[nmfretz]

@tiero @altafan, here is the warning I have added to the top of the release notes. Does this look okay to you?

We could also include a link to your Telegram channel and say something like:

If you encounter any issues when recovering your wallet after update, please reach out to the official TDEX Telegram group: https://t.me/tdexnetwork

If any users reach out with issues after updating, their old tdex daemon data directory will be located at: /home/umbrel/umbrel/app-data/tdex/tdex-data

The new tdex daemon data directory is located at: /home/umbrel/umbrel/app-data/tdex/tdexd

---

Can you please take a look at this issue and see if you can recreate:

I just tested updating TDEX from v0.9.1 to v1.0.1. I went through the wallet recovery process and see this error in the UI:

I see this in the console:

But nothing in the networking tab, and nothing related in the container logs:

$ sudo docker logs -f tdex_tdexd_1
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] starting daemon
INFO[0000] wallet interface is listening on :9092
DEBU[0008] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0008] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0016] /tdex_daemon.v2.WalletService/UnlockWallet
DEBU[0021] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0021] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0021] stopped server
INFO[0021] wallet interface is listening on :9092
INFO[0021] operator interface is listening on :9092
INFO[0021] trade interface is listening on :9092
DEBU[0044] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0044] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0063] /tdex_daemon.v2.WalletService/GetStatus
DEBU[0063] /tdex_daemon.v2.WalletService/GetInfo
DEBU[0600] Total allocated: 0.852GB, Heap allocated: 0.485GB, Allocated objects count: 1725134, Freed objects count: 1431569
DEBU[0600] Num of go routines: 102

$ sudo docker logs -f tdex_oceand_1
INFO[0000] All 1 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 4
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] All 0 tables opened in 0s
INFO[0000] Discard stats nextEmptySlot: 0
INFO[0000] Set nextTxnTs to 0
INFO[0000] service: registered wallet handler on public interface
INFO[0000] service: registered account handler on public interface
INFO[0000] service: registered transaction handler on public interface
INFO[0000] service: registered notification handler on public interface
DEBU[0000] scanner: start listening to messages from electrum server
INFO[0000] service: started blockchain scanner
INFO[0000] service: start listening on :18000
DEBU[0000] gRPC method: /ocean.v1.WalletService/Status
DEBU[0000] gRPC method: /ocean.v1.NotificationService/UtxosNotifications
DEBU[0000] gRPC method: /ocean.v1.NotificationService/TransactionNotifications
DEBU[0000] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0009] gRPC method: /ocean.v1.WalletService/Status
DEBU[0009] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0016] gRPC method: /ocean.v1.WalletService/Unlock
DEBU[0021] wallet repository: publish event WalletUnlocked
DEBU[0021] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0021] gRPC method: /ocean.v1.WalletService/Status
DEBU[0022] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0044] gRPC method: /ocean.v1.WalletService/Status
DEBU[0044] gRPC method: /ocean.v1.WalletService/GetInfo
DEBU[0064] gRPC method: /ocean.v1.WalletService/Status
DEBU[0064] gRPC method: /ocean.v1.WalletService/GetInfo

sudo docker logs -f tdex_dashboard_1
{"level":"info","ts":1718688563.9792867,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","ts":1718688563.9834294,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1718688563.9839225,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000764180"}
{"level":"info","ts":1718688563.9841962,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","ts":1718688563.9845257,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1718688563.9845374,"msg":"serving initial configuration"}
{"level":"info","ts":1718688563.984608,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
{"level":"info","ts":1718688563.984663,"logger":"tls","msg":"finished cleaning storage units"}

On fresh install I see a different error:

[nmfretz]

Hey @altafan @tiero just pinging again in case you missed this comment here: https://github.com/getumbrel/umbrel-apps/pull/754#issuecomment-2175075416

[tiero]

@altafan @janaka-steph Can you please double-check this when you got time? To test with latest version (without involving migration of db)

[nmfretz]

@altafan @Janaka-Steph @tiero, just pinging in case this has fallen off your radar.

[Janaka-Steph]

I just tried to run tdex 1.0.1 in multipass VM. I encountered the same error "verification failed: signature mismatch after caveat verification". But after refreshing the page I was able to access the Tdex provider and load the wallet. Feel free to merge, it is a minor issue.

[nmfretz]

I just tried to run tdex 1.0.1 in multipass VM. I encountered the same error "verification failed: signature mismatch after caveat verification". But after refreshing the page I was able to access the Tdex provider and load the wallet. Feel free to merge, it is a minor issue.

Thanks @Janaka-Steph. Did you also try a fresh install and wallet creation where I encounter this:

If you are comfortable moving forward with both issues then we can merge.

[nmfretz]

@Janaka-Steph @tiero pinging again for the above. Thanks.

[Janaka-Steph]

@nmfretz I just tried and I didn't get error. It should have been fixed. Ok for merge!

[nmfretz]

Thanks @Janaka-Steph! Clearing local storage has gotten rid of this issue for me FYI, in case anyone reaches out with a similar issue. Going live.

[github-actions[bot]]

🎉   Linting finished with no errors or warnings   🎉

Thank you for your submission! This is an automated linter that checks for common issues in pull requests to the Umbrel App Store.

Please review the linting results below and make any necessary changes to your submission.

Linting Results

Severity	File	Description
ℹ️	tdex/docker-compose.yml	Mounted file/directory "/tdex/torrc" doesn't exist: The volume "${APP_DATA_DIR}/torrc:/etc/tor/torrc:ro" tries to mount the file/directory "/tdex/torrc", but it is not present. This can lead to permission errors!
ℹ️	tdex/docker-compose.yml	External port mapping "${APP_TDEX_PORT}:${APP_TDEX_PORT}": Port mappings may be unnecessary for the app to function correctly. Docker's internal DNS resolves container names to IP addresses within the same network. External access to the web interface is handled by the app_proxy container. Port mappings are only needed if external access is required to a port not proxied by the app_proxy, or if an app needs to expose multiple ports for its functionality (e.g., DHCP, DNS, P2P, etc.).
ℹ️	tdex/docker-compose.yml	Potentially using unsafe user in service "oceand": The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.
ℹ️	tdex/docker-compose.yml	Potentially using unsafe user in service "tdexd": The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.
ℹ️	tdex/docker-compose.yml	Potentially using unsafe user in service "dashboard": The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.
ℹ️	tdex/docker-compose.yml	Potentially using unsafe user in service "caddy": The default container user "root" can lead to security vulnerabilities. If you are using the root user, please try to specify a different user (e.g. "1000:1000") in the compose file or try to set the UID/PUID and GID/PGID environment variables to 1000.

Legend

Symbol	Description
❌	Error: This must be resolved before this PR can be merged.
⚠️	Warning: This is highly encouraged to be resolved, but is not strictly mandatory.
ℹ️	Info: This is just for your information.