status-go icon indicating copy to clipboard operation
status-go copied to clipboard

chore_: create v2 endpoints for status.go to solve EndpointsUnsupported in endpoints.go

Open qfrank opened this issue 1 year ago • 14 comments

I have a dream that the status-mobile can use the status-backend via an HTTP server so that the status-backend can run separately. Now the dream is getting closer. Only the update in status-mobile is left.

It would be great if you (@ilmotta) could help with the frontend!

relate status-go PR relate comment

qfrank avatar Oct 14 '24 13:10 qfrank

Jenkins Builds

Click to see older builds (105)
:grey_question: Commit :hash: Finished (UTC) Duration Platform Result
:heavy_multiplication_x: 4d955ff4 #1 2024-10-14 13:26:26 ~3 min tests :page_facing_up:log
:heavy_check_mark: 4d955ff4 #1 2024-10-14 13:27:56 ~4 min ios :package:zip
:heavy_check_mark: 4d955ff4 #1 2024-10-14 13:28:05 ~5 min linux :package:zip
:heavy_check_mark: 4d955ff4 #1 2024-10-14 13:28:35 ~5 min android :package:aar
:heavy_check_mark: 4d955ff4 #1 2024-10-14 13:28:44 ~5 min tests-rpc :page_facing_up:log
:heavy_check_mark: 852695a0 #2 2024-10-14 13:36:49 ~2 min android :package:aar
:heavy_check_mark: 852695a0 #2 2024-10-14 13:37:26 ~2 min linux :package:zip
:heavy_check_mark: 852695a0 #2 2024-10-14 13:37:59 ~3 min ios :package:zip
:heavy_check_mark: 852695a0 #2 2024-10-14 13:39:44 ~5 min tests-rpc :page_facing_up:log
:heavy_check_mark: 852695a0 #2 2024-10-14 14:06:38 ~31 min tests :page_facing_up:log
:heavy_check_mark: 381f29a8 #3 2024-10-17 03:16:58 ~2 min android :package:aar
:heavy_check_mark: 381f29a8 #3 2024-10-17 03:17:30 ~2 min linux :package:zip
:heavy_check_mark: 381f29a8 #3 2024-10-17 03:17:45 ~3 min ios :package:zip
:heavy_check_mark: 381f29a8 #3 2024-10-17 03:19:58 ~5 min tests-rpc :page_facing_up:log
:heavy_check_mark: 381f29a8 #3 2024-10-17 03:46:37 ~31 min tests :page_facing_up:log
:x: 750ea132 #4 2024-10-17 08:59:29 ~43 sec ios :page_facing_up:log
:x: 750ea132 #4 2024-10-17 08:59:38 ~54 sec android :page_facing_up:log
:x: 750ea132 #4 2024-10-17 08:59:46 ~59 sec linux :page_facing_up:log
:heavy_multiplication_x: 750ea132 #4 2024-10-17 09:00:42 ~1 min tests :page_facing_up:log
:heavy_multiplication_x: 750ea132 #4 2024-10-17 09:00:50 ~1 min tests-rpc :page_facing_up:log
:heavy_check_mark: faa1ece8 #5 2024-10-17 09:21:40 ~2 min android :package:aar
:heavy_check_mark: faa1ece8 #5 2024-10-17 09:22:07 ~2 min linux :package:zip
:heavy_check_mark: faa1ece8 #5 2024-10-17 09:22:32 ~2 min ios :package:zip
:heavy_check_mark: faa1ece8 #5 2024-10-17 09:25:08 ~5 min tests-rpc :page_facing_up:log
:heavy_check_mark: faa1ece8 #5 2024-10-17 09:51:31 ~31 min tests :page_facing_up:log
:heavy_check_mark: 7c37f6cd #6 2024-10-17 10:39:58 ~2 min android :package:aar
:heavy_check_mark: 7c37f6cd #6 2024-10-17 10:40:23 ~2 min linux :package:zip
:heavy_check_mark: 7c37f6cd #6 2024-10-17 10:40:43 ~3 min ios :package:zip
:heavy_check_mark: 7c37f6cd #6 2024-10-17 10:42:45 ~5 min tests-rpc :page_facing_up:log
:heavy_check_mark: 7c37f6cd #6 2024-10-17 11:09:31 ~31 min tests :page_facing_up:log
:x: 8c6121c9 #7 2024-10-20 11:49:27 ~11 sec linux :page_facing_up:log
:heavy_multiplication_x: 8c6121c9 #7 2024-10-20 11:51:31 ~2 min tests :page_facing_up:log
:heavy_check_mark: 8c6121c9 #7 2024-10-20 11:51:36 ~2 min android :package:aar
:heavy_check_mark: 8c6121c9 #7 2024-10-20 11:52:15 ~3 min ios :package:zip
:heavy_check_mark: 8c6121c9 #7 2024-10-20 11:54:27 ~5 min tests-rpc :page_facing_up:log
:x: d23d34d6 #8 2024-10-20 11:53:37 ~10 sec linux :page_facing_up:log
:heavy_check_mark: d23d34d6 #8 2024-10-20 11:55:04 ~1 min android :package:aar
:heavy_check_mark: d23d34d6 #8 2024-10-20 11:56:03 ~2 min ios :package:zip
:heavy_check_mark: d23d34d6 #8 2024-10-20 11:59:36 ~5 min tests-rpc :page_facing_up:log
:x: d23d34d6 #9 2024-10-20 12:08:53 ~12 sec linux :page_facing_up:log
:heavy_multiplication_x: d23d34d6 #8 2024-10-20 12:25:04 ~31 min tests :page_facing_up:log
:heavy_multiplication_x: d23d34d6 #9 2024-10-21 01:31:25 ~4 min tests :page_facing_up:log
:heavy_check_mark: a62bc7ee #9 2024-10-21 02:46:17 ~2 min android :package:aar
:heavy_check_mark: a62bc7ee #9 2024-10-21 02:47:04 ~3 min ios :package:zip
:heavy_check_mark: a62bc7ee #10 2024-10-21 02:48:41 ~4 min linux :package:zip
:heavy_check_mark: a62bc7ee #9 2024-10-21 02:49:11 ~5 min tests-rpc :page_facing_up:log
:heavy_check_mark: a62bc7ee #10 2024-10-21 03:17:35 ~33 min tests :page_facing_up:log
:heavy_check_mark: 4189380f #10 2024-10-22 10:56:29 ~2 min ios :package:zip
:heavy_check_mark: 4189380f #11 2024-10-22 10:58:25 ~4 min linux :package:zip
:heavy_check_mark: 4189380f #10 2024-10-22 10:59:07 ~5 min android :package:aar
:heavy_multiplication_x: 4189380f #10 2024-10-22 11:00:23 ~6 min tests-rpc :page_facing_up:log
:heavy_check_mark: 4189380f #11 2024-10-22 11:27:42 ~33 min tests :page_facing_up:log
:heavy_multiplication_x: 916c5128 #12 2024-10-29 09:54:16 ~3 min tests :page_facing_up:log
:heavy_check_mark: 916c5128 #11 2024-10-29 09:55:39 ~4 min ios :package:zip
:heavy_check_mark: 916c5128 #12 2024-10-29 09:55:43 ~4 min linux :package:zip
:heavy_check_mark: 916c5128 #11 2024-10-29 09:56:22 ~5 min android :package:aar
:heavy_multiplication_x: 916c5128 #11 2024-10-29 09:56:59 ~5 min tests-rpc :page_facing_up:log
:x: 265de93f #12 2024-10-29 11:18:39 ~51 sec ios :page_facing_up:log
:x: 265de93f #1 2024-10-29 11:19:59 ~2 min macos :page_facing_up:log
:heavy_multiplication_x: 265de93f #13 2024-10-29 11:20:14 ~2 min tests :page_facing_up:log
:x: 265de93f #12 2024-10-29 11:20:22 ~2 min android :page_facing_up:log
:heavy_multiplication_x: 265de93f #12 2024-10-29 11:21:05 ~3 min tests-rpc :page_facing_up:log
:x: 265de93f #13 2024-10-29 11:21:14 ~3 min linux :page_facing_up:log
:x: 265de93f #1 2024-10-29 11:21:57 ~4 min windows :page_facing_up:log
:x: 265de93f #1 2024-10-29 11:23:31 ~5 min macos :page_facing_up:log
:heavy_check_mark: e5873700 #2 2024-10-29 12:20:09 ~3 min macos :package:zip
:heavy_multiplication_x: e5873700 #14 2024-10-29 12:20:55 ~4 min tests :page_facing_up:log
:heavy_check_mark: e5873700 #13 2024-10-29 12:21:20 ~4 min tests-rpc :page_facing_up:log
:heavy_check_mark: e5873700 #14 2024-10-29 12:21:32 ~4 min linux :package:zip
:heavy_check_mark: e5873700 #13 2024-10-29 12:22:13 ~5 min ios :package:zip
:heavy_check_mark: e5873700 #2 2024-10-29 12:22:45 ~6 min macos :package:zip
:heavy_check_mark: e5873700 #13 2024-10-29 12:23:52 ~7 min android :package:aar
:heavy_multiplication_x: e5873700 #2 2024-10-29 12:25:40 ~8 min windows :package:zip
:heavy_check_mark: e255fb8b #3 2024-10-29 12:38:18 ~4 min macos :package:zip
:heavy_check_mark: e255fb8b #3 2024-10-29 12:39:11 ~5 min macos :package:zip
:heavy_check_mark: e255fb8b #14 2024-10-29 12:40:03 ~6 min ios :package:zip
:heavy_multiplication_x: e255fb8b #3 2024-10-29 12:46:10 ~12 min windows :package:zip
:heavy_check_mark: e255fb8b #14 2024-10-29 12:49:01 ~15 min tests-rpc :page_facing_up:log
:heavy_check_mark: e255fb8b #15 2024-10-29 12:49:32 ~15 min linux :package:zip
:heavy_check_mark: e255fb8b #14 2024-10-29 12:49:32 ~15 min android :package:aar
:heavy_multiplication_x: e255fb8b #15 2024-10-29 13:23:18 ~49 min tests :page_facing_up:log
:heavy_check_mark: 625173e3 #15 2024-11-08 03:02:54 ~4 min tests-rpc :page_facing_up:log
:heavy_check_mark: 625173e3 #4 2024-11-08 03:03:13 ~4 min macos :package:zip
:heavy_check_mark: 625173e3 #15 2024-11-08 03:03:32 ~4 min ios :package:zip
:heavy_check_mark: 625173e3 #15 2024-11-08 03:03:49 ~5 min android :package:aar
:heavy_check_mark: 625173e3 #16 2024-11-08 03:03:49 ~5 min linux :package:zip
:heavy_check_mark: 625173e3 #4 2024-11-08 03:04:26 ~5 min windows :package:zip
:heavy_check_mark: 625173e3 #4 2024-11-08 03:08:12 ~9 min macos :package:zip
:heavy_check_mark: 625173e3 #16 2024-11-08 03:33:05 ~34 min tests :page_facing_up:log
:heavy_check_mark: a83d5310 #5 2024-11-12 14:18:28 ~4 min windows :package:zip
:heavy_check_mark: a83d5310 #16 2024-11-12 14:18:38 ~4 min tests-rpc :page_facing_up:log
:heavy_check_mark: a83d5310 #5 2024-11-12 14:18:44 ~4 min macos :package:zip
:heavy_check_mark: a83d5310 #16 2024-11-12 14:18:59 ~4 min ios :package:zip
:heavy_check_mark: a83d5310 #17 2024-11-12 14:19:51 ~5 min linux :package:zip
:heavy_check_mark: a83d5310 #16 2024-11-12 14:19:52 ~5 min android :package:aar
:heavy_check_mark: a83d5310 #5 2024-11-12 14:21:42 ~7 min macos :package:zip
:heavy_check_mark: a83d5310 #17 2024-11-12 14:48:40 ~34 min tests :page_facing_up:log
:heavy_check_mark: 64dd6d78 #6 2024-11-13 00:19:26 ~4 min windows :package:zip
:heavy_check_mark: 64dd6d78 #6 2024-11-13 00:19:31 ~4 min macos :package:zip
:heavy_check_mark: 64dd6d78 #17 2024-11-13 00:19:32 ~4 min tests-rpc :page_facing_up:log
:heavy_check_mark: 64dd6d78 #17 2024-11-13 00:19:52 ~4 min ios :package:zip
:heavy_check_mark: 64dd6d78 #18 2024-11-13 00:20:30 ~5 min linux :package:zip
:heavy_check_mark: 64dd6d78 #17 2024-11-13 00:20:44 ~5 min android :package:aar
:heavy_check_mark: 64dd6d78 #6 2024-11-13 00:23:43 ~8 min macos :package:zip
:heavy_check_mark: 64dd6d78 #18 2024-11-13 00:48:58 ~33 min tests :page_facing_up:log
:grey_question: Commit :hash: Finished (UTC) Duration Platform Result
:heavy_check_mark: 399c3d58 #7 2024-11-14 11:14:24 ~3 min macos :package:zip
:heavy_check_mark: 399c3d58 #7 2024-11-14 11:14:41 ~4 min windows :package:zip
:heavy_check_mark: 399c3d58 #18 2024-11-14 11:15:10 ~4 min ios :package:zip
:heavy_check_mark: 399c3d58 #18 2024-11-14 11:15:43 ~5 min tests-rpc :page_facing_up:log
:heavy_check_mark: 399c3d58 #19 2024-11-14 11:16:05 ~5 min linux :package:zip
:heavy_check_mark: 399c3d58 #18 2024-11-14 11:16:18 ~5 min android :package:aar
:heavy_check_mark: 399c3d58 #7 2024-11-14 11:18:42 ~8 min macos :package:zip
:heavy_check_mark: 399c3d58 #19 2024-11-14 11:45:49 ~35 min tests :page_facing_up:log
:heavy_check_mark: 2f653980 #8 2024-11-18 12:07:29 ~3 min windows :package:zip
:heavy_check_mark: 2f653980 #19 2024-11-18 12:07:49 ~4 min tests-rpc :page_facing_up:log
:heavy_check_mark: 2f653980 #8 2024-11-18 12:07:53 ~4 min macos :package:zip
:heavy_check_mark: 2f653980 #20 2024-11-18 12:08:04 ~4 min linux :package:zip
:heavy_check_mark: 2f653980 #19 2024-11-18 12:08:53 ~5 min ios :package:zip
:heavy_check_mark: 2f653980 #19 2024-11-18 12:09:17 ~5 min android :package:aar
:heavy_check_mark: 2f653980 #8 2024-11-18 12:12:59 ~9 min macos :package:zip
:heavy_check_mark: 2f653980 #20 2024-11-18 12:36:48 ~33 min tests :page_facing_up:log

status-im-auto avatar Oct 14 '24 13:10 status-im-auto

Codecov Report

Attention: Patch coverage is 19.14191% with 245 lines in your changes missing coverage. Please review.

Project coverage is 60.84%. Comparing base (3a70219) to head (2f65398). Report is 4 commits behind head on develop.

Files with missing lines Patch % Lines
mobile/status.go 0.00% 225 Missing :warning:
protocol/requests/send_transaction.go 0.00% 3 Missing :warning:
cmd/status-backend/main.go 0.00% 2 Missing :warning:
mobile/requests/app_state_change.go 85.71% 1 Missing and 1 partial :warning:
...nection_string_for_bootstrapping_another_device.go 0.00% 2 Missing :warning:
...nection_string_for_importing_keypairs_keystores.go 0.00% 2 Missing :warning:
protocol/requests/delete_multiaccount.go 0.00% 2 Missing :warning:
protocol/requests/verify_account_password.go 0.00% 2 Missing :warning:
server/server.go 80.00% 1 Missing and 1 partial :warning:
server/server_media.go 81.81% 1 Missing and 1 partial :warning:
... and 1 more
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #5943      +/-   ##
===========================================
- Coverage    60.96%   60.84%   -0.12%     
===========================================
  Files          814      821       +7     
  Lines       109253   109496     +243     
===========================================
+ Hits         66606    66626      +20     
- Misses       34817    35025     +208     
- Partials      7830     7845      +15     
Flag Coverage Δ
functional 13.53% <5.28%> (-0.01%) :arrow_down:
unit 60.06% <19.26%> (-0.12%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
api/app_state.go 100.00% <100.00%> (ø)
mobile/callog/status_request_log.go 91.66% <ø> (ø)
...uests/input_connection_string_for_bootstrapping.go 100.00% <100.00%> (ø)
server/pairing/client.go 53.98% <100.00%> (+0.12%) :arrow_up:
api/geth_backend.go 54.69% <88.88%> (-0.03%) :arrow_down:
cmd/status-backend/main.go 0.00% <0.00%> (ø)
mobile/requests/app_state_change.go 85.71% <85.71%> (ø)
...nection_string_for_bootstrapping_another_device.go 0.00% <0.00%> (ø)
...nection_string_for_importing_keypairs_keystores.go 0.00% <0.00%> (ø)
protocol/requests/delete_multiaccount.go 0.00% <0.00%> (ø)
... and 5 more

... and 34 files with indirect coverage changes

codecov[bot] avatar Oct 14 '24 13:10 codecov[bot]

It would be great if you (@ilmotta) could help with the frontend!

Thanks for this PR @qfrank!


@ulisesmac, this is the PR I mentioned to you these days. Would you be interested in making the integration work in the mobile client? Note: this work has less priority than other important bugs affecting the release.

To get started, there's a README in /cmd/status-backend/README.md.

make generate
make run-status-backend PORT=9050

@qfrank @igor-sirotin in the README I couldn't find any explanation about how to subscribe to signals. Any tips?

ilmotta avatar Oct 15 '24 06:10 ilmotta

I couldn't find any explanation about how to subscribe to signals. Any tips?

pls see this and relate test code @ilmotta

qfrank avatar Oct 15 '24 06:10 qfrank

It would be great if you (@ilmotta) could help with the frontend!

Thanks for this PR @qfrank!

@ulisesmac, this is the PR I mentioned to you these days. Would you be interested in making the integration work in the mobile client? Note: this work has less priority than other important bugs affecting the release.

To get started, there's a README in /cmd/status-backend/README.md.

make generate
make run-status-backend PORT=9050

@qfrank @igor-sirotin in the README I couldn't find any explanation about how to subscribe to signals. Any tips?

I'd love to do it :heart: I'll probably be starting with this by Thursday :smile:

ulisesmac avatar Oct 15 '24 17:10 ulisesmac

status-mobile can use the status-backend via an HTTP server so that the status-backend can run separately

While this is a good idea for developing, I think for release builds status-go should remain a shared library linked to the app.

There's no need to use the whole TCP/IP/HTTP stack for this when we can call the method directly. It will be slower with the network and OS involved.

I suggest to make this optional, used only in local debug builds.

cc @ilmotta @qfrank

igor-sirotin avatar Oct 16 '24 09:10 igor-sirotin

I couldn't find any explanation about how to subscribe to signals. Any tips?

Indeed I missed to write this, will do today. In short, you should connect to ws://localhost:<port>/signals as the first thing when running status-backend. You'll receive all signals.

cc @ilmotta @ulisesmac

igor-sirotin avatar Oct 16 '24 09:10 igor-sirotin

There's no need to use the whole TCP/IP/HTTP stack for this when we can call the method directly. It will be slower with the network and OS involved.

I suggest to make this optional, used only in local debug builds.

Fully agree @igor-sirotin. But, sorry there's always a but :) I wonder (on a very personal level) if running status-go in a separate machine in my LAN would work well.

The overhead could be compensated by some other advantages. I always dreamed of having Status clients running as simpler, thin shells, which could help with some use cases:

  • status-go could run much faster and wouldn't be restricted by the hardware the client is running on.
  • The overhead of the extra layers could be negligible if latency is low enough (in a LAN for example). Reminds me of Phoenix LiveView.
  • Using a mobile device as a full node when the mobile device is in my home. If the UI catches up with features like community administration, it would allow eliminating the need for the combo desktop app + mobile app, and some users could appreciate that. When I say mobile device I also include tablets in the equation. Besides, foldable phones might become more of a thing.
  • Could be a nice push in the architectural direction of keeping clients thin (nowadays the mobile client for example has too many responsibilities that should be in status-go IMO). With truly thin clients, the community would be able to more easily fork the clients and build their own versions.
  • Running status-go separately would give some users the power to more easily debug release builds, using actual data (I mean not just for development).

Of course, this capability would be optional and not the default. If we could make the release build configurable to support the HTTP backend it would be 🔥 Or is it already? The default for release builds would still be to embed as a shared lib, I fully agree with you 👍🏼

Curious to hear your thoughts :)

cc @Samyoul @osmaczko @qfrank

ilmotta avatar Oct 16 '24 11:10 ilmotta

There's no need to use the whole TCP/IP/HTTP stack for this when we can call the method directly. It will be slower with the network and OS involved. I suggest to make this optional, used only in local debug builds.

Fully agree @igor-sirotin. But, sorry there's always a but :) I wonder (on a very personal level) if running status-go in a separate machine in my LAN would work well.

The overhead could be compensated by some other advantages. I always dreamed of having Status clients running as simpler, thin shells, which could help with some use cases:

  • status-go could run much faster and wouldn't be restricted by the hardware the client is running on.
  • The overhead of the extra layers could be negligible if latency is low enough (in a LAN for example). Reminds me of Phoenix LiveView.
  • Using a mobile device as a full node when the mobile device is in my home. If the UI catches up with features like community administration, it would allow eliminating the need for the combo desktop app + mobile app, and some users could appreciate that. When I say mobile device I also include tablets in the equation. Besides, foldable phones might become more of a thing.
  • Could be a nice push in the architectural direction of keeping clients thin (nowadays the mobile client for example has too many responsibilities that should be in status-go IMO). With truly thin clients, the community would be able to more easily fork the clients and build their own versions.
  • Running status-go separately would give some users the power to more easily debug release builds, using actual data (I mean not just for development).

Of course, this capability would be optional and not the default. If we could make the release build configurable to support the HTTP backend it would be 🔥 Or is it already? The default for release builds would still be to embed as a shared lib, I fully agree with you 👍🏼

Curious to hear your thoughts :)

cc @Samyoul @osmaczko @qfrank

HAHA, We have the same dream, let's start with local debug build first

qfrank avatar Oct 17 '24 02:10 qfrank

To get started, there's a README in /cmd/status-backend/README.md.

make generate
make run-status-backend PORT=9050

I'm new to status-go, so I need some help, when I run make run-status-backend ... I always get the following output at the end:

go run ./cmd/status-backend --address localhost:12345
Configuring Nix shell for target 'default'...
vendor/github.com/waku-org/go-zerokit-rln/rln/link/x86_64.go:8:8: cannot find module providing package github.com/waku-org/go-zerokit-rln-x86_64/rln: import lookup disabled by -mod=vendor
	(Go version in go.mod is at least 1.14 and vendor directory exists.)
make: *** [Makefile:181: run-status-backend] Error 1

Before I built status-go as specified here: https://github.com/status-im/status-go/blob/develop/_docs/how-to-build.md#build-status-go

What is the step I'm missing out? :thinking:

ulisesmac avatar Oct 18 '24 23:10 ulisesmac

To get started, there's a README in /cmd/status-backend/README.md.

make generate
make run-status-backend PORT=9050

I'm new to status-go, so I need some help, when I run make run-status-backend ... I always get the following output at the end:

go run ./cmd/status-backend --address localhost:12345
Configuring Nix shell for target 'default'...
vendor/github.com/waku-org/go-zerokit-rln/rln/link/x86_64.go:8:8: cannot find module providing package github.com/waku-org/go-zerokit-rln-x86_64/rln: import lookup disabled by -mod=vendor
	(Go version in go.mod is at least 1.14 and vendor directory exists.)
make: *** [Makefile:181: run-status-backend] Error 1

Before I built status-go as specified here: https://github.com/status-im/status-go/blob/develop/_docs/how-to-build.md#build-status-go

What is the step I'm missing out? 🤔

what if install go 1.22 manually? feel free to DM me on discord @ulisesmac

qfrank avatar Oct 20 '24 08:10 qfrank

vendor/github.com/waku-org/go-zerokit-rln/rln/link/x86_64.go:8:8: cannot find module providing package github.com/waku-org/go-zerokit-rln-x86_64/rln: import lookup disabled by -mod=vendor

@ulisesmac you need to build it with --tags gowaku_no_rln.

Not sure why it works for everyone else, but we should add it to the command:

--- a/Makefile
+++ b/Makefile
@@ -178,7 +178,7 @@ status-backend: build/bin/status-backend
 run-status-backend: PORT ?= 0
 run-status-backend: generate
 run-status-backend: ##@run Start status-backend server listening to localhost:PORT
-       go run ./cmd/status-backend --address localhost:${PORT}
+       go run --tags '$(BUILD_TAGS)' ./cmd/status-backend --address localhost:${PORT}
 
 statusd-prune-docker-image: SHELL := /bin/sh
 statusd-prune-docker-image: ##@statusd-prune Build statusd-prune docker image

igor-sirotin avatar Oct 22 '24 11:10 igor-sirotin

  • Using a mobile device as a full node when the mobile device is in my home. If the UI catches up with features like community administration, it would allow eliminating the need for the combo desktop app + mobile app, and some users could appreciate that. When I say mobile device I also include tablets in the equation. Besides, foldable phones might become more of a thing.

Hey @ilmotta, I like the idea in general, but I see a few potential challenges and would need further clarification on your vision. For instance, what happens when you're away from home? Would the mobile app still function in that case? If so, how would it operate without a connection to the status-go backend?

If the connection to the status backend is designed to intelligently switch to a LAN version when at home, there are a few concerns. First, switching from a restricted mobile version to a full-node experience could introduce significant complexity on both the backend and UI side. Second, typical home wifi routers can become unstable if overwhelmed by the high volume of traffic generated by running a full node. I'm not referring to the messages propagated by Waku itself, but rather the logic within status-go. For example, in an abstract scenario like importing a community from an archive, thousands of signals (websocket messages in this case) could be sent per millisecond, potentially overwhelming the router. That said, the latter is just a guess—perhaps this wouldn't be an issue in practice.

osmaczko avatar Oct 23 '24 19:10 osmaczko

Hey @osmaczko, you are raising great questions. Thanks for coming back to this subject.

I haven't thought too much about technical feasibility or even about how the experience would be. I guess the use case I described can be useful, but only to a tiny 🪡 fraction of the Status user base. For everything I say here, please take it lightly, I'm not representing anything from Status :)

There are a few constraints from my particular context:

  • I work from home the majority of the time.
  • I have access to fiber and I believe bandwidth wouldn't be a problem for my plan.
  • I'd be willing to upgrade my router if I'm limited by it.
  • I'd be willing to pay for running the node in the cloud. Maybe my wallet won't like it 😅 I'd be of course very concerned about security, so I wouldn't use this for important stuff until "proven safe".
  • I have a few machines to spare at home. Running a full node wouldn't affect my work machines (except maybe overloading the network, but there are ways around this).

I mentioned about the local network use case, but the most interesting one to me would be to run the server in the cloud or publicly exposed via my own local infra. If we eventually solve the problem of good incentives for running full nodes, the infra cost could be partially offset.

In order to help justify the operational cost, as a user, I would like to have access to all possible Status features on "any" device. But this is not possible nowadays, clients have rightfully being developed with platform restrictions in mind, thus the mobile client is restricted. It's kind of a chicken and egg problem.

One other scenario I thought before was about having one status-go for multiple clients. Let's say, when running the desktop app at home, that "instance" of status-go could be the main one managing my other clients (in case they are paired and only if they're in the same local network), sort of a creating a seamless experience to sync data between them instantly without relying exclusively on the Waku layer. Okay, this is kind of a stretch, would complicate many things. Just sharing because why not :)

If we keep brainstorming, new use cases will pop-up. Whether they can justify cost of development is a completely different story.

For instance, what happens when you're away from home? Would the mobile app still function in that case? If so, how would it operate without a connection to the status-go backend?

First, switching from a restricted mobile version to a full-node experience could introduce significant complexity on both the backend and UI side.

The UX aspect I would be totally fine doing something manually. For example, while outdoors, the mobile app would detect the abrupt connection drop and enter a "frozen" state where I would have to manually switch to the standard light mode and restart the app.

A more drastic measure could be to have two completely distinct apps, compiled quite differently even. I would restore the same Status profiles in both apps. One app would be the standard, good for outdoors usage, the other would only work at home and with more features. But there are many problems with this approach.

The experience side of things I think is solvable, especially because it would be a niche part of the product, so users would be willing to make some sacrifices. On the infra side, we could help by giving users quick ways to securely deploy. I see Syncthing as a great example in the FOSS world.

For example, in an abstract scenario like importing a community from an archive, thousands of signals (websocket messages in this case) could be sent per millisecond, potentially overwhelming the router.

I would expect websockets to handle at least a handful of thousands of messages per second even in cheap desktop machines. The router I can't say much. Probably in the case you described, a mobile client would be overwhelmed way before the router. I lack context here, but a solution could be to apply backpressure correctly in the importing process (limit throughout dynamically). Added complexity as you said 👍🏼

One thing I'm curious about is to understand how much worse (if worse at all in practice) would the experience be to users, if status-backend was the only way of integrating with status-go.

I sincerely believe the complexity in the backend is very high and with many unknowns. And there's no strong evidence to justify this effort. But devs might gradually explore this architecture on their own. I started to think more about these problems when I attempted to create a messaging app within Emacs, without all the cruft in the desktop/mobile clients. Emacs is single-threaded and it's hard to optimize some things and I wanted to keep the client as light as possible. I'd pay to use Status like that :) Similar to this client, for example https://github.com/alphapapa/ement.el

ilmotta avatar Oct 24 '24 00:10 ilmotta

I have some concerns regarding MEDIA_HTTPS thingy, please take a look at #6060 (review).

Also, I highly recommend to split this PR into fewer smaller ones. I have already approved this PR 3 weeks ago, an you could have merged it back then, but the PR evolved and there're now more changes on top. And now new changes are holding old ones, although they're technically not dependent on each other. And this is not even counting the burden of reviewers to read so many changes.

Let me explain it a bit as you might don't have the full context. You can review it by commit, you have approved this PR already, I didn't expect you review it again ... You can just review the later one. It doesn't mean I can merge it once I got 2 approvals, I need to pass manual QA. The relate mobile PRs ( 21450 / 21550 ) need full regression test, so we had a conclusion to combile these PRs to test once. Maybe they're technically not dependent on each other in your point view, These PRs all aim to achieve the same goal: ensuring the mobile frontend can properly use the status backend server. Since this PR hasn't been merged yet, I need to test the status backend server locally. To address the HTTPS issue, I must create a separate PR based on this one. From the frontend's perspective, PRs 5943 and 6060 are functionally related. @igor-sirotin

qfrank avatar Nov 19 '24 01:11 qfrank

@qfrank I see you intention, but I still think it should've been done a bit differently. I guess at this point it doesn't make sense to change, but I'll post it for the future PRs.

... I need to pass manual QA

I think this is a bit wrong approach in this case.

There're 2 things to be tested:

  • New status-go code
  • Integration to status-mobile

Because 99% of changes in this PR is new code, you're not affecting desktop/mobile old behaviour.

So It is the integration to status-mobile that requires manual QA and full regression test. But the status-go changes can be merged independently with it's own testing.

Maybe you're using status-mobile regression tests to test status-go changes, but this is also not quite right. status-go should be self-tested, rather then with status-mobile. I know functional tests framework is still not in production condition, so I was ready to merge it without coverage. Just if you have tested each new endpoint manually (e.g. with Postman) would be ok in this case.

igor-sirotin avatar Nov 19 '24 07:11 igor-sirotin

@qfrank I see you intention, but I still think it should've been done a bit differently. I guess at this point it doesn't make sense to change, but I'll post it for the future PRs.

... I need to pass manual QA

I think this is a bit wrong approach in this case.

There're 2 things to be tested:

* New `status-go` code

* Integration to `status-mobile`

Because 99% of changes in this PR is new code, you're not affecting desktop/mobile old behaviour.

So It is the integration to status-mobile that requires manual QA and full regression test. But the status-go changes can be merged independently with it's own testing.

Maybe you're using status-mobile regression tests to test status-go changes, but this is also not quite right. status-go should be self-tested, rather then with status-mobile. I know functional tests framework is still not in production condition, so I was ready to merge it without coverage. Just if you have tested each new endpoint manually (e.g. with Postman) would be ok in this case.

make sense, pls merge it, thank you @igor-sirotin

qfrank avatar Nov 19 '24 08:11 qfrank

Merging now. MEDIA_HTTPS will be removed in an upcoming PR, details in https://github.com/status-im/status-go/pull/6060#discussion_r1847978959

igor-sirotin avatar Nov 19 '24 09:11 igor-sirotin