status-go icon indicating copy to clipboard operation
status-go copied to clipboard

refactor: only use shards

Open chaitanyaprem opened this issue 1 year ago • 13 comments

Closes #5277

chaitanyaprem avatar Jul 03 '24 09:07 chaitanyaprem

Jenkins Builds

Click to see older builds (126)
:grey_question: Commit :hash: Finished (UTC) Duration Platform Result
:heavy_check_mark: 547a3e54 #1 2024-07-03 09:32:09 ~4 min linux :package:zip
:heavy_check_mark: 547a3e54 #1 2024-07-03 09:32:42 ~4 min ios :package:zip
:heavy_check_mark: 547a3e54 #1 2024-07-03 09:33:40 ~5 min android :package:aar
:heavy_check_mark: 547a3e54 #2 2024-07-03 09:35:58 ~2 min ios :package:zip
:heavy_multiplication_x: 547a3e54 #1 2024-07-03 09:36:42 ~8 min tests :page_facing_up:log
:heavy_check_mark: 547a3e54 #2 2024-07-03 09:44:03 ~11 min linux :package:zip
:heavy_check_mark: 547a3e54 #2 2024-07-03 09:45:47 ~11 min android :package:aar
:heavy_multiplication_x: 547a3e54 #2 2024-07-03 10:23:49 ~46 min tests :page_facing_up:log
:heavy_check_mark: 5d80b57d #3 2024-07-03 14:22:24 ~2 min ios :package:zip
:heavy_multiplication_x: 5d80b57d #3 2024-07-03 14:30:57 ~11 min tests :page_facing_up:log
:heavy_check_mark: 5d80b57d #3 2024-07-03 14:31:26 ~11 min linux :package:zip
:heavy_check_mark: 5d80b57d #3 2024-07-03 14:33:45 ~14 min android :package:aar
:heavy_check_mark: 3369d267 #4 2024-07-03 15:55:21 ~5 min ios :package:zip
:heavy_check_mark: 3369d267 #4 2024-07-03 15:59:59 ~10 min android :package:aar
:heavy_check_mark: 3369d267 #4 2024-07-03 16:02:17 ~12 min linux :package:zip
:heavy_multiplication_x: 3369d267 #4 2024-07-03 16:03:15 ~13 min tests :page_facing_up:log
:heavy_multiplication_x: 519e54d3 #5 2024-07-03 17:22:58 ~1 min tests :page_facing_up:log
:heavy_check_mark: 519e54d3 #5 2024-07-03 17:23:44 ~2 min linux :package:zip
:heavy_check_mark: 519e54d3 #5 2024-07-03 17:24:31 ~2 min ios :package:zip
:heavy_check_mark: 519e54d3 #5 2024-07-03 17:26:56 ~5 min android :package:aar
:heavy_check_mark: cbbdb93a #6 2024-07-03 17:34:54 ~2 min ios :package:zip
:heavy_check_mark: cbbdb93a #6 2024-07-03 17:38:01 ~5 min android :package:aar
:heavy_check_mark: cbbdb93a #6 2024-07-03 17:38:59 ~6 min linux :package:zip
:heavy_multiplication_x: cbbdb93a #6 2024-07-03 17:42:36 ~10 min tests :page_facing_up:log
:heavy_check_mark: 3a2a08f1 #7 2024-07-03 18:29:42 ~1 min android :package:aar
:heavy_check_mark: 3a2a08f1 #7 2024-07-03 18:29:44 ~1 min linux :package:zip
:heavy_check_mark: 3a2a08f1 #7 2024-07-03 18:30:20 ~2 min ios :package:zip
:heavy_multiplication_x: 3a2a08f1 #7 2024-07-03 19:06:12 ~38 min tests :page_facing_up:log
:heavy_check_mark: 3a2a08f1 #9 2024-07-03 21:42:21 ~1 min android :package:aar
:heavy_check_mark: 3a2a08f1 #9 2024-07-03 21:43:01 ~1 min linux :package:zip
:heavy_check_mark: 3a2a08f1 #9 2024-07-03 21:43:33 ~2 min ios :package:zip
:heavy_multiplication_x: 3a2a08f1 #9 2024-07-03 21:46:25 ~5 min tests :page_facing_up:log
:heavy_check_mark: 639c5718 #8 2024-07-03 21:01:36 ~2 min linux :package:zip
:heavy_check_mark: 639c5718 #8 2024-07-03 21:01:39 ~2 min android :package:aar
:heavy_check_mark: 639c5718 #8 2024-07-03 21:02:32 ~3 min ios :package:zip
:heavy_multiplication_x: 639c5718 #8 2024-07-03 21:36:28 ~36 min tests :page_facing_up:log
:heavy_multiplication_x: 958551c6 #10 2024-07-03 22:17:01 ~1 min tests :page_facing_up:log
:heavy_check_mark: 958551c6 #10 2024-07-03 22:18:02 ~2 min android :package:aar
:heavy_check_mark: 958551c6 #10 2024-07-03 22:18:08 ~2 min linux :package:zip
:heavy_check_mark: 958551c6 #10 2024-07-03 22:18:47 ~2 min ios :package:zip
:heavy_check_mark: f4ad281b #11 2024-07-03 22:25:45 ~1 min android :package:aar
:heavy_check_mark: f4ad281b #11 2024-07-03 22:26:11 ~1 min linux :package:zip
:heavy_check_mark: f4ad281b #11 2024-07-03 22:26:46 ~2 min ios :package:zip
:heavy_multiplication_x: f4ad281b #11 2024-07-03 22:47:58 ~23 min tests :page_facing_up:log
:heavy_check_mark: f4ad281b #12 2024-07-03 23:43:44 ~43 min tests :page_facing_up:log
:heavy_check_mark: f4ad281b #12 2024-07-04 05:07:23 ~2 min android :package:aar
:heavy_check_mark: f4ad281b #12 2024-07-04 05:07:26 ~2 min linux :package:zip
:heavy_check_mark: f4ad281b #12 2024-07-04 05:08:09 ~2 min ios :package:zip
:heavy_check_mark: f4ad281b #13 2024-07-04 05:50:13 ~44 min tests :page_facing_up:log
:x: f4d63d5d #13 2024-07-04 05:41:59 ~27 sec android :page_facing_up:log
:x: f4d63d5d #13 2024-07-04 05:42:03 ~30 sec ios :page_facing_up:log
:x: f4d63d5d #13 2024-07-04 05:42:06 ~31 sec linux :page_facing_up:log
:heavy_check_mark: 5759c16c #14 2024-07-04 05:45:58 ~2 min android :package:aar
:heavy_check_mark: 5759c16c #14 2024-07-04 05:46:02 ~2 min linux :package:zip
:heavy_check_mark: 5759c16c #14 2024-07-04 05:46:43 ~2 min ios :package:zip
:heavy_check_mark: 4ae62d69 #15 2024-07-04 05:49:40 ~2 min android :package:aar
:heavy_check_mark: 4ae62d69 #15 2024-07-04 05:49:40 ~2 min linux :package:zip
:heavy_check_mark: 4ae62d69 #15 2024-07-04 05:50:05 ~2 min ios :package:zip
:heavy_check_mark: 4ae62d69 #14 2024-07-04 06:34:13 ~43 min tests :page_facing_up:log
:heavy_check_mark: d59d4512 #16 2024-07-05 00:39:57 ~2 min android :package:aar
:heavy_check_mark: d59d4512 #16 2024-07-05 00:40:05 ~2 min linux :package:zip
:heavy_check_mark: d59d4512 #1 2024-07-05 00:40:17 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: d59d4512 #16 2024-07-05 00:41:25 ~3 min ios :package:zip
:heavy_check_mark: d59d4512 #15 2024-07-05 01:22:08 ~44 min tests :page_facing_up:log
:heavy_multiplication_x: d72fee2a #16 2024-07-10 09:04:52 ~1 min tests :page_facing_up:log
:heavy_check_mark: d72fee2a #17 2024-07-10 09:06:36 ~3 min ios :package:zip
:heavy_check_mark: d72fee2a #17 2024-07-10 09:06:46 ~3 min linux :package:zip
:heavy_check_mark: d72fee2a #17 2024-07-10 09:07:29 ~4 min android :package:aar
:heavy_check_mark: 436ad545 #18 2024-07-10 09:09:09 ~2 min linux :package:zip
:heavy_multiplication_x: 436ad545 #3 2024-07-10 09:09:20 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: 436ad545 #18 2024-07-10 09:09:26 ~1 min android :package:aar
:heavy_check_mark: 436ad545 #18 2024-07-10 09:09:49 ~2 min ios :package:zip
:heavy_multiplication_x: 436ad545 #17 2024-07-10 09:13:21 ~6 min tests :page_facing_up:log
:heavy_check_mark: 24702442 #19 2024-07-10 09:34:06 ~2 min ios :package:zip
:heavy_check_mark: 24702442 #5 2024-07-10 09:34:19 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: 24702442 #19 2024-07-10 09:34:49 ~3 min linux :package:zip
:heavy_check_mark: 24702442 #19 2024-07-10 09:35:35 ~4 min android :package:aar
:heavy_check_mark: 24702442 #18 2024-07-10 10:15:34 ~44 min tests :page_facing_up:log
:heavy_multiplication_x: ea9fc20b #19 2024-07-13 01:15:57 ~1 min tests :page_facing_up:log
:heavy_check_mark: ea9fc20b #6 2024-07-13 01:16:24 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: ea9fc20b #20 2024-07-13 01:16:43 ~2 min linux :package:zip
:heavy_check_mark: ea9fc20b #20 2024-07-13 01:16:43 ~2 min android :package:aar
:heavy_check_mark: ea9fc20b #20 2024-07-13 01:17:36 ~3 min ios :package:zip
:heavy_check_mark: 66fec5d7 #21 2024-07-13 01:20:52 ~1 min linux :package:zip
:heavy_check_mark: 66fec5d7 #7 2024-07-13 01:21:02 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: 66fec5d7 #21 2024-07-13 01:21:37 ~2 min ios :package:zip
:heavy_check_mark: 66fec5d7 #21 2024-07-13 01:23:19 ~4 min android :package:aar
:heavy_multiplication_x: 66fec5d7 #20 2024-07-13 01:57:34 ~38 min tests :page_facing_up:log
:heavy_multiplication_x: 66fec5d7 #21 2024-07-13 02:58:30 ~37 min tests :page_facing_up:log
:heavy_multiplication_x: 66fec5d7 #22 2024-07-13 04:54:17 ~37 min tests :page_facing_up:log
:heavy_multiplication_x: 66fec5d7 #23 2024-07-13 05:34:38 ~37 min tests :page_facing_up:log
:heavy_check_mark: 66fec5d7 #23 2024-07-15 07:02:08 ~1 min android :package:aar
:heavy_check_mark: 66fec5d7 #23 2024-07-15 07:02:34 ~1 min linux :package:zip
:heavy_check_mark: 66fec5d7 #9 2024-07-15 07:02:53 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: 66fec5d7 #23 2024-07-15 07:03:33 ~2 min ios :package:zip
:heavy_multiplication_x: 66fec5d7 #25 2024-07-15 07:38:43 ~37 min tests :page_facing_up:log
:heavy_check_mark: a4ada5a1 #8 2024-07-13 23:13:33 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: a4ada5a1 #22 2024-07-13 23:13:47 ~2 min android :package:aar
:heavy_check_mark: a4ada5a1 #22 2024-07-13 23:14:33 ~3 min ios :package:zip
:heavy_check_mark: a4ada5a1 #22 2024-07-13 23:15:20 ~3 min linux :package:zip
:heavy_multiplication_x: a4ada5a1 #24 2024-07-13 23:49:21 ~37 min tests :page_facing_up:log
:heavy_check_mark: ad22714b #10 2024-07-15 07:43:09 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: ad22714b #24 2024-07-15 07:43:11 ~2 min android :package:aar
:heavy_check_mark: ad22714b #24 2024-07-15 07:43:17 ~2 min linux :package:zip
:heavy_check_mark: ad22714b #24 2024-07-15 07:44:07 ~3 min ios :package:zip
:heavy_check_mark: ad22714b #26 2024-07-15 08:25:49 ~44 min tests :page_facing_up:log
:heavy_check_mark: dc8ea451 #25 2024-07-15 08:15:20 ~1 min android :package:aar
:heavy_check_mark: dc8ea451 #25 2024-07-15 08:15:43 ~1 min linux :package:zip
:heavy_check_mark: dc8ea451 #11 2024-07-15 08:15:59 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: dc8ea451 #25 2024-07-15 08:16:39 ~2 min ios :package:zip
:heavy_check_mark: dc8ea451 #27 2024-07-15 09:10:21 ~44 min tests :page_facing_up:log
:heavy_multiplication_x: 4853a007 #28 2024-07-15 10:13:23 ~56 sec tests :page_facing_up:log
:heavy_check_mark: 4853a007 #12 2024-07-15 10:14:39 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: 4853a007 #26 2024-07-15 10:14:48 ~2 min linux :package:zip
:heavy_check_mark: 4853a007 #26 2024-07-15 10:15:49 ~3 min ios :package:zip
:heavy_check_mark: 4853a007 #26 2024-07-15 10:16:35 ~4 min android :package:aar
:x: 4d847526 #27 2024-07-15 10:36:35 ~32 sec ios :page_facing_up:log
:x: 4d847526 #27 2024-07-15 10:36:38 ~32 sec linux :page_facing_up:log
:x: 4d847526 #27 2024-07-15 10:36:40 ~36 sec android :page_facing_up:log
:heavy_multiplication_x: 4d847526 #13 2024-07-15 10:37:10 ~1 min tests-rpc :page_facing_up:log
:heavy_multiplication_x: 4d847526 #29 2024-07-15 10:37:26 ~1 min tests :page_facing_up:log
:heavy_check_mark: f9ab393d #28 2024-07-15 10:54:43 ~1 min android :package:aar
:heavy_check_mark: f9ab393d #14 2024-07-15 10:55:22 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: f9ab393d #28 2024-07-15 10:55:30 ~2 min linux :package:zip
:heavy_check_mark: f9ab393d #28 2024-07-15 10:56:10 ~2 min ios :package:zip
:heavy_check_mark: f9ab393d #30 2024-07-15 11:37:59 ~44 min tests :page_facing_up:log
:grey_question: Commit :hash: Finished (UTC) Duration Platform Result
:heavy_multiplication_x: 2d2be5c8 #31 2024-07-15 14:21:59 ~1 min tests :page_facing_up:log
:heavy_check_mark: 2d2be5c8 #29 2024-07-15 14:23:09 ~2 min linux :package:zip
:heavy_check_mark: 2d2be5c8 #15 2024-07-15 14:23:22 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: 2d2be5c8 #29 2024-07-15 14:23:36 ~2 min android :package:aar
:heavy_check_mark: 2d2be5c8 #29 2024-07-15 14:24:42 ~3 min ios :package:zip
:heavy_check_mark: 2b5cde1a #30 2024-07-15 14:32:05 ~2 min android :package:aar
:heavy_check_mark: 2b5cde1a #16 2024-07-15 14:32:12 ~2 min tests-rpc :page_facing_up:log
:heavy_check_mark: 2b5cde1a #30 2024-07-15 14:33:14 ~3 min ios :package:zip
:heavy_check_mark: 2b5cde1a #30 2024-07-15 14:33:41 ~3 min linux :package:zip
:heavy_multiplication_x: 2b5cde1a #32 2024-07-15 14:36:51 ~6 min tests :page_facing_up:log
:heavy_check_mark: 2b5cde1a #33 2024-07-15 15:21:35 ~43 min tests :page_facing_up:log

status-im-auto avatar Jul 03 '24 09:07 status-im-auto

For the scenarios that are failing, I think we're sending the messages on the wrong pubsub topic, i.e.

rawMessage := common.RawMessage{
		Payload:             payload,
		CommunityID:         community.ID(),
		SkipEncryptionLayer: true,
		MessageType:         protobuf.ApplicationMetadataMessage_COMMUNITY_EDIT_SHARED_ADDRESSES,
		PubsubTopic:         community.PubsubTopic(), // TODO: confirm if it should be sent in community pubsub topic
		ResendType:          common.ResendTypeRawMessage,
	}

I think this should have probably been sent on shard.DefaultNonProtectedPubsubTopic() (If I change that TestMessengerCommunitiesTokenPermissionsSuite/TestEditSharedAddresses passes).

I'm investigating the other cases.

richard-ramos avatar Jul 03 '24 20:07 richard-ramos

@jrainville, @mprakhov Please do check the latest commit. I did change the pubsub topic of some messages.

The question is: Can these messages be sent before the user has joined a community? If so, they should be send/received in either the default non protected topic or in the default shard, and not in the community shard. Pls let me know.

richard-ramos avatar Jul 03 '24 21:07 richard-ramos

Can these messages be sent before the user has joined a community? If so, they should be send/received in either the default non protected topic or in the default shard, and not in the community shard. Pls let me know.

@richard-ramos both of these can only be called when the user has already joined the community.

jrainville avatar Jul 03 '24 21:07 jrainville

I'll revert last commit then

richard-ramos avatar Jul 03 '24 21:07 richard-ramos

'kay, all green now :)

richard-ramos avatar Jul 03 '24 23:07 richard-ramos

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

igor-sirotin avatar Jul 04 '24 08:07 igor-sirotin

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this. But both of them seem to be OOO , not sure who will test.

chaitanyaprem avatar Jul 04 '24 08:07 chaitanyaprem

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Here is the status-desktop PR https://github.com/status-im/status-desktop/pull/15455

chaitanyaprem avatar Jul 04 '24 08:07 chaitanyaprem

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this. But both of them seem to be OOO , not sure who will test.

Let's ping @glitchminer and @qoqobolo 🙂

igor-sirotin avatar Jul 04 '24 10:07 igor-sirotin

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this. But both of them seem to be OOO , not sure who will test.

@chaitanyaprem thank you for the PR and pinging QA team. Could you please rebase status go branch to the latest develop and create status mobile PR as well?

Also, would appreciate some details on what exactly has been affected and how should we test this PR from user perspective (what test scenarios/areas)? Thank you.

pavloburykh avatar Jul 04 '24 14:07 pavloburykh

Manual testing probably is good idea for this one? cc @anastasiyaig @churik

Yep, was going to create sttus-desktop PR with this. But both of them seem to be OOO , not sure who will test.

@chaitanyaprem thank you for the PR and pinging QA team. Could you please rebase status go branch to the latest develop and create status mobile PR as well?

Also, would appreciate some details on what exactly has been affected and how should we test this PR from user perspective (what test scenarios/areas)? Thank you.

Done and here it is https://github.com/status-im/status-mobile/pull/20658. Not sure if many scenarios can be tested from mobile though, because they involve creating commuities and assigning shards to them. Maybe mobile can be used as a device to test send/receive messages and joini communities in this case.

chaitanyaprem avatar Jul 05 '24 01:07 chaitanyaprem

@chaitanyaprem I have left a comment in mobile PR, posting it here as well, so everyone is aware about the problem.

Unfortunately we cannot proceed with testing as currently we experience global Waku issues which result in communities failed to be fetched. This is a reason why e2e run in this PR has mostly failed.

I have already pinged Waku team about the problem, waiting for the fix.

Here is example of logs of a user from this PR who failed to fetch community when following community link.

Status-debug-logs - 2024-07-05T101140.382.zip

pavloburykh avatar Jul 05 '24 07:07 pavloburykh

Hey @cammellos! What do you think, is this a priority to test this PR on mobile or we can rely on e2e results from our side? Note that our e2e do not cover sharded communities.

pavloburykh avatar Jul 12 '24 13:07 pavloburykh

@pavloburykh thanks, given the priorities for 2.30, I don't think we have enough capacity in the QA team to test something that's not strictly needed for 2.30. If @chaitanyaprem thinks that this is not impacting anything unless the community is manually sharded, then e2e tests is probably good enough, but if it impacts production code, best to leave it for later

cammellos avatar Jul 12 '24 13:07 cammellos

@pavloburykh thanks, given the priorities for 2.30, I don't think we have enough capacity in the QA team to test something that's not strictly needed for 2.30. If @chaitanyaprem thinks that this is not impacting anything unless the community is manually sharded, then e2e tests is probably good enough, but if it impacts production code, best to leave it for later

afaik it doesn't affect any other code apart from sharded communities. Also, i have tested by creating a community and assigning it a shard 128 via RPC (which requires a privateKey) to be specified and i was able to fetch the community from both mobile and desktop. Maybe there is a bug in the UI for assigning shard which is causing the issue.

But i'll want a yes from the great @richard-ramos as well to be sure.

chaitanyaprem avatar Jul 13 '24 00:07 chaitanyaprem

IMO it should be fine to merge this PR. The normal behavior of the app is maintained, and I think that as long as the custom shard RPC methods still work fine, which as far as we know, it is the case, we should be good to go!

richard-ramos avatar Jul 13 '24 01:07 richard-ramos