go-threads
go-threads copied to clipboard
textileio
Response closed without grpc-status (headers only)
This is a common error that we see coming from various Threads Clients. It is essentially the default error coming back from a Client that doesn't receive a valid response from the Proxy gRPC service. There are a number of common reasons for this error:
- The Service the Client is attempting to connect to isn't running.
- This can happen if your local threadsd isn't running, or was shut down.
- Check that your threadsd is still running.
- Alternatively, if you are connecting to a remote service, make sure the host url you are using is correct.
- The request being sent to the service is malformed.
- This can happen if there are issues with some input to the Client (perhaps the ThreadID is invalid, etc), or the Client isn't correctly marshaling the data into a protobuf.
- Check that your inputs are correct/the right types.
- If you are running in React Native or some environment other than Go or Typescript/Javascript, it is possible there are issues with Bytes handling.
- You might want to ask in the Slack channel if this might be the case.
We will add to the above list of possible issues and fixes as we discover them, and attempt to close some of those potential gotchas along the way.
Thanks @carsonfarmer! One small point of clarification, this error seemed to be slightly different from the one we get if the node is down. For that we get Response closed without headers but this says Response closed without grpc-status (headers only). Not sure maybe they are both the same error under the hood but wanted to point that out in case it helps.
Just more info to add:
- In our particular case, those queries work most of the time and it was just the once or twice so far it happened (out of lets say a few hundred queries) so this points to something in the last couple of your bullets being the cause.
- With port 4006 open, CPU was at a constant 15% and network IO was maxed out at 5 Mbps so what I blocked that port since we are only accessing it via 6007 for the time being. This brought both CPU and network IO down so will see if that error comes up again (since network IO was maxed out, and there were no logs of the request on
threadsdside, thought maybe the request just didn't go through, then again if that was the case I suspect we would have seen this more often).
So looks like blocking port 4006 didn't prevent the same error from happening again so the theory that it might have been due to network load is gone. It's happening on the same spot as before, which is making this a threadsClient.find<> call. What I'll do is log the parameters we are passing into that for now and see if there is anything fishy.
I have seen the Response closed without grpc-status (Headers only) today while working with the local daemon. For this error, when I've seen it, it has been none fatal. It gives me the feeling that it should be straight-forward for most clients to just retry when this flavor of error occurs.
I have yet to see Response closed without headers
Just to add more info, today we got the error on a threadsClient.create create as well so it wasn't limited to threadsClient.find<>.
It gives me the feeling that it should be straight-forward for most clients to just retry when this flavor of error occurs.
We are getting this issues error from the threads-client js library, so it might be worth it if the client library also listens for this particular grpc error and retry at least once in cases like find() operations.
Did anyone ever discover a solution for this? I've tried using the standard textile/hub package as well as the textile/threads-client package but they produce the same end result
My basic stack is using Expo with Textile, and the issue only seems to be present in the iOS build. I first see it when you call the getToken function.
Here is my hacky solution, but I believe it could be returned to the byte handling
const getTokenFromAPI = (client, identity, count = 0) => client.getToken(identity)
.then()
.catch(err => {
if(count < 3) {
return getTokenFromAPI(client, identity, count + 1);
}
});
I see you mentioned Expo. This appears to be mostly limited to React Native these days. It appears to stem from connections being cut prematurely. Can you elaborate on what you mean by
I believe it could be returned to the byte handling
I'm scheduling some time to dedicate to tracing this bug. It is inconsistent and hard to nail down, which is why it is such an annoying and lingering bug :(
Yeah you are correct. It's funny though that I've never been able to reproduce this message in Expo running the Android simulator.
In regards to byte handling, I was just throwing that idea out there since you mentioned it above. My knowledge in regards to Textile is very limited as I only recently just started using it. I looked over your proposed reasons for problems above, but it was the only one that made any bit of sense given my truly limited scope in the implementation, since I am using React Native and trying to hit the hub API
I setup an example repo to help get you started: https://github.com/aetheode/expo-textile
Hi,
I am also using the Textile in React Native CLI project, and I am using the Account key in my app.
Here is example code:
let buckets = await Buckets.withKeyInfo({key: USER_API_KEY}); // This line runs successfully.
await buckets.getOrCreate('DummyBucket') // I am getting issue at this line
can anyone help me in this issue?
Note: Also tried User group key, but same issue.
Issue only occurs in iOS, working fine in Android.
@carsonfarmer, let me know if you need any other eyes on this issue. I have no experience with Go but am willing to learn if you point me in what you think might be the right direction
Maybe consider https://github.com/grpc/grpc-web ? Seems they have it working https://github.com/grpc/grpc-web/issues/90#issuecomment-699502164
Retries are no the solution;
[Fri Jul 23 2021 20:51:05.700] LOG starting step 0
[Fri Jul 23 2021 20:51:05.701] LOG retry[generateIdentity]: retries=3
[Fri Jul 23 2021 20:51:05.705] LOG retry[Client.withKeyInfo]: retries=3
[Fri Jul 23 2021 20:51:05.706] LOG retry[db.getToken]: retries=3
[Fri Jul 23 2021 20:51:06.641] LOG retry[db.getToken]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:06.844] LOG retry[db.getToken]: retries=2
[Fri Jul 23 2021 20:51:07.613] LOG retry[db.getToken]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:07.815] LOG retry[db.getToken]: retries=1
[Fri Jul 23 2021 20:51:08.620] LOG retry[db.getToken]: out of retries=0 [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:08.622] ERROR [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:38.947] LOG starting step 0
[Fri Jul 23 2021 20:51:38.948] LOG retry[generateIdentity]: retries=3
[Fri Jul 23 2021 20:51:38.952] LOG retry[Client.withKeyInfo]: retries=3
[Fri Jul 23 2021 20:51:38.953] LOG retry[db.getToken]: retries=3
[Fri Jul 23 2021 20:51:39.888] LOG retry[db.getToken]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:40.190] LOG retry[db.getToken]: retries=2
[Fri Jul 23 2021 20:51:40.954] LOG retry[db.getToken]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:41.406] LOG retry[db.getToken]: retries=1
[Fri Jul 23 2021 20:51:42.354] LOG retry[db.getToken]: out of retries=0 [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:42.354] ERROR [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:47.716] LOG starting step 0
[Fri Jul 23 2021 20:51:47.716] LOG retry[generateIdentity]: retries=10
[Fri Jul 23 2021 20:51:47.720] LOG retry[Client.withKeyInfo]: retries=10
[Fri Jul 23 2021 20:51:47.721] LOG retry[db.getToken]: retries=10
[Fri Jul 23 2021 20:51:48.591] LOG retry[db.getToken]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:48.895] LOG retry[db.getToken]: retries=9
[Fri Jul 23 2021 20:51:49.655] LOG retry[db.getToken]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:50.107] LOG retry[db.getToken]: retries=8
[Fri Jul 23 2021 20:51:52.830] LOG starting step 1
[Fri Jul 23 2021 20:51:52.830] LOG retry[getCachedUserThread]: retries=10
[Fri Jul 23 2021 20:51:53.298] LOG starting step 2
[Fri Jul 23 2021 20:51:53.299] LOG retry[db.create]: retries=10
[Fri Jul 23 2021 20:51:54.520] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:54.354] LOG retry[db.create]: retries=9
[Fri Jul 23 2021 20:51:55.276] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:55.727] LOG retry[db.create]: retries=8
[Fri Jul 23 2021 20:51:56.625] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:57.310] LOG retry[db.create]: retries=7
[Fri Jul 23 2021 20:51:58.187] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:51:59.200] LOG retry[db.create]: retries=6
[Fri Jul 23 2021 20:52:00.119] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:01.639] LOG retry[db.create]: retries=5
[Fri Jul 23 2021 20:52:02.558] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:04.837] LOG retry[db.create]: retries=4
[Fri Jul 23 2021 20:52:05.608] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:09.280] LOG retry[db.create]: retries=3
[Fri Jul 23 2021 20:52:09.790] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:14.917] LOG retry[db.create]: retries=2
[Fri Jul 23 2021 20:52:15.688] LOG retry[db.create]: retrying... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:23.378] LOG retry[db.create]: retries=1
[Fri Jul 23 2021 20:52:24.304] LOG retry[db.create]: out of retries=0 [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:24.305] ERROR [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:26.319] LOG starting step 0
[Fri Jul 23 2021 20:52:26.320] LOG retry[generateIdentity]: retries=10
[Fri Jul 23 2021 20:52:26.324] LOG retry[Client.withKeyInfo]: retries=10
[Fri Jul 23 2021 20:52:26.325] LOG retry[db.getToken]: retries=10
[Fri Jul 23 2021 20:52:27.262] LOG retry[db.getToken]: retrying in 300 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:27.564] LOG retry[db.getToken]: retries=9
[Fri Jul 23 2021 20:52:28.344] LOG retry[db.getToken]: retrying in 450 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:28.797] LOG retry[db.getToken]: retries=8
[Fri Jul 23 2021 20:52:29.572] LOG retry[db.getToken]: retrying in 675 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:30.258] LOG retry[db.getToken]: retries=7
[Fri Jul 23 2021 20:52:32.242] LOG starting step 1
[Fri Jul 23 2021 20:52:32.242] LOG retry[getCachedUserThread]: retries=10
[Fri Jul 23 2021 20:52:33.457] LOG starting step 2
[Fri Jul 23 2021 20:52:33.458] LOG retry[db.create]: retries=10
[Fri Jul 23 2021 20:52:34.361] LOG retry[db.create]: retrying in 300 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:34.662] LOG retry[db.create]: retries=9
[Fri Jul 23 2021 20:52:35.414] LOG retry[db.create]: retrying in 450 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:35.866] LOG retry[db.create]: retries=8
[Fri Jul 23 2021 20:52:36.631] LOG retry[db.create]: retrying in 675 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:37.316] LOG retry[db.create]: retries=7
[Fri Jul 23 2021 20:52:38.730] LOG retry[db.create]: retrying in 1012.5 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:39.870] LOG retry[db.create]: retries=6
[Fri Jul 23 2021 20:52:41.219] LOG starting step 3
[Fri Jul 23 2021 20:52:41.220] LOG retry[db.find]: retries=10
[Fri Jul 23 2021 20:52:41.911] LOG retry[db.find]: retrying in 300 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:42.213] LOG retry[db.find]: retries=9
[Fri Jul 23 2021 20:52:42.915] LOG retry[db.find]: retrying in 450 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:43.367] LOG retry[db.find]: retries=8
[Fri Jul 23 2021 20:52:44.151] LOG retry[db.find]: retrying in 675 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:44.838] LOG retry[db.find]: retries=7
[Fri Jul 23 2021 20:52:45.520] LOG retry[db.find]: retrying in 1012.5 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:46.535] LOG retry[db.find]: retries=6
[Fri Jul 23 2021 20:52:47.337] LOG retry[db.find]: retrying in 1518.75 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:48.855] LOG retry[db.find]: retries=5
[Fri Jul 23 2021 20:52:49.545] LOG retry[db.find]: retrying in 2278.125 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:51.825] LOG retry[db.find]: retries=4
[Fri Jul 23 2021 20:52:52.537] LOG retry[db.find]: retrying in 3417.1875 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:52:55.954] LOG retry[db.find]: retries=3
[Fri Jul 23 2021 20:52:56.640] LOG retry[db.find]: retrying in 5125.78125 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:53:01.766] LOG retry[db.find]: retries=2
[Fri Jul 23 2021 20:53:02.599] LOG retry[db.find]: retrying in 7688.671875 ms... previous error: [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:53:10.290] LOG retry[db.find]: retries=1
[Fri Jul 23 2021 20:53:11.111] LOG retry[db.find]: out of retries=0 [Error: Response closed without grpc-status (Headers only)]
[Fri Jul 23 2021 20:53:11.112] ERROR [Error: Response closed without grpc-status (Headers only)]
This might be the cause?
https://github.com/textileio/js-textile/blob/master/packages/grpc-connection/src/index.ts#L25-L26
When maybe this might make more sense? https://github.com/improbable-eng/grpc-web/tree/master/client/grpc-web-react-native-transport