meilisearch-php
meilisearch-php copied to clipboard
meilisearch
Error in the HTTP2 framing layer
Description I'm using Meilisearch with Laravel Scout. It has been working flawlessly for about the last year. I recently upgraded Laravel to version 9, but versions of Scout and meilisearch-php have not changed.
In the past few days, I'm getting frequent errors that look like this:
Error in the HTTP2 framing layer for "https://[MEILISEARCH SERVER]/indexes/[INDEX NAME]/documents?primaryKey=id"
There is a CommunicationException being thrown here.
The error doesn't happen every time, but it does happen frequently (perhaps 5-10% of indexing requests).
Environment (please complete the following information):
- OS: Ubuntu 18.04
- Meilisearch version: v0.23
- meilisearch-php version: v0.19.2
Hi @adamthehutt I don't have a good idea about your problem, but let's try something together (this problem only happened with you, the other users in our other SDKs didn't report anything at all).
Did you set up Meilisearch server with HTTP2 configuration? https://docs.meilisearch.com/learn/cookbooks/http2_ssl.html
@brunoocasali Sorry for the slow reply. Since logging this issue we have switched to the official cloud-hosted Meilisearch service and the error is still occurring. This is with version v0.27.2 of the server and v0.23.2 of the PHP client.
I assume the official cloud service has the HTTP2 layer configured correctly?
@adamthehutt I'm in touch with the team to try to solve this problem, and thanks a lot for adopting our cloud solution ❤️
Can you tell me which kind of HTTP client are you using? Guzzle and which version?
And @adamthehutt can you tell me how many concurrent requests you usually do? I don't have much confidence this is a problem related to the PHP integration itself. But yes in the server configuration, I think we are reaching some limits, so every piece of information you can share with me will be helpful.
@brunoocasali I'm using Guzzle version 7.4.3.
I'm not sure how many concurrent connections there are. The indexing happens in a queued job, so there definitely could be multiple requests in parallel. But we don't have a huge volume of data/activity so I'm sure you have other clients with many more concurrent connections.
For what it's worth, I believe the underlying error message is from curl.
@brunoocasali I have temporarily fixed the problem at an application level by changing the way we retry failed index jobs. When a job fails due to this error, the queue now delays a retry for 5 seconds, and then 30 seconds after a second failure. Since making that change, we haven't had any cases of an indexing job failing 3 times.
The Meilisearch error is still happening, of course, so I'd like to figure out why. But the above suggests that it must be a function of temporary server state (e.g., perhaps too many concurrent requests, as you suggest).
That's nice to hear @adamthehutt!
I'll let the issue open at least until I could try to reproduce your problem, since we have other priorities right now this will have to wait for a little.
Thanks a lot for your involvement 😃. Meilisearch is better with people like you! 💪
It's been a while since we don't comment on this issue. Did you update your meilisearch version these days @adamthehutt? Do you think this problem is still happening?
@brunoocasali thanks for checking in.
This issue is still happening. We're still using the cloud-hosted service, and it generally works great, with the exception of this issue. The temporary fix I described above is still in effect and still working. Since I posted that comment, there have been over 6,000 cases of this error happening. Every time the job is retried in 5 seconds and then 30 seconds. There has not been a single case of the same index job failing three times.
At this point the workaround is basically functional, so there's no emergency. It's annoying and would be great to fix, but it's not preventing us from using Meilisearch.
If it would help, I could provide some timestamps of some of those 6,000+ errors so that your engineers could compare with your logs.
@brunoocasali My apologies! I somehow missed your latest reply/question.
In Meilisearch Cloud the project is on 0.27.2. The php client is 0.23.3.
Having the same issue, also about 5% of times. 1.0.2 self-hosted with Laravel 10 and Scout
It would be nice to check if that happens with other Http libraries as well, if someone could do this test would be extremely helpful!
If you use guzzle or any other http library, check which curl-versions you are using. If it is older then 7.75 you might need to update curl. I did have the same issue as you reported but with another service. So it is not related to Meilisearch but to curl :)
Le me know if that helped.
That is also reported in https://curl.se/mail/lib-2021-05/0065.html
Thanks @azngeek!
@ahtinurme @adamthehutt please check the curl version, that could be the issue as stated by @azngeek. Let me know the results then!
I will close this issue due to inactivity. Feel free to reopen if needed!