imageproxy
imageproxy copied to clipboard
willnorris
Limit number of the transformations threads
When a search bot requests many images at the same time and these images are not in the cache, all transformation goroutines start in parallel causing the huge memory allocation, that leads to the termination of the process by OOM.
This fix is adding the limiter in the TransformingTransport that limits the number of running transformation threads to the number of logical CPUs. The metric http_requests_in_flight is adding as well to monitor the length of the requests queue.
Fixes #200.
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
:memo: Please visit https://cla.developers.google.com/ to sign.
Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.
What to do if you already signed the CLA
Individual signers
- It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.
Corporate signers
- Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
- The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
- The email used to register you as an authorized contributor must also be attached to your GitHub account.
ℹ️ Googlers: Go here for more info.
Codecov Report
Merging #235 into master will increase coverage by
0.17%. The diff coverage is100.00%.
@@ Coverage Diff @@
## master #235 +/- ##
==========================================
+ Coverage 89.16% 89.34% +0.17%
==========================================
Files 6 6
Lines 674 685 +11
==========================================
+ Hits 601 612 +11
Misses 50 50
Partials 23 23
| Impacted Files | Coverage Δ | |
|---|---|---|
| imageproxy.go | 85.00% <100.00%> (+0.55%) |
:arrow_up: |
| metrics.go | 100.00% <100.00%> (ø) |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 3c7d08f...b2a9772. Read the comment docs.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hey @striker2000, thanks for this patch! Sorry it's taken a couple of week to get back to you.
I think this is certainly a good approach; my only reservation is about where exactly inside of TransformingTransport.RoundTrip that we block. I haven't had a chance to profile this code to know exactly where the out of memory issue in #200 is occurring. If it's inside the cache, then where you have the limiter check is probably right. But if it's actually happening during the transform step, then we probably want to block right before calling Transform. That way, the should304 call can still return 304 responses without tying up the limiter.
Does that make sense?
We're seeing fairly frequent OOM's (~10 a day) which are likely related to transformations. Is anyone using this patch successfully?
@mdkent thsi PR has a problem with inflight request.
I have updated with the current main branch and this PR. https://github.com/Jorgevillada/imageproxy
error copying response: readfrom tcp 127.0.0.1:8080->127.0.0.1:53150: write tcp 127.0.0.1:8080->127.0.0.1:53150: write: broken pipe
Some request never finish.
with main branch, the behavior with the ram is weird with invalid requests.
