imageproxy

imageproxy copied to clipboard

After using imageproxy for 2 weeks it looks like it restarts a lot because of OOMKilled (out of memory errors).

Here's the number of restarts:

kubectl get pods -l app=imageproxy
NAME                          READY     STATUS    RESTARTS   AGE
imageproxy-3940674779-nm521   1/1       Running   54         12d
imageproxy-3940674779-swbgj   1/1       Running   57         12d

Here's the description of one pod:

Name:           imageproxy-3940674779-swbgj
Namespace:      default
Node:           NODE_HOST/XXXXX
Start Time:     Wed, 07 Jun 2017 18:21:42 XXXX
Labels:         app=imageproxy
                pod-template-hash=XXXX
Status:         Running
IP:             XXXXX
Controllers:    ReplicaSet/imageproxy-3940674779
Containers:
  imageproxy:
    Container ID: XXXX
    Image:              imageproxy
    Image ID:  XXXX
    Port:               80/TCP
    Args:
      -addr
      0.0.0.0:80
      -cache
      /tmp/imageproxycache
    State:              Running
      Started:          Tue, 20 Jun 2017 08:14:33 XXXX
    Last State:         Terminated
      Reason:           OOMKilled
      Exit Code:        137
      Started:          Mon, 19 Jun 2017 17:17:43 XXXX
      Finished:         Tue, 20 Jun 2017 08:14:32 XXX
    Ready:              True
    Restart Count:      57
    Limits:
      cpu:      2
      memory:   1Gi
    Requests:
      cpu:              200m
      memory:           512Mi
    Liveness:           http-get http://:80/health-check delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:          http-get http://:80/health-check delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:        <none>
Conditions:
  Type          Status
  Initialized   True
  Ready         True
  PodScheduled  True
QoS Class:      Burstable
Node-Selectors: <none>
Tolerations:    <none>
Events:         <none>

The logs are only after restart and there is no hint as to why there so much memory being used. The upper memory limit (from the config) is 1Gi. Probably a memory leak right? Any idea how to diagnose this?

likely related to #92? Try with a build from current master.

I've rebuilt from master, will let it run for a day or so and get back here

I'm still getting restarts even with the HEAD:

    State:		Running
      Started:		Thu, 22 Jun 2017 12:43:16 -XXXX
    Last State:		Terminated
      Reason:		OOMKilled
      Exit Code:	137
      Started:		Thu, 22 Jun 2017 11:03:39 -XXXX
      Finished:		Thu, 22 Jun 2017 12:43:15 -XXXX
    Ready:		True
    Restart Count:	5

It's been about 1 year or so since I touched Go so I'm not entirely up to date on how to diagnose this. Any help I can get with this would be great! Thanks.

I have the same issue with the current latest commit (e7f9017). I originally did not put memory limits on my pods, but imageproxy would end up hogging the memory.

I added limits to have imageproxy killed and now it gets killed all the time.

I start imageproxy with this: imageproxy -addr 0.0.0.0:80 -cache memory:100:4h

The memory limit in k8s is 512Mi.

Here's the state of things on my end currently:

NAME                                         READY  STATUS             RESTARTS    AGE
po/production-imageproxy-746b675cbc-5tg7z     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-6rp9r     0/1   CrashLoopBackOff   12          1h
po/production-imageproxy-746b675cbc-7h5xz     0/1   CrashLoopBackOff   10          1h
po/production-imageproxy-746b675cbc-7x8wx     0/1   CrashLoopBackOff   10          1h
po/production-imageproxy-746b675cbc-84brs     0/1   CrashLoopBackOff   8           1h
po/production-imageproxy-746b675cbc-9sqlg     0/1   CrashLoopBackOff   10          1h
po/production-imageproxy-746b675cbc-chpkb     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-hx7vl     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-j9d5c     0/1   CrashLoopBackOff   7           1h
po/production-imageproxy-746b675cbc-k2w28     0/1   CrashLoopBackOff   8           1h
po/production-imageproxy-746b675cbc-kfndx     0/1   CrashLoopBackOff   11          1h
po/production-imageproxy-746b675cbc-m4rxs     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-m6rsx     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-phd82     0/1   CrashLoopBackOff   7           1h
po/production-imageproxy-746b675cbc-skmxx     1/1   Running            9           1h
po/production-imageproxy-746b675cbc-v94hv     0/1   CrashLoopBackOff   7           1h
po/production-imageproxy-746b675cbc-w46tq     1/1   Running            9           1h
po/production-imageproxy-746b675cbc-w52ks     1/1   Running            9           1h

I'm also running into this. With memory limit of 1Gi, very frequent restart. Increased to 2Gi, less frequent but still. Now at 3Gi and it restarted once after 20 minute...

When loading a web page with a lot of proxy URLs so that the browser attempts to download them all immediately, the process is killed because it's out of memory. I don't know how to get around this, since it's not something I can control in browsers. It seems like imageproxy should be rate limiting these requests at some point to avoid OOM issues.

@aaronpk, yeah that sounds like a perfectly reasonable request. Tracking in #200