scrapoxy icon indicating copy to clipboard operation
scrapoxy copied to clipboard

Published 20 hours ago • verified publisher icon fabienvauchelles

How to limit the number of instances that get created in DigitalOcean

Open shairyar opened this issue 7 years ago • 3 comments

Hi,

I am looking for a way to limit the number of droplets that get created in DigitalOcean so while looking at the documentaion I came across this link however that does not seem to work, right now I am seeing 10 droplets and i want to reduce that to 2 if possible, I am just playing around with this library

{
    "commander": {
        "password": "password"
    },
    "instance": {
        "port": 3128,
        "scaling": {
            "min": 1,
            "max": 2
        },
        "log": {
            "path": "/log"
        }
    },
    "providers": [
        {
            "type": "digitalocean",
            "token": "the-token-goes-here",
            "region": "SGP1",
            "size": "s-1vcpu-2gb",
            "sshKeyName": "ubuntu",
            "imageName": "name-of-image",
            "tags": "proxy,instance",
            "max": 2
        }
    ]
}

My understanding is by adding the max I should have been able to limit the droplets but I see no difference, is this the right please to add? What am i missing here?

I even tried changing the size to "size": "s-1vcpu-1gb" but that did not effect anything, still it is creating more instances that max specified

shairyar avatar Aug 01 '18 11:08 shairyar

Hello @shairyar ,

I cannot reproduce on DO. It is working fine for me.

Can you provide me a full logtrace ? I'll try to understand what it is happening.

Thanks, Fabien.

scrapoxy avatar Aug 17 '18 16:08 scrapoxy

Hi @fabienvauchelles

This is what I see in my terminal, if there is anything you need me to provide please let me know.

I got it to work with amazon which was a relief.

Just so that you know i tried disabling the firewall and also with firewall enabled and I added the port 3128 as well on DigitalOcean.

I followed the steps mentioned in documentation very carefully and i went through the process multiple times just to make sure I did not miss anything

~/Sites/scraper scrapoxy start conf.json -d

2018-08-19T08:11:15.479Z - info: [Main] The selected providers are digitalocean/SGP1
2018-08-19T08:11:15.525Z - debug: [Main] listen
2018-08-19T08:11:15.528Z - info: [Commander] GUI is available at http://localhost:8889
2018-08-19T08:11:15.529Z - debug: [Manager] start
2018-08-19T08:11:15.530Z - info: Proxy is listening at http://localhost:8888
2018-08-19T08:11:25.532Z - debug: [Manager] checkInstances
::1 - - [19/Aug/2018:08:11:26 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /styles/vendor-7ee78e5317.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /styles/app-bc17cddbef.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /scripts/vendor-4dbec0db50.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /scripts/app-e4523c691d.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /assets/fonts/icomoon.ttf?-mts32b HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /assets/fonts/OpenSans-Regular.woff HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /api/scaling HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /api/instances HTTP/1.1" 200 2 "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /assets/images/logo35h.png HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:11:28.037Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:28.037Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:28.038Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:11:35.536Z - debug: [Manager] checkInstances
2018-08-19T08:11:38.265Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:38.265Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:38.265Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:11:45.539Z - debug: [Manager] checkInstances
2018-08-19T08:11:47.289Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:47.289Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:47.289Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:11:55.539Z - debug: [Manager] checkInstances
2018-08-19T08:11:57.486Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:57.486Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:57.486Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:05.539Z - debug: [Manager] checkInstances
2018-08-19T08:12:08.846Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:08.846Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:08.847Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:15.542Z - debug: [Manager] checkInstances
::1 - - [19/Aug/2018:08:12:17 +0000] "GET /api/stats?retention=60000 HTTP/1.1" 200 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:12:17.804Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:17.804Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:17.804Z - debug: [ProviderDigitalOcean] createInstances: count=1
::1 - - [19/Aug/2018:08:12:20 +0000] "GET /api/stats?retention=18000000 HTTP/1.1" 200 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:21 +0000] "GET /api/stats?retention=86400000 HTTP/1.1" 200 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:12:25.546Z - debug: [Manager] checkInstances
2018-08-19T08:12:28.149Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:28.149Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:28.149Z - debug: [ProviderDigitalOcean] createInstances: count=1
::1 - - [19/Aug/2018:08:12:33 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /styles/vendor-7ee78e5317.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /styles/app-bc17cddbef.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /scripts/vendor-4dbec0db50.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /scripts/app-e4523c691d.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/fonts/OpenSans-Regular.woff HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/fonts/icomoon.ttf?-mts32b HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /api/scaling HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /api/instances HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/images/logo35h.png HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/images/logo32.png HTTP/1.1" 200 17953 "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:12:35.548Z - debug: [Manager] checkInstances
2018-08-19T08:12:37.794Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:37.794Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:37.794Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:45.551Z - debug: [Manager] checkInstances
2018-08-19T08:12:49.413Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:49.413Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:49.413Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:55.556Z - debug: [Manager] checkInstances
2018-08-19T08:12:58.958Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:58.958Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:58.958Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:13:05.559Z - debug: [Manager] checkInstances
^C2018-08-19T08:13:06.842Z - debug: [Main] shutdown
2018-08-19T08:13:06.842Z - debug: [Master] shutdown
2018-08-19T08:13:06.843Z - debug: [Manager] stop
2018-08-19T08:13:06.844Z - info: [Main] All instances are stopped.`

shairyar avatar Aug 19 '18 08:08 shairyar

I'm experiencing the same problem with DO... Scrapoxy seems to keep adding more and more droplets (way more than my limit) and it also is unable to stop them in the end, I had to stop all of them manually.

marcelgwerder avatar Oct 26 '18 21:10 marcelgwerder

corrected in 4.0.0

scrapoxy avatar Oct 04 '23 20:10 scrapoxy

Hey there! 🌟 Exciting news! Scrapoxy 4 is ready to rock 🚀. Check it out at Scrapoxy.io (explore the "get started" guide, deployment documentation, and more 📚). I can't wait to hear your feedback on this new version! Send me your coolest screenshots with as many proxies as possible! 📸💻 Join the Discord community if you have any questions or just want to chat. You can also open a GitHub issue for any bug or feature request 🐞✨. See you soon! 😎 Fabien

scrapoxy avatar Dec 12 '23 22:12 scrapoxy