node-horseman icon indicating copy to clipboard operation
node-horseman copied to clipboard

new Horseman ({proxy}) failed to get url

Open delormemarie240 opened this issue 9 years ago • 13 comments

Hello, Without proxy options in new Horseman, script is working great. But when I run this script I get this error: "Unhandled rejection Error: Failed to GET url: http://www.check2ip.com". When I turn proxyType as 'none' I access to check2ip but IP is not 142.91.204.5

var Horseman = require("node-horseman");

var horseman = new Horseman({
proxy: '142.91.204.5:22',

 proxyType: 'socks5',

 proxyAuth: 'root:msn43w1'

});


horseman
    .open('http://www.check2ip.com')
    .screenshot('check2ip.jpg')
    .log()
    .close();

delormemarie240 avatar Aug 09 '16 23:08 delormemarie240

Are you sure you have the right username, password, and proxy type? It seems like you're failing to connect to the proxy.

awlayton avatar Aug 10 '16 01:08 awlayton

yes I use right user name, pw and proxy type. I can connect putty or bitvise with it.

delormemarie240 avatar Aug 10 '16 08:08 delormemarie240

Can you post the output from running it with DEBUG='horseman' and BLUEBIRD_DEBUG=1 set?

awlayton avatar Aug 10 '16 16:08 awlayton

I im having this issue as well.

horseman .setup() creating phantom instance 21 +267ms
Trying to get: https://xxxxxxxxxxxxxxxxxxxxxxx, with proxy: xx.xx.xx.xx:xxxx
horseman phantom created +53ms
horseman phantom version 2.1.1 +7ms
horseman page created +6ms
horseman phantomjs onLoadFinished triggered +5ms success NaN
horseman phantom created +19ms
horseman injected jQuery +2ms
horseman phantom version 2.1.1 +1ms
horseman page created +6ms
horseman phantomjs onLoadFinished triggered +7ms success NaN
horseman injected bluebird +4ms
horseman phantom created +5ms
horseman phantom version 2.1.1 +3ms
horseman .userAgent() set +4ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman page created +1ms
horseman .open() +0ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman injected jQuery +3ms
horseman phantomjs onLoadFinished triggered +4ms success NaN
horseman injected bluebird +14ms
horseman injected jQuery +2ms
horseman .userAgent() set +4ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman .open() +1ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman injected bluebird +7ms
horseman .userAgent() set +5ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman .open() +0ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman phantom created +116ms
horseman phantom version 2.1.1 +3ms
horseman page created +4ms
horseman phantomjs onLoadFinished triggered +5ms success NaN
horseman injected jQuery +11ms
horseman injected bluebird +11ms
horseman .userAgent() set +7ms Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0
horseman .open() +0ms https://xxxxxxxxxxxxxxxxxxxxxxx
horseman phantomjs onLoadFinished triggered +376ms fail 1
[Error: Failed to GET url: https://xxxxxxxxxxxxxxxxxxxxxxx

when i access it without a proxy everything works, this seems to happen on https sites but not all.

notice the horseman phantomjs onLoadFinished triggered +4ms success and then horseman phantomjs onLoadFinished triggered +376ms fail 1

aryeharmon avatar Aug 11 '16 07:08 aryeharmon

I did a test and sites that only are available over https this always happens.

aryeharmon avatar Aug 11 '16 07:08 aryeharmon

Why is your proxy running on the SSH port @delormemarie240? Also, that IP does not seem to be up...

awlayton avatar Aug 11 '16 16:08 awlayton

Can you connect to HTTPS sites _without_ a proxy @aryeharmon?

awlayton avatar Aug 11 '16 16:08 awlayton

1)out put : horseman using PhantomJS from phantomjs-prebuilt module +0ms horseman .setup() creating phantom instance 1 +6ms horseman phantom created +180ms horseman phantom version 2.1.1 +15ms horseman page created +7ms horseman phantomjs onLoadFinished triggered +6ms success NaN horseman injected jQuery +14ms horseman .open() +3ms https://www.check2ip.com/ horseman phantomjs onLoadFinished triggered +54ms fail 1 horseman .close(). +5ms Unhandled rejection Error: Failed to GET url: https://www.check2ip.com/

  1. I run proxy on port 22 by default as it is working on bitvise or putty. Which proxy should I used ? 80 isn't working and 1080 access to Apache webpage server.

  2. I give you rights proxy informations : proxy: '142.91.104.1:80', proxyAuth: 'root:flipiflop'

delormemarie240 avatar Aug 12 '16 12:08 delormemarie240

80 is a weird port as well @delormemarie240 , that's HTTP. In fact, there's a web server running on 80. If you're just picking random ports use high numbers (like in the 1000's or higher).

I do not know anything about using bitvise or putty. Are you running the proxy on the machine that is running horseman?

awlayton avatar Aug 12 '16 18:08 awlayton

Yes I run proxy on the machine that is running horseman. As I use pm2 (process monitoring) I would like to - from one proxy - start many horseman instances on others proxies. It would be faster than start horseman instance from each proxy one by one ... Could you suggest to me which hosting supplier provides working proxy on horseman?

delormemarie240 avatar Aug 13 '16 12:08 delormemarie240

Horseman should work with any SOCKSv5 proxy. Right now my only guess as to what could be happening is either:

  1. Your proxy is not setup the way you think, or
  2. there is a problem with proxies in PhatnomJS.

I do not see how the horseman code could be causing this (it just passes the proxy options right to phantom), and I think 2 is _very_ unlikely.

awlayton avatar Aug 15 '16 15:08 awlayton

@delormemarie240, can you try setting the horseman timeout to something big, like 30 seconds, and retrying this?

johntitus avatar Aug 22 '16 23:08 johntitus

had the same issue and was just a timeout issue. when I did .waitForSelector('body')before everything else it worked fine.

spacedevin avatar Oct 03 '17 16:10 spacedevin