IOT endpoint Timed out after 30sec. Connection couldn't be established.

[stevenfitzpatrick]

Hi,

I keep running into the following issue when trying to call the proxy chrome set up when I try call chromeless with the remote set up.

"Timed out after 30sec. Connection couldn't be established."

I tried from Windows, Ubuntu and from another AWS Lambda always resulting in the above error message.

Have completed step 1 for the proxy set up and deployed The Proxy service to AWS Lambda.

Steps I have done

• Changed the url for awsIotHost in the serverless.yml
• Changed region in the serverless.yml
• ran npm run deploy successfully and made note of the api key and url.

But the after that I am unable to call this proxy without running into the error "Timed out after 30sec. Connection couldn't be established.".

The script I am running is pretty simple i.e.

  const chromeless = new Chromeless({
    remote: {
      endpointUrl: 'https://xxxxxxxxxx.execute-api.eu-west-1.amazonaws.com/dev',
      apiKey: mykey
    }
  });

const links = await chromeless
    .goto('https://www.google.com')
    .type('chromeless', 'input[name="q"]')
    .press(13)
    .wait('#resultStats')
    .evaluate(() => {
      // this will be executed in headless chrome
      const links = [].map.call(document.querySelectorAll('.g h3 a'), a => ({
        title: a.innerText,
        href: a.href
      }));
      return JSON.stringify(links);
    })
    // you can still use the method chaining API after evaluating
    // when you're done, at any time you can call `.then` (in our case `await`)
    .scrollTo(0, 1000);

Always results in

node chrome.js Error: Timed out after 30sec. Connection couldn't be established. at Timeout._onTimeout (C:\dev\cb-headless-chrome\node_modules\chromeless\dist\src\chrome\remote.js:73:52) at ontimeout (timers.js:488:11) at tryOnTimeout (timers.js:323:5) at Timer.listOnTimeout (timers.js:283:5)

Is there some kind of permission, role I need that maybe is not in the documentation ?

Would love some help since I have no idea how to progress on this, even some idea which url it can't connect to ?

EDIT : After some more debugging I am finding its the IOT Host url that I can't connect to i.e.

wss://xxxxxxx.iot.eu-west-1.amazon.com/.....

Is there a step I am missing to allow connection to this ? I can see when I put my IOT endpoint url I can see the same kind of timeout behaviour.

[adieuadieu]

Hi @stevenfitzpatrick Hm.. To help debug this, let's confirm that the session Lambda handler is working correctly. Could you try the curl command I describe in https://github.com/graphcool/chromeless/issues/58#issuecomment-318603329 and share all of the result here?

Also, you've mentioned that you changed the region in serverless.yml. Is your region eu-west-1?

[stevenfitzpatrick]

Yes region is eu-west-1

Output here

• timeout on name lookup is not supported
• Trying 54.192.28.58...
• TCP_NODELAY set
• Connected to xxxxxx.execute-api.eu-west-1.amazonaws.com (54.192.28.58) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
• successfully set certificate verify locations:
• CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt CApath: none
• TLSv1.2 (OUT), TLS header, Certificate Status (22):
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS change cipher, Client hello (1):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS change cipher, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
• ALPN, server accepted to use h2
• Server certificate:
• subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.execute-api.eu-west-1.amazonaws.com
• start date: May 30 00:00:00 2017 GMT
• expire date: Feb 28 23:59:59 2018 GMT
• subjectAltName: host "xxxxxx.execute-api.eu-west-1.amazonaws.com" matched cert's "*.execute-api.eu-west-1.amazonaws.com"
• issuer: C=US; O=Symantec Corporation; OU=Symantec Trust Network; CN=Symantec Class 3 Secure Server CA - G4
• SSL certificate verify ok.
• Using HTTP2, server supports multi-use
• Connection state changed (HTTP/2 confirmed)
• Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
• Using Stream ID: 1 (easy handle 0x68a6f0)

GET /dev HTTP/1.1 Host: xxxxxx.execute-api.eu-west-1.amazonaws.com User-Agent: curl/7.51.0 Accept: / x-api-key: lSExxxxxxP32pEyjsz7xxxxxx2xy79cbq

• Connection state changed (MAX_CONCURRENT_STREAMS updated)! < HTTP/2 200 < content-type: application/json < content-length: 812 < date: Sun, 30 Jul 2017 15:15:11 GMT < x-amzn-requestid: e1add1b3-7539-11e7-8f36-991948b29d85 < x-amzn-trace-id: sampled=0;root=1-597df7ff-2e7c5cabed1df2f207fe8858 < x-cache: Miss from cloudfront < via: 1.1 e5740b731d03d61279af4365058ca2f2.cloudfront.net (CloudFront) < x-amz-cf-id: Q6jCgVvwx62Ex4fAxxxxxxdvXh77exxxxxxqI51cGNPQ== < {"url":"wss://xxxxxx.iot.eu-west-1.amazon.com/mqtt?X-Amz-Date=20170730T151511Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJ7MYCYZUZTZ4LYWA%2F20170730%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=f1bf85968b0bd88626412c6acda946bd62a51a9fb5bb5bfe2949735150d923e6&X-Amz-Security-Token=FQoDYXdzEJD%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDFUUf%2BsRSvInzKqjLyL6AfemldFtN14xyBp57G5gP3amsxoJUpWnbdebTkIr9evdL4GgeklMaK1vLywOjp1p6c3hwTF7LdTCUqpWLfOinbGZazNfRTexc%2FLmfTk%2F%2FZvAFxxxxxxwPc784Ah9F4iR3h%2FF5L7zMJpnM50Y7T1vL9yVvJuzKjn7kExxxxxxO2aWXIs4bthHlz4y%2BQ6DagAysDjyWKNrI1Nnvl7OjdffyWDB8QnAtX9OJ7IBZZ0hIxiljyv5MyHKtQRfvptP7IRUvodxJ8kURIXNz23fMToX2Dapkfy0FVwcvxfDhv8AhUoi%2Bj3ywU%3D","channelId":"cj5qval12000401p49j1ks9rl"}* Curl_http_done: called premature == 0
• Connection #0 to host xxxxxx.execute-api.eu-west-1.amazonaws.com left intact

[stevenfitzpatrick]

One thing I didn't understand from the documentation is which you consider as the "Session" GET end point.

I.e after npm run deploy, you get the following end points.

endpoints: GET - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/version OPTIONS - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/ GET - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/

Is that last get end point the one you consider the session one ?

[adieuadieu]

@stevenfitzpatrick yes that's correct—GET /dev endpoint is the new-session endpoint. I'll clarify that in the documentation.

[stevenfitzpatrick]

The only other thing I wanted to check was inserting the IOT Endpoint in the serverless.yml. I simply replaced the variable

"${env:AWS_IOT_HOST}"

with string

"xxxxxxxxxxx.iot.eu-west-1.amazonaws.com"

Is that also correct ?

[stevenfitzpatrick]

The timeout seems to be happening here in chrome/remote.js

channel_1 = mqtt_1.connect(url, {
                          will: {
                            topic: 'chrome/last-will',
                            payload: JSON.stringify({ channelId: channelId_1 }),
                            qos: 1,
                            retain: false
                          }
                        });

Funny behaviour that I noticed when I print out the url it is connecting to.

wss://xxxxxxxxxx.iot.eu-west-1.amazon.com/mqtt?X-Amz-Date

The url has been changed from amazonaws.com to amazon.com at this point

[adieuadieu]

@stevenfitzpatrick yes that's also correct (the AWS_IOT_HOST change you've made in serverless.yml) 😄

[stevenfitzpatrick]

ok then I am kinda at a loss why it's not working. I am pretty sure it's due to this line

wss://xxxxxxxxx.eu-west-1.amazon.com

Is there maybe a role I need to add to my credentials, some kind of cert installed on my machine etc ?

Did the curl output help at all ?

[adieuadieu]

@stevenfitzpatrick about amazonaws.com -> amazon.com .. that might be the problem. As far as I know it should be amazonaws.com Something weird must be happening for the domain to get changed.. In aws4 package's code we use to create the presigned URL, there's no mention of amazon.com.

[stevenfitzpatrick]

My chromeless version is 1.0.1 and serverless is 1.18.1

[stevenfitzpatrick]

@adieuadieu Totally my fault I put down amazon.com instead of amazonaws.com in the serverless config.

[shawnliujw]

my serverless version 1.8.1 , when use curl , i got `< HTTP/1.1 200 OK < Content-Type: application/json < Content-Length: 815 < Connection: keep-alive < Date: Wed, 02 Aug 2017 05:25:16 GMT < x-amzn-RequestId: f7e6475d-7742-11e7-9732-d536e1f97d30 < X-Amzn-Trace-Id: sampled=0;root=1-5981623c-8fdf8fd359b663c2a046fa79 < X-Cache: Miss from cloudfront < Via: 1.1 296daa1256a646119edb8b5abbf1c774.cloudfront.net (CloudFront) < X-Amz-Cf-Id: 6WVkXcnEiqM5h5pu-EYaBESjSrUmfcJfPjy5sAOoAEzPFE9HwZEl4Q== <

• Connection #0 to host km83etka57.execute-api.eu-west-1.amazonaws.com left intact {"url":"wss://xxxxxxxx.iot.eu-west-1.amazonaws.com/mqtt?X-Amz-Date=20170802T052516Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAI4ZGA6WPV6G3Z6AQ%2F20170802%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=a0c1984f283eea68c77b93cx13bd16837f24dba75cfea4ea01cea&X-Amz-Security-Token=FQoDYXdzEM7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDJcSjIPaYhqUe44t3yL6AQZg7AXnEWaRJjBhYiysWjL5955OBlBFrgrz7UZY5s7DZSgj8ehhtaQRdB7Nx52w%2FucP9gQGARX1UCH%2FjUSEHuQtCrDLihM3zBkigXtL0%2F9Nzq9Lu%2FbWAuKJGUqOx8aA9Kzi%2BxxKFrGuIP0EyX6MU513rQ%2FdafH7r8LYQVTUkt1rxxxxxxZDya4AXI%2FSDC2FE0drpFHFNBVru6bjT33Yk1jFW130sOP4HlkGVXBoywc2bbu1Mlmz0NTcNaYwNF1ndDWErjHm9hh1Y42iBIgOLWplfTyhZD8%2B06dI%2FhME3kPc54bsoh7yFzAU%3D","channelId":"cj5ukjhzt000101p6gqqbt09p"}`

but still got 30 seconds timeout , could anyone help me figure it out ? thanks

Just debug , the channel go to offline directly before it connected

[FabioAntunes]

@adieuadieu I'm having the same issue: If I do the curl everything is fine:

*   Trying 54.230.11.103...
* TCP_NODELAY set
* Connected to xxxxxxx.execute-api.eu-west-2.amazonaws.com (54.230.11.103) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.eu-west-2.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /dev/ HTTP/1.1
> Host: xxxxxxx.execute-api.eu-west-2.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> x-api-key: MY_API_KEY
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 831
< Connection: keep-alive
< Date: Thu, 03 Aug 2017 14:30:37 GMT
< x-amzn-RequestId: 517f1e75-7858-11e7-8b92-1976755a6dfc
< X-Amzn-Trace-Id: sampled=0;root=1-5983338d-3baf6c0a22e7da0a000369e8
< X-Cache: Miss from cloudfront
< Via: 1.1 2297ae65d06c070d63c49ed7259f5fbb.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 8ioi901ViVi65L8YOH0pR9COW2YdxAmqeSUpTJoRTeJALmKvF_RmIQ==
<
* Curl_http_done: called premature == 0
* Connection #0 to host xxxxx.execute-api.eu-west-2.amazonaws.com left intact
{"url":"wss://https://xxxxx.iot.eu-west-2.amazonaws.com/mqtt?X-Amz-Date=20170803T143037Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASSAQEQIN6YSWTTMJH2A%2F20170803%2Feu-west-2%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=9b9fa351facf65c88364667acd84855a294d7d44cf08e7af4cf22b44621fa9dc&X-Amz-Security-Token=SOME_TOKEN","channelId":"cj5wjgoaf000201qcr7efo48q"}

Adding the debug: true I get:

WebSocket offline
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/fabio/playground/chromeless-test/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:469:11)
    at tryOnTimeout (timers.js:304:5)
    at Timer.listOnTimeout (timers.js:264:5)

The test I'm trying to run:

const Chromeless = require('chromeless').default

async function run() {
  const chromeless = new Chromeless({
    remote: {
      endpointUrl: 'https://xxxxx.execute-api.eu-west-2.amazonaws.com/dev/',
      apiKey: 'xxxxxx'
    },
    debug: true
  })

  const screenshot = await chromeless
    .goto('https://www.google.com')
    .type('chromeless', 'input[name="q"]')
    .press(13)
    .wait('#resultStats')
    .screenshot()

  console.log(screenshot) // prints local file path or S3 url

  await chromeless.end()
}

run().catch(console.error.bind(console))

I double tested the values on the endpointUrl and apiKey with curl command and they work, so there's no wrong copy paste stuff

[FabioAntunes]

I decided to redo everything again, this time I using a different time region, eu-west-1

Still getting timeouts, but a different message when using debug:

Connected to message broker.
Error: Timed out after 30sec. Connection couldn't be established.
    at Timeout._onTimeout (/Users/fabio/playground/chromeless-test/node_modules/chromeless/dist/src/chrome/remote.js:73:52)
    at ontimeout (timers.js:469:11)
    at tryOnTimeout (timers.js:304:5)
    at Timer.listOnTimeout (timers.js:264:5)

the curl works fine fine:

*   Trying 54.230.11.142...
* TCP_NODELAY set
* Connected to xxxxxxx.execute-api.eu-west-1.amazonaws.com (54.230.11.142) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.execute-api.eu-west-1.amazonaws.com
* Server certificate: Symantec Class 3 Secure Server CA - G4
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> GET /dev HTTP/1.1
> Host: xxxxx.execute-api.eu-west-1.amazonaws.com
> User-Agent: curl/7.51.0
> Accept: */*
> x-api-key: xxxxxx
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 825
< Connection: keep-alive
< Date: Thu, 03 Aug 2017 16:33:32 GMT
< x-amzn-RequestId: 7ce040e4-7869-11e7-a3e9-e326343e7894
< X-Amzn-Trace-Id: sampled=0;root=1-5983505c-2aa9848cd55bdb1bfaf046c8
< X-Cache: Miss from cloudfront
< Via: 1.1 7cddfedd82c4e2198668a64be4934fd0.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: jjwhqOOqqdiNzwvOvDPtn8XqlQSn1qgM4bCGVeDcz60T7A97QJd14A==
<
* Curl_http_done: called premature == 0
* Connection #0 to host xxxxxx.execute-api.eu-west-1.amazonaws.com left intact
{"url":"wss://xxxxxx.iot.eu-west-1.amazonaws.com/mqtt?X-Amz-Date=20170803T163332Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJUVCD2JRUATTLOUA%2F20170803%2Feu-west-1%2Fiotdevicegateway%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=3d1a5501384450b98cfde614ce023b3d8d304dc2fbae414e9d69845549ecffc4&X-Amz-Security-Token=token","channelId":"cj5wnuq9q000301qmplem18li"}

[adieuadieu]

Hi @FabioAntunes at the moment I'm not really sure what might be causing this. I have a number of Proxy improvements and refactors I'm working on which will provide more verbose debug output for tracking these issues down.

[danthegoodman]

We're getting the same error. Looking in the cloudwatch logs for the lambda run function, I'm seeing

Unable to import module 'src/run': Error

Follow up: After adding in some console logs into the compiled code, it appears that there is an error when requiring the chromeless package.

[danthegoodman]

It appears that this line is causing the lighthouse-logger to not be packaged, which is currently required by the chrome-launcher. After commenting it out on our end, we were able to get farther but something is still not right. Now we're seeing from the test suite:

TypeError: Cannot read property 'port' of undefined

There's no stacktrace attached to the error, so we're at a loss as to how to proceed.

[rfink]

Having an identical issue to @danthegoodman

[Kivol]

I downloaded the chromeless zip from S3 and extracted and npm install again with node.js 6.10.3 and uploaded again, and I was able to pass the errors that @danthegoodman and @rfink faced, but now my CloudWatch log shows these messages and errors:

@serverless-chrome/lambda Spawning headless shell
@serverless-chrome/lambda ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: spawn EACCES

And in my local shell, I'm still getting 30sec the timeout error.

[furkanarius]

I'm having the same problem with @Kivol, and still found no solution. In fact, I followed the exact the same steps, but no good.

[adieuadieu]

Hi @danthegoodman @rfink @Kivol @furkanarius the Unable to import module 'src/run' and 'port' of undefined errors are now fixed in master branch and will be available in the next release (v1.2) sometime later today/tomorrow. A temporary work-around is to downgrade the chromeless package to [email protected] in serverless/package.json.

[nbirkes]

Thanks @adieuadieu ! Downgrading to @1.0.1 worked for @danthegoodman and I.

[icarusmiles]

@adieuadieu I am still getting 'port' of undefined and I downgraded to chromeless 1.0.1. Is the next release still on track for release soon?

[adieuadieu]

v1.2.0 has been released. Please update and let us know how it goes. Remember to do a npm install/yarn install after fetching/pulling latest from the repository.

[mfrye]

I think I found one relevant bit here... Seems that the regions have to match currently. The iot endpoint has to be in the same region as the functions.

[scosbet]

I've updated to the latest version this morning but am still seeing the same Unable to import module 'src/run': Error issue.

I also tried going back to 1.0.1 as suggested as a workaround but am getting the same result.

Is there anything I could be doing wrong? Really not sure what else to try

[Kivol]

I've upgraded to v1.2.0 and tried removing the whole stack with ./node_modules/.bin/serverless and deployed again with npm run deploy.

I still see the following error from chromeless-serverless-dev-run lambda function log:

@serverless-chrome/lambda Spawning headless shell
@serverless-chrome/lambda ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: spawn EACCES

[Kivol]

I've resolved the error above. Since I deployed in Windows, the Chrome binary seems to lack the +x permission when uploaded to S3 and lambda. When I tried deploying in Linux it works fine now!

Thanks for the support.

[adieuadieu]

The Proxy won't work right when deployed from Windows 😞 . There's a known issue with Serverless packaging executables on Windows upstream in https://github.com/adieuadieu/serverless-chrome/issues/21 and upstream from there in https://github.com/serverless/serverless/issues/3557.

Two work-around solutions are in progress in serverless-chrome:

• download chrome on lambda invocation (ick)
• check chrome executable permission on invocation. If missing, copy chrome to /tmp (only place we have chmod permission), and chmod +x, then launch (ugh)

Both are lame, but it doesn't seem like the root issue will be addressed in https://github.com/serverless/serverless/issues/3557 anytime soon.

[icarusmiles]

I can confirm all issues resolved from me, using Linux and chromeless 1.2.0. Great job guys! This project is moving quickly.