HTTP API for running scripts

[yosefy]

do you plan to add api for running script via http request?

[bugy]

Hi @yosefy, actually I was thinking about it, but I'm not sure about use cases.

And actually all the client-server communication is based on AJAX, so HTTP API is there (but undocumented). The only complication is websockets, which are used for script input/output and for script parameters information (now it's dynamic).

So, at the moment you can start/stop a script via HTTP request. However you would need a websocket to get the output: I stream the data as soon as it's available from the script, without waiting until it finishes. Sending output by HTTP request is a bit more complicated and I see 2 ways to do it:

• Send it as a stream with server sent events (SSE), but this is not supported by IE11/Edge and if used outside of browser would require advanced HTTP client/library
• Make a blocking call until script finishes and send the whole output in one go.

[yosefy]

i think blocking would be good for the most of the services

[yosefy]

amm can you share an example how to start script via http request? also can i get at least failed or finished now? or what will happen after sending request by curl for example

thanks

[yosefy]

also if i use websockets client like "https://github.com/aio-libs/aiohttp" so i will get all the output right?

[bugy]

Hi @yosefy, do a POST request to HOST/executions/start with all parameters as a form data, and a __script_name form data for the script name

E.g. curl -X POST -F '__script_name=Apartment Info' -F 'apartmentId=13697677' localhost:5000/executions/start It will start the script and return its ID

Then you could connect to HOST/executions/io/{ID} via websocket to get output (it will be in chunks and have some wrapping object) Any websocket client should be ok for it.

[yosefy]

and last question, if security is enabled? for example via google

thanks

[bugy]

Hi @yosefy, this is a bit complicated - authentication is based on a "token", which is stored as a cookie in a browser (and valid for 30 days atm).

So there are 2 ways to get this token:

• Easy way: copy this cookie from the browser every N days and use it for all API requests

• Hard way: perform authentication and store the token from your client. To get this token your need to send credential information to POST /login endpoint with a form data:

• for ldap it will be your username and password

• for google auth it is your code field, which you will get after redirection

So, for google auth you would need to manually go through authentication process with google redirection. This is out of Script server control, unfortunately, as per google oauth protocol (at least I couldn't find a way how to make it automacically).

So, I believe, for a REST client and google oauth, getting token from a browser is only viable option for now.

[yosefy]

ok so i thought i will run another script server with different conf (on different port with no auth) which will work with same scripts

the question is , can i still send username some how for authorisation? because scripts have

  "allowed_users": [
    "@group"
  ],

in their config

thanks

[bugy]

Unfortunately this is impossible. You can add your IP to the group and add this IP to the trusted IPs in the server configuration

[yosefy]

worked! thanks

On Thu, Feb 28, 2019 at 1:18 PM Iaroslav Shepilov [email protected] wrote:

Unfortunately this is impossible. You can add your IP to the group and add this IP to the trusted IPs in the server configuration

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-468236025, or mute the thread https://github.com/notifications/unsubscribe-auth/AMVy7VUoZNCkZir-L5YNQ_b9dz8CNb-5ks5vR7sNgaJpZM4YKh5y .

[hilafish]

Hi @bugy ,

First of all, thanks for this script-server. It's very useful :)

I’ve sent a curl POST like your example above. The script is running - I see the output of it. However, when running the curl, it doesn’t return any ID. Actually, no output is returned from running the curl. It’s problematic because my script takes time to run and I’d like to check when it’s finished. Can you attempt it on your side and see if you do get an ID?

the Curl I ran: curl -X POST -F ‘__script_name=script-name’ -F ‘param1=param1_value’ -F ‘param2=param2_value’ -F ‘param 3=param3_value’ localhost:5001/executions/start

5001 = script server is on port 5000 with google authentication. We have another script-server on port 5001 without google authentication.

Thanks Hila

[bugy]

Hi @hilafish , to be honest I was not able to reproduce the problem, I ran curl -X POST -F '__script_name=Multiple words' -F 'Param1=22' 192.168.2.115:5001/executions/start And it worked for me, returning execution ID in a response.

The only thing I noticed, is that you are probably using wrong quotes characters: the command with your quotes didn't work for me. But I was getting an error message in response: Script with name "None" not found

In your example it is ’ and for me worked only '

[hilafish]

Probably in the copy process it switched the ' to ’ , but it does work.

the script I'm running (json file, actually) runs shell script that invokes ansible playbook command. Any way this could be the reason why it doesn't return execution ID?

[bugy]

Hi @hilafish, good point about the running script.

Script-server should always return assigned execution id. Did you try to execute anything from GUI?

Could you try to run the command from a tool like Postman? Is return code 200?

[hilafish]

If I run the curl with -i , I get: HTTP/1.1 200 OK Server: TornadoServer/5.1.1 Content-Type: text/html; charset=UTF-8 Date: Sun, 05 May 2019 14:16:09 GMT X-Frame-Options: DENY Content-Length: 1

The issue isn't that it's not running- it is running. But no response is returned.

In the server.log I do see the execution ID- 2019-05-05 13:58:22,433 [tornado.access.INFO] 200 POST /executions/cleanup/179 (10.177.158.121) 1.52ms

[bugy]

Hi @hilafish, I see there is "Content-Length: 1" in the response, so I expect something to be in response.

Could it be, that you overlooked the id at the beginning of the line, for me it looks like:

me@my-pc:~$ curl -X POST -F '__script_name=Multiple words' -F 'Param1=22' 192.168.2.115:5001/executions/start
2me@my-pc:~$

To be honest I have no other ideas why it could happen :( Tornado version looks good, for responses I always write a generated ID or some error.

One more thing to check, in the log files (logs\processes) you should be able to see corresponding files for each script run. Could you open any of those, which were started with CURL and check, if ID is there indeed (should be the first line, id: xxx).

Another way is to find in server log a record, looking like: 2019-05-05 16:30:00,780 [script_server.execution_service.INFO] Calling script #47: ....

In the server.log I do see the execution ID-
2019-05-05 13:58:22,433 [tornado.access.INFO] 200 POST /executions/cleanup/179 (10.177.158.121) 1.52ms

This REST endpoint is called from GUI

[hilafish]

Hi,

It works now, returns execution ID - I had to tweak the conf-api.json file , although haven't changed much. After restart of the service it worked.

Thanks for your quick support!

[bugy]

Glad that it worked :) Sorry for not helping you much

[yosefy]

hi @bugy :) do you plan to continue with this we need to fire up scripts on github PR and PUSH so i thought... :)

[bugy]

Hi, I'm planning to, but not in the nearest future, to be honest :(

yosefy [email protected] schrieb am Mo., 18. Mai 2020, 08:22:

hi @bugy https://github.com/bugy :) do you plan to continue with this we need to fire up scripts on github PR and PUSH so i thought... :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-629971087, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJXPJILSBJLU5EQQVMTHXTRSDHZLANCNFSM4GBKDZZA .

[bugy]

There is an API for starting a script and getting an execution status. So you could do busy waiting for the status after starting a script

Yaroslav Shepilov [email protected] schrieb am Mo., 18. Mai 2020, 08:31:

Hi, I'm planning to, but not in the nearest future, to be honest :(

yosefy [email protected] schrieb am Mo., 18. Mai 2020, 08:22:

hi @bugy https://github.com/bugy :) do you plan to continue with this we need to fire up scripts on github PR and PUSH so i thought... :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-629971087, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJXPJILSBJLU5EQQVMTHXTRSDHZLANCNFSM4GBKDZZA .

[yosefy]

amm exit status?

On Mon, May 18, 2020 at 9:33 AM Iaroslav Shepilov [email protected] wrote:

There is an API for starting a script and getting an execution status. So you could do busy waiting for the status after starting a script

Yaroslav Shepilov [email protected] schrieb am Mo., 18. Mai 2020, 08:31:

Hi, I'm planning to, but not in the nearest future, to be honest :(

yosefy [email protected] schrieb am Mo., 18. Mai 2020, 08:22:

hi @bugy https://github.com/bugy :) do you plan to continue with this we need to fire up scripts on github PR and PUSH so i thought... :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/bugy/script-server/issues/138#issuecomment-629971087>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAJXPJILSBJLU5EQQVMTHXTRSDHZLANCNFSM4GBKDZZA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-629975342, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADCXF3OMLCHAR2UVHKGQAMDRSDJERANCNFSM4GBKDZZA .

[bugy]

/executions/status/{id} will give you running or finished response /history/execution_log/long/{id} will give you full info about the script (the same as on history detail page)

[yosefy]

hi @bugy how are you can you please enable api point to get exit status anyway? i see that you have this info as it shows up in ui history page

so something like /executions/exitstatus/{id}

may be

it is important as we run script and we need to react differently if exit status is not 0 so finished is not enough

so can you help with this please?

thanks

[bugy]

Hi @yosefy I'll try to do in this or the next week

[yosefy]

just to update we are still hoping for this thing. can't use api from airflow otherwise thanks :)

[bugy]

Oops, sorry Could you remind me please, why /history/execution_log/long/{id} doesn't work for you?

[yosefy]

I need to know that script exited with 0 Meaning all good and we can continue

On Wed, Jan 27, 2021, 3:10 PM Iaroslav Shepilov [email protected] wrote:

Oops, sorry Could you remind me please, why /history/execution_log/long/{id} doesn't work for you?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-768274798, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADCXF3JHBZCIEV5JNTGWZCLS4AGEDANCNFSM4GBKDZZA .

[bugy]

But there is also exitCode included. This is an example from script-server.net for me:

{
  "id": "20834",
  "startTime": "2021-01-27T13:19:32.733000+00:00",
  "user": "abcde",
  "script": "Ping",
  "status": "finished",
  "exitCode": -15,
  "command": "ping -c 3 -i 1 google.com",
  "log": "PING google.com (172.217.3.174) 56(84) bytes of data.\r\n64 bytes from sea15s11-in-f14.1e100.net (172.217.3.174): icmp_seq=1 ttl=98 time=80.8 ms\n\n>> STOPPED BY USER\n"
}

[yosefy]

Oh I have missed it somehow. That is good enough

Thanks :)

On Wed, Jan 27, 2021, 3:22 PM Iaroslav Shepilov [email protected] wrote:

But there is also exitCode included. This is an example from script-server.net for me:

{ "id": "20834", "startTime": "2021-01-27T13:19:32.733000+00:00", "user": "abcde", "script": "Ping", "status": "finished", "exitCode": -15, "command": "ping -c 3 -i 1 google.com", "log": "PING google.com (172.217.3.174) 56(84) bytes of data.\r\n64 bytes from sea15s11-in-f14.1e100.net (172.217.3.174): icmp_seq=1 ttl=98 time=80.8 ms\n\n>> STOPPED BY USER\n" }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-768281471, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADCXF3MY37ZJFIDZX2GQNVDS4AHRDANCNFSM4GBKDZZA .

[bugy]

Hi everyone here. I'd like to share, that for the dev version I implemented XSRF protection (#79). This is not backward compatible from API perspective. Now clients should send X-XSRFToken with every POST/PUT request. This token can be get from _xsrf cookie. Alternatively, xsrf_protections can be relaxed (by using headers) or disabled: https://github.com/bugy/script-server/wiki/Server-configuration#--xsrf_protection I would recommend going header approach, if default token protection is too complicated.

[matiasrivera]

Hi @yosefy, this is a bit complicated - authentication is based on a "token", which is stored as a cookie in a browser (and valid for 30 days atm).

So there are 2 ways to get this token:

• Easy way: copy this cookie from the browser every N days and use it for all API requests
• Hard way: perform authentication and store the token from your client. To get this token your need to send credential information to POST /login endpoint with a form data:
• for ldap it will be your username and password
• for google auth it is your code field, which you will get after redirection

So, for google auth you would need to manually go through authentication process with google redirection. This is out of Script server control, unfortunately, as per google oauth protocol (at least I couldn't find a way how to make it automacically).

So, I believe, for a REST client and google oauth, getting token from a browser is only viable option for now.

Hi @bugy when you say you have two options to access the authentication token. I've tried both and failed

I have tried looking at my browser cookies and I don't see a "token" cookie, only 4 different ones "JSESSIOND","_xsrf","sessionID" and "username" one in two different servers (one configured with htpasswd and another with google auth)

I also tried on the server configured with htpasswd making a post request to /login with username and password parameters but I always get a "Forbbiden" response.

Any Ideas?

Thanks in advance

Matias

[bugy]

Hi @matiasrivera, try using "username" cookie. It should be enough. But you would have to refresh it every N days

[yosefy]

I have finished with two script servers on same runners and scripts but different port and different auth. One with Google for users, one htpassortpasswordd for API calls. So it's just two systemd service files that differ and conf.json

On Fri, Jan 21, 2022, 1:33 PM Iaroslav Shepilov @.***> wrote:

Hi @matiasrivera https://github.com/matiasrivera, try using "username" cookie. It should be enough. But you would have to refresh it every N days

— Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-1018426548, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADCXF3KW3IA6VLEQHMEVFETUXE77NANCNFSM4GBKDZZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[matiasrivera]

Hi, Unfortunately it seems that no matter if I add the username cookie I still get Forbbiden for every request.

Here are my request headers [image: image.png]

And here is the response

HTTPHeaderDict({'Server': 'TornadoServer/6.1', 'Content-Type': 'text/html; charset=UTF-8', 'Date': 'Fri, 21 Jan 2022 13:25:21 GMT', 'X-Frame-Options': 'DENY', 'Content-Length': '9', 'Vary': 'Accept-Encoding', 'Set-Cookie': '_xsrf=2|a38bebe2|3e24579d7c5c69a6cce9fcf219023c53|1642771521; Path=/'})

Every time I hit the server I get 403 Forbbiden response.

Any ideas?

Matias

On Fri, Jan 21, 2022 at 9:15 AM yosefy @.***> wrote:

I have finished with two script servers on same runners and scripts but different port and different auth. One with Google for users, one htpassortpasswordd for API calls. So it's just two systemd service files that differ and conf.json

On Fri, Jan 21, 2022, 1:33 PM Iaroslav Shepilov @.***> wrote:

Hi @matiasrivera https://github.com/matiasrivera, try using "username" cookie. It should be enough. But you would have to refresh it every N days

— Reply to this email directly, view it on GitHub < https://github.com/bugy/script-server/issues/138#issuecomment-1018426548>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADCXF3KW3IA6VLEQHMEVFETUXE77NANCNFSM4GBKDZZA

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-1018454456, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMJSCMR42ZQWKWAIJY5P3UXFE4PANCNFSM4GBKDZZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[bugy]

@matiasrivera I cannot see the attached image

[matiasrivera]

Here's the attached request images.

Regards,

Matias

On Fri, Jan 21, 2022 at 10:40 AM Iaroslav Shepilov @.***> wrote:

@matiasrivera https://github.com/matiasrivera I cannot see the attached image

— Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-1018514294, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMJSDMMGL3RLG2LP4KBJDUXFO5BANCNFSM4GBKDZZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[bugy]

Still not here. Please send it to my email: [email protected]

[matiasrivera]

Attaching the image once again

Thanks

Matias

On Fri, Jan 21, 2022 at 11:05 AM Iaroslav Shepilov @.***> wrote:

Still not here. Please send it to my email: @.***

— Reply to this email directly, view it on GitHub https://github.com/bugy/script-server/issues/138#issuecomment-1018533198, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEMJSHB36B34F3A7SSZAC3UXFR3FANCNFSM4GBKDZZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[bugy]

Hi @matiasrivera, i'm really sorry, I totally forgot about it: please check this message: https://github.com/bugy/script-server/issues/138#issuecomment-821554743 You have to submit a XSRF token (or disable it)

[matiasrivera]

Hi @bugy I successfully disabled xsfr and now I am being able to loggin in /login with my credentials and getting the username cookie back.

Nonetheless I'm getting an Internal Server Error when trying to launch a script. Looking at script server log I see the following stack trace. Any ideas?

Traceback (most recent call last): File "/home/server/python/trading/lib/python3.9/site-packages/tornado/web.py", line 1681, in _execute result = self.prepare() File "/home/server/python/script-server/src/web/web_auth_utils.py", line 46, in wrapper return func(self, *args, **kwargs) File "/home/server/python/script-server/src/web/server.py", line 347, in prepare self.form_reader = StreamingFormReader(self.request.headers, upload_folder) File "/home/server/python/script-server/src/web/streaming_form_reader.py", line 50, in init raise Exception('Unsupported content type: ' + content_type) Exception: Unsupported content type: application/x-www-form-urlencoded 2022-01-21 17:19:38,325 [tornado.access.ERROR] 500 POST /executions/start (192.168.0.1) 8.12ms

[vanscurvy]

Hi @bugy - first of all, great app! I am trying to make use of this API call and have had success with Postman, but when I replicate the same POST call with the built-in API functionality of another program (Appsmith), it isn't working for some reason.

http://<script-server ip>:5000/executions/start with a form-data text payload of key:__script_name, value:test

I have a ticket open with them but am trying to further characterize the problem. Here is a screenshot of the script-server output from a successful Postman call and a failed Appsmith call done consecutively. I have PCAPs of both API calls and see some differences but I am not sure how to interpret (can share privately if that would be helpful). Any ideas? Like the above post, I am hoping it's possible to infer something from the stack trace error.

Also, @matiasrivera, not sure if you are still having an issue, but from your post it looks like the body of your POST call is x-www-form-urlencoded, whereas I believe it should be multipart form-data. At least that is what the script-server UI is doing.

[bugy]

Hi @vanscurvy, unfortunately it seems to be incompatibility between Script server and Appsmith. Script server expects the following header: 'content-type: multipart/form-data; boundary=----WebKitFormBoundaryzEGlClez0qGU5zD8'

and the data structure like this '------WebKitFormBoundaryzEGlClez0qGU5zD8\r\nContent-Disposition: form-data; name="operand 1"\r\n\r\n1\r\n

(boundary token would be different for every session/browser, but it's important that it matches the token in the data field). For whatever reason, this structure is different for Appsmith. Is there any way to debug a raw request, which is sent by Appsmith?

[vanscurvy]

@bugy - thanks for the response. I've asked Appsmith - will report back. Should I be able to set this, including the data structure, with headers? Appsmith has an interface that is very similar to Postman, and headers can be manually set.

[bugy]

I don't think that the headers would help. There should be boundary token in the header and the same one in the data. If Appsmith is not sending it, then you cannot fix it in headers only. If you can specify raw data content (instead of key-value form), then you would be able to submit it. But it would look weird

[vanscurvy]

It is possible to send raw data content. It's not clear to me how I should format the raw data - do you mind providing an example?

[bugy]

@vanscurvy it looks like this (you can see it in a browser dev tools, I copied it from https://script-server.net/index.html#/Ping):

------WebKitFormBoundaryo4HY2qwFibq79miB
Content-Disposition: form-data; name="Count"

3
------WebKitFormBoundaryo4HY2qwFibq79miB
Content-Disposition: form-data; name="Interval (sec)"

1
------WebKitFormBoundaryo4HY2qwFibq79miB
Content-Disposition: form-data; name="Print timestamp"

false
------WebKitFormBoundaryo4HY2qwFibq79miB
Content-Disposition: form-data; name="destination"

google.com
------WebKitFormBoundaryo4HY2qwFibq79miB
Content-Disposition: form-data; name="__script_name"

Ping
------WebKitFormBoundaryo4HY2qwFibq79miB--

It's important, that every new line is separated by \r\n combination, not just \n

You can see that every chunk is started with ------WebKitFormBoundaryo4HY2qwFibq79miB token (which is random for every request). And the whole body is ended with extra --

[vanscurvy]

Ok, I will give this a shot. Thank you @bugy !

[dutangp]

I am not sure if this thread is still in progress but I found the way to request execution via python requests

of course there is all the login process and cookie

import requests

headers = {"x-requested-with": "XMLHttpRequest"}
files = {'__script_name': (None, 'colortest')}
r = requests.post('https://script-server.net/executions/start', files=files, headers=headers)
r.json()