allow install es by url

[theundefined]

Currently - there is a need to download plugin, and then put it into environment, and install it. It would be nice to allow install it directly by url like mentioned in https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/installation.html

[theundefined]

plugin install try to download plugins from paths like:

root@default-ubuntu-1604:~# /usr/share/elasticsearch/bin/plugin install sscarduzio/elasticsearch-readonlyrest-plugin/1.13.1_es2.3.5
-> Installing sscarduzio/elasticsearch-readonlyrest-plugin/1.13.1_es2.3.5...
Trying https://download.elastic.co/sscarduzio/elasticsearch-readonlyrest-plugin/elasticsearch-readonlyrest-plugin-1.13.1_es2.3.5.zip ...
Trying https://search.maven.org/remotecontent?filepath=sscarduzio/elasticsearch-readonlyrest-plugin/1.13.1_es2.3.5/elasticsearch-readonlyrest-plugin-1.13.1_es2.3.5.zip ...
Trying https://oss.sonatype.org/service/local/repositories/releases/content/sscarduzio/elasticsearch-readonlyrest-plugin/1.13.1_es2.3.5/elasticsearch-readonlyrest-plugin-1.13.1_es2.3.5.zip ...
Trying https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/archive/1.13.1_es2.3.5.zip ...
Trying https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/archive/master.zip ...
Downloading .............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONE
Verifying https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin/archive/master.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in plugin zip

[sscarduzio]

I do like this and it's fine:

bin/elasticsearch-plugin install "https://readonlyrest-data.s3-eu-west-1.amazonaws.com/build/1.13.1/readonlyrest-XXXX_es5.1.2.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJEKIPNTOTIVGQ4EQ/20170118/eu-west-1/s3/aws4_request&X-Amz-Date=20170118T062604Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=b7acf93829ae4afae0ecb00c9ff01b0dbe45d2a6df91606821284d2218093e3b"

PS: the URL above doesn't work, paste in your email's URL.

I opted to have an official website as a single point of download for various reasons:

• keep track of the number of downloads
• keep track of what versions are downloaded most (useful to decide when old ES branches can be unsuppoted)
• keep track of unique users downloads (having logged in users might be better in the future)
• Guarantee that users download ONLY the latest version (this is a security plugin, and this is important)
• Be able to notify you guys back when a vulnerability is fixed and you should upgrade

[theundefined]

I understand that it's nice when people register to download ;) But I disagree that good for people. Keeping track of number of downwload - github doesn't have that? For security - i think most of people just have "copy/paste" or automated rules to install plugins. When register is needed to download plugin - they will stuck with already downloaded version. And - doesn't public plugins prevent from checking it.

btw:

undefine@undefine-ThinkPad-T430s:~/Pobrane$ file readonlyrest-1.13.0_es2.4.3.zip.sha1 
readonlyrest-1.13.0_es2.4.3.zip.sha1: Zip archive data, at least v1.0 to extract

When files will be publicly available - i'm sure that it has been detected earlier.

[sscarduzio]

If someone wants to automate, I'm pretty sure they can host the tiny binary somewhere else than my S3 bucket. The only thinking I care is that I will never distribute outdated builds, for the rest I can't prevent people from shooting themselves in the foot.

BTW I've been thinking to make the plugin check for updates at startup and show a warning log line when a newer version is available or the network is unreachable.

Of course the http call would run in a separate thread and won't prevent the normal use of the installed build.

[theundefined]

I agree. Hosting binaries is a "must be" for production environment. But - register to download ever single binary for every single version after every update... it not good :) Checking for updates - nice idea (but i think it should be possible to disable warning in config file). Yet better if there will be tool for automatic upgrade. Or tool to download binaries after specifing key(?)

[sscarduzio]

"register to download ever single binary for every single version after every update... it not good"

As I said, I definitely need a login system with "remember me" cookie. This constant form-filling should be one off.

"but i think it should be possible to disable warning in config file"

I kinda disagree on this, the fact that you have outdated security software should be as tedious as possible (without compromising the continuity of the service).

"tool to download binaries after specifing key"

Interesting, would you agree that this "tool" would boil down to a wget one-liner with a personal per-user API-key as a query parameter that redirects you to the usual signed S3 URL that you get now in the email?

I.e.

wget https://readonlyrest.com/latest?es_version=xxx&user_secret=yyy 
--> HTTP 304 redirect to:
 https://readonlyrest-data.s3.amazonaws.com/build/<WHATEVER_LATEST_PLUGIN_VERSION>/readonlyrest-<WHATEVER_LATEST_PLUGIN_VERSION>_es<ES_VERSION>.zip?AWSAccessKeyId=XXX&Expires=XXX&x-amz-security-token=XXX

[theundefined]

why not? If it doesn't need to manually enter into www-site and wait for an e-mail it's better than now :) If it's possible to integrate into automated continous-integration environment - it's perfect :)

[sscarduzio]

The problem with the above "personal link to latest version" hard wired in your deploy script is that as an ops guy, you'd basically have zero control on when the a new version of the plugin is deployed.

So if stuff was working for you in the previous version due to a buggy behaviour, or if I introduce a regression, you'd be paged as soon as a node is restarted. If I was the ops guy, I'd 100% not use that link.

I'm not convinced about the utility of this, which BTW hits on my AWS lambda + S3 bucket + data transfer bill.

PS: there's no download count in GitHub

[theundefined]

For test if a new version works fine there are a test/stage environment and automated system testing before deploy to production env. It doesn't show introduced security bugs, but show if specific version works like expected - what means that specific urls are available and other - not. And after notification about new version it's simpler to change used plugin version in chef/puppet/ansible configuration, deploy it into test env what provide download plugin into dml, do automated tests and finaly deploy into production env, than manually download plugins for every used elastic version and.. do the same ;)

every automated thing spares hours in future ;)

But ok - i just hate sites where i need to manually download software instead of just changing versions in configuration. Maybe it's only my personal phobia :)

[sscarduzio]

Let's talk again about this when we have a proper login+cookie system ;)

[sscarduzio]

Hello folks, we finally introduced an API endpoint, so you can install ROR Free for Elasticsearch programmatically, without going through the email:

bin/elasticsearch-plugin install -b \
 "https://api.beshu.tech/download/es?esVersion=${ES_VERSION}&pluginVersion=${ROR_VERSION}"

Both ES_VERSION and ROR_VERSION query parameters are optional and will default to the latest versions available.

[kiblik]

Hi @sscarduzio, Do you publish somewhere hashes of ZIPs of the plugin? Like SHA1 in the email or like SHA512 on ES website. I would like to check hash after download it. Thanks

[coutoPL]

@sscarduzio ?

[sscarduzio]

ATM the sha1 file is delivered as a link in the download emails. But it could be provided by the API as well. Will update this ticket when it's done.

[sscarduzio]

OK now you can add checksum=true query param and it will download the <file>.zip.sha1.

 curl -vvv  "https://api.beshu.tech/download/es?esVersion=8.6.0&pluginVersion=1.46.0&checksum=true" 
*   Trying 172.67.160.34:443...
* Connected to api.beshu.tech (172.67.160.34) port 443 (#0)
[...]
> GET /download/es?esVersion=8.6.0&pluginVersion=1.46.0&checksum=true HTTP/2
> Host: api.beshu.tech
> user-agent: curl/7.85.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 302 
< date: Tue, 31 Jan 2023 08:22:30 GMT
< content-type: text/html; charset=utf-8
< location: https://readonlyrest-data.s3.amazonaws.com/build/1.46.0/readonlyrest-1.46.0_es8.6.0.zip?AWSAccessKeyId=AKIA5SJIWBO53SA2JIS2&Signature=gBtq0U%2FuysqgSfWWgOhf0tSTj7k%3D&Expires=1675239750
< cf-ray: 79210a31790e0c47-MRS
< access-control-allow-origin: *
< cf-cache-status: DYNAMIC
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=3hNN8dKXi0apAtxcB%2ByD5HAlBQduub9CwF8OJLvJ5bNMCqjGKNvbMX3%2FSHmIO4hxCMbgwLnONYYprGS7%2BCb7Ik5ZckaFkCQymo0AhLwYnH6BWtVuRE11z7%2FY96llIFh2XQ%3D%3D"}],"group":"cf-nel","max_age":604800}