File transfer HTTP APIs for tedge-agent

[albinsuresh]

Is your feature request related to a problem? Please describe.

Expose REST endpoints to exchange binary files between thin-edge components and child devices.

Describe the solution you'd like

Expose the following HTTP REST endpoints from tedge-agent to support binary management for operations:

• Upload operation binary: PUT http://{thin-edge-ip}/tedge/<some>/<path>/<id>
• Download operation binary: GET http://{thin-edge-ip}/tedge/<some>/<path>/<id>
• Delete operation binary: DELETE http://{thin-edge-ip}/tedge/<some>/<path>/<id>

as defined in the prototype summary here

AC

1. The HTTP server must bind to mqtt.bind_address and mqtt.external.bind_address or mqtt.external.bind_interface
2. The uploaded files are stored in the tedge file repository(/var/tedge/file-transfer)
3. The directory structure of the uploaded file replicates the path in the HTTP URL. For e.g: if the URL is http://{thin-edge-ip}/tedge/<some>/<path>/<file>, that file will be stored in /var/tedge/file-transfer/<some>/<path>/<file>.
4. An uploaded file can be downloaded using the same path used to upload the file
5. All uploaded files are deleted explicitly with a DELETE call
6. Choose a low-footprint HTTP Rust library that supports https also for future enhancements

Not In Scope

1. Securing the endpoints to authenticated clients only
2. Local file-system based APIs for upload/download without HTTP
3. Automatic deletion of files as soon as a client downloads that file

[mbay-ODW]

Maybe those ideas can be combined @albinsuresh

https://github.com/thin-edge/thin-edge.io/issues/1315

[albinsuresh]

Maybe those ideas can be combined @albinsuresh

#1315

What you were requesting in #1315 is different from what was being proposed here. The upload/download that's referred here is about uploading and downloading binaries to/from thin-edge and not with the cloud. What you were requesting was something to ease the same with cloud. So, that will have to be handled separately.

[cmosd]

Testing

~use build files from: https://github.com/initard/thin-edge.io/actions/runs/3059214100~ use build files from: https://github.com/thin-edge/thin-edge.io/actions/runs/3197300446

To test this ticket one needs to the newest version of tedge agent. It is important to run the initialisation of the tedge_agent binary:

• $: sudo tedge_agent --init Once that is done, the tedge-agent service is started:
• $: sudo systemctl start tedge-agent.service This will allow you to write GET, PUT, and DELETE requests to http://127.0.0.1:80/tedge/file-transfer/ (Note the /tedge/file-transfer/ should be there. If it is something else it should NOT work)

The tests can be described as follows:

GET

For a GET requests one needs to first create a file in /var/tedge/file-transfer/<file or path to file>. For example:

touch /var/tedge/file-transfer/file_a
echo "content" > /var/tedge/file-transfer/file_a 

or

mkdir /var/tedge/file-transfer/dir 
touch /var/tedge/file-transfer/dir/file_b
echo "content" > /var/tedge/file-transfer/dir/file_b

Then perform a GET request to http://127.0.0.1:80/tedge/file-transfer/file_a or http://127.0.0.1:80/tedge/file-transfer/dir/file_b with the library of your choice.

DELETE

For a DELETE requests one needs to first create a file in /var/tedge/file-transfer/<file or path to file>. For example:

touch /var/tedge/file-transfer/file_a
echo "content" > /var/tedge/file-transfer/file_a 

or

mkdir /var/tedge/file-transfer/dir 
touch /var/tedge/file-transfer/dir/file_b
echo "content" > /var/tedge/file-transfer/dir/file_b

Then perform a DELETE request to http://127.0.0.1:80/tedge/file-transfer/file_a or http://127.0.0.1:80/tedge/file-transfer/dir/file_b with the library of your choice.

PUT

For a PUT request one needs to perform a PUT request to http://127.0.0.1:80/tedge/file-transfer/<file> (or any other URI of your choosing as long as it starts with /tedge/). Add to the PUT request some payload with content such as "content" and send the request.

This should create a file in /var/tedge/file-transfer/

below is a python example of what the requests and tests could look like:

import os
import requests

FILE_TRANSFER_ROOT: str = "/var/tedge/file-transfer/{}"
BASE_URL: str = "http://127.0.0.1:80/tedge/file-transfer/{}"

def create_file_with_content():
    with open(FILE_TRANSFER_ROOT.format("file_a"), "w") as handle:
        handle.write("content\n")

def get():
    assert os.path.exists(FILE_TRANSFER_ROOT.format("file_a")) == True

    response = requests.get(BASE_URL.format("file_a"))
    assert response.status_code == 200
    assert response.content.decode("utf8") == "content\n"

def delete():
    response = requests.delete(BASE_URL.format("file_a"))
    assert response.status_code == 202
    assert os.path.exists(FILE_TRANSFER_ROOT.format("file_a")) == False

def put():
    data = "content\n"
    response = requests.put(BASE_URL.format("file_a"), data=data)
    assert response.status_code == 201
    assert os.path.exists(FILE_TRANSFER_ROOT.format("file_a")) == True

    with open(FILE_TRANSFER_ROOT.format("file_a"), "r") as handle:
        content = handle.read()

    assert content == data

if __name__ == "__main__":
    # GET request test
    create_file_with_content()
    get()
    # DELETE request test
    create_file_with_content()
    delete()
    # PUT request test
    put()

[gligorisaev]

@initard tedge_agent installation is resulting in error:

pi@raspberrypi:~ $ sudo dpkg -i ./tedge_agent_0.7.4_arm64.deb 
(Reading database ... 36162 files and directories currently installed.)
Preparing to unpack ./tedge_agent_0.7.4_arm64.deb ...
Unpacking tedge_agent (0.7.4) over (0.7.4) ...
Setting up tedge_agent (0.7.4) ...
2022-09-30T06:42:51.838605958Z  INFO flockfile::unix: Lockfile created "/run/lock/tedge_agent.lock"
2022-09-30T06:42:51.838669013Z  INFO tedge_agent::agent: tedge_agent starting
Error: Creating the directory failed: "/var/tedge".
dpkg: error processing package tedge_agent (--install):
 installed tedge_agent package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 tedge_agent

[cmosd]

@gligorisaev Interesting, are you able to make the /var/tedge directory manually? What do you see if you execute: sudo -u tedge mkdir /var/tedge

[cmosd]

@gligorisaev, I updated the comment above with the right build file location, but putting it here also: https://github.com/thin-edge/thin-edge.io/actions/runs/3197300446

[rina23q]

QA check passed, therefore close this issue.