scp functionality

[svenschwermer]

In addition to the balena ssh command, it would be helpful to be able to have balena scp. As far as I understand, this will require a change in the proxy server (ssh.balena-devices.com:22) as well.

Ref: support thread (restricted access)

[pimterry]

Thanks @svenschwermer! We don't have plans for that in the short-term, but we'll definitely consider it for the roadmap in future. I'll leave this open to track that in the meantime.

[feychenie]

oh yes, I would love that feature too

[Ramblurr]

In the meantime is it possible to pull files off of a device?

[pdcastro]

Comment deleted in favor of my answer to user magiconair below.

[andrewjaykeller]

Yes please add this feature! Thanks @pdcastro for the advice

[magiconair]

This feature is essential and not just a nice-to-have also for production devices. For our current provisioning flow we copy secrets onto the device. Sometimes for debugging I want to copy an executable onto the device. While I can transfer a PEM file with vi and the clipboard this is no longer possible with a 1.5MB base64 encoded executable.

This would also be extremely useful for copying config files onto the device so that we don't have to use base64 encoded environment variables. We are looking into using https://github.com/kelseyhightower/confd for config management but it shouldn't be that hard to get files onto and off a device.

[pdcastro]

@magiconair, I fully understand it. As far as workarounds go, please note that the standard ssh and scp tools can be used with balenaOS devices, including standard ssh port forwarding. It works with both development and production balenaOS images. In the case of a production balenaOS image, a ssh key should be added to the config.json file -- see sshKeys section of the meta-balena README file.

The ssh/scp server on a device listens on TCP port number 22222. This port is not blocked by the device host OS, not even in production. If this port is blocked by a firewall or router on the device's local network, the balena tunnel command can be used as in the examples below.

Example 1 (port 22222 not blocked in the device's local network)

ssh -p 22222 root@<device_ip_address>
scp -P 22222 my_local_file root@<device_ip_address>:/mnt/data/

Example 2 (port 22222 is blocked in the device's local network)

balena tunnel <deviceUUID> -p 22222:4321
ssh -p 4321 [email protected]
scp -P 4321 my_local_file [email protected]:/mnt/data/

[magiconair]

Ok. That’s good to know. Will this also allow me to do ssh port forwarding to external hosts?

[pdcastro]

Will this also allow me to do ssh port forwarding to external hosts?

@magiconair, you mean like you've reported in issue #1155, right? I have successfully tested the following with both development and production balenaOS images:

# Run the following commands on your laptop/desktop

ssh -L 1111:www.google.com:443 -p 22222 root@<balenaDeviceIpAddress>
curl -ik https://127.0.0.1:1111

In the case of a production balenaOS image, a ssh key should be added to the config.json file -- see sshKeys section of the config.json documentation.

If port 22222 is blocked by a firewall / router in the device's local network, balena tunnel can be used to get around it:

# Run the following commands on your laptop/desktop

balena tunnel <deviceUUID> -p 22222:4321
ssh -L 1111:www.google.com:443 -p 4321 [email protected]
curl -ik https://127.0.0.1:1111

[magiconair]

Is there a reason this is documented in the meta-balena README? Honestly, it would never occur to me to look there.

[pdcastro]

Is there a reason this is documented in the meta-balena README?

The meta-balena README documents the aspects of config.json that are relevant to the configuration of balenaOS, which includes the configuration of the ssh server shipped with balenaOS. The scripts that copy the public ssh keys from config.json to the location where the ssh server will find them are part of the meta-balena repo.

I agree that CLI users cannot be expected to look there, although balena ssh itself does not require any keys to be added anywhere. I think it would make sense for balena help ssh to mention that standard ssh/scp can also be used, with a pointer to further documentation elsewhere. The documentation of config.json should also be expanded, like to include an example of the public ssh key string, as a format reference:

"os": {
    "sshKeys": [
        "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7ls9NzKL/EsrH04gy0ByVU2X4v+Bte2rBtLZCvOQBRz3W7bNsfkvE7l0ZW2Od/oA35QjqomBsWvK+NJ6655lzTJ0Zl1PsozIuaAcdegN/bmaQJA9c5ef5kV0vmijn/CswGliEmp1bOBWXdooBe1JvvYJ9Bjg9nQMIx668PjVc7lP8GqrEKhEROvn6j7fwkvWOXN0qxn0Tg4KeuPrQrEodUNW3tONutfhvGSeImpgHaJ3E+A/kCS6ZoYORo5kAIlu0seiHUnC85KnB+fxfImxb/A5SOz9HGDieK7LELGuPS5nrjHPm6DaCUNhXWstzLchS0pfd6aQpqVxFUBJu3GtD [email protected]"
    ]
}

It is the same format as in the web dashboard → user icon → Preferences → SSH keys tab.

[r4space]

Anyone used the suggested scp approach recently

balena tunnel doesn't seem to be an available command any more

[pdcastro]

balena tunnel doesn't seem to be an available command any more

@r4space, balena tunnel was introduced in balena-cli version 9.12.3 and remains available in the latest version (10.0.1 at the time of this writing):

$ balena version
10.0.1

$ balena help tunnel
Usage: tunnel <uuid>
Use this command to open local ports which tunnel to listening ports on your balenaOS device.
...

I suggest checking your CLI version by running the commands above.

[kraigher]

Since balena ssh works without adding SSH-keys to the device config.json it is a pity that there is not a similarly simple way to balena scp. Adding SSH-keys to config.json when it otherwise would not be required just to transfer files is very inconvenient.

[pdcastro]

Implementation note for the balena team: the following backend issue currently blocks a simple "scp workaround" using remote command execution with balena ssh (piping file contents to stdout): https://github.com/balena-io/balena-proxy/issues/466

[svenschwermer]

The following two bash functions provide a workaround:

# Set variables uuid, local_path, remote_path & SSH_USER

function to_device {
    tmp=$(mktemp)
    trap 'rm "$tmp"' EXIT
    gzip -c -9 "$local_path" > "$tmp"
    size=$(stat -f '%z' "$tmp")
    ssh -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=no \
        "$SSH_USER"@ssh.resindevice.io host -s "$uuid" \
            "dd bs=1 count=$size status=none | gunzip > \"$remote_path\"" < "$tmp"
}

function from_device {
    ssh -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ControlMaster=no \
        "$SSH_USER"@ssh.resindevice.io host -s "$uuid" \
            "gzip -c \"$remote_path\"" | gunzip > "$local_path"
}

[jellyfish-bot]

[sradevski] This issue has attached support thread https://jel.ly.fish/56a596a6-8ac2-4207-8041-58dcaa74dc0f

[pdcastro]

While investigating the best way of implementing balena scp, I've come up with a proof-of-concept implementation (separate from the balena CLI, at least for now) that is ready for use, with advantages over other workarounds / solutions:

• Does not require a ssh key to be added to config.json, for both development and production variants of balenaOS.
• Does not require balena tunnel, even for remote devices identified by UUID.
• Has full compatibility with standard scp -- all scp features, command-line options, configuration files and environment variables can be used. (scp-uuid is just a thin wrapper around standard scp, and passes through the command line options.)
• Allows transferring files directly to service containers through rsync, tar or cat (see usage examples).

Notable restrictions:

• Requires balenaOS v2.44.0 or later - released October 2019.
• On Windows, for now, it requires bash such as WSL - Windows Subsystem for Linux. (It does not currently work with native PowerShell or cmd.exe prompts.)

Check the usage examples in the project's README:

• File transfer with scp
• File transfer with cat
• File transfer with tar
• File transfer with rsync

As a proof of concept, it was implemented as a bash script, executed as scp-uuid and ssh-uuid. The README file has installation instructions.

Feedback is welcome. Depending on feedback and further discussions with the balena team, scp-uuid may find its way to the balena CLI as a balena scp command, while balena ssh could be rewritten along the lines of ssh-uuid.

[pdcastro]

scp-uuid update: version 0.2.0 (git tag v0.2.0 or current master branch) adds proper support for the --service option to copy files directly to/from service containers.

[cbecker]

I've been looking for an alternative to scp, and @pdcastro's solution works great. Maybe eventually we'll have that within balena-cli, I hope :)

[jrc]

This is what I do (for text files):

echo "cat /PATH/TO/FILENAME; exit;" | balena ssh DEVICE_UUID | tail -n +4 > FILENAME

The tail -n +4 part cuts off the "Welcome to balenaOS" header.

Looking forward to a real balena scp!

[hraftery]

@pdcastro's solution looks handy but after too-long battle I'm thwarted by a prompt to login to <my-username>@uuid.balena (like I see sometimes with balena ssh if the session has expired).

I went with a different work around instead. It seems balenaOS itself has scp! So I balena ssh in and then scp from the host while on the target, instead of being on the host and scp'ing to the target.

Naturally this requires you run a ssh server on your host, but that was already the case for me.

[nargetdev]

@pdcastro you are a godsend. Massive missing portion of balena-cli ecosystem.

I was able to get rsync working in all places (os x/ubuntu) after installing rsync on the remote container rsync -av -e 'ssh-uuid --service webcam-capture' 10e204a28e7f0ad09549d2eebade8ec1.balena:/data/img/video0/2022-08-01 .

... and scp working on Ubuntu. I think my OS X implementation is broken more generally

[shaunco]

Awesome work by @pdcastro - but it is absolutely maddening that even Balena Cloud doesn't have a simple file transfer option built into it. On Mender it is easily our most used tool during troubleshooting. Even worse that this ticket has been open for 5 years.