Added scp transport for hotstart

[jasonfleming]

resolves #715 not tested yet

(edited by @wwlwpd below)

We discussed on 4/9/24, updating this PR:

• [ ] changing scp:// to ssh:// due to the upstream desire to deprecated scp
• [ ] if it looks like a file (i.e., doesn't have http://, https://, or ssh://), look for the standard HOST:/path/to/file; this can be done easily by splitting on the :, with awk -F: '{print $1}' and awk -F '{print $2}'.

These things to do will round out this support nicely. In the future, other protocols may become important (e.g., s3://, etc); so we definitely want to be able to have a clear place to support them and a consistent way to support them across the ASGS toolset.

[jasonfleming]

Ok thank you. I think I made these changes as a conversation starter and never got to the testing phase. This is an important enabler, so this will be next on my readiness agenda.

[wwlwpd]

Thanks @jasonfleming - bear in mind if we just want to support it in this particular case, the I think the current implementation is fine. If we want to add scp:// support in other places, we should create a function that does parsing of the path so we're not re-implementing that all over the place. I am happy to help here.

[wwlwpd]

@jasonfleming just wondering what the state of this PR draft was

[jasonfleming]

Implemented but not tested. I just bumped the priority of the underlying issue to important non-critical

[wwlwpd]

We can discuss this after #1277 gets in. I've opted to not use a URI scheme for now, instead assuming ssh is the transport, and just using the direct syntax used by ssh, scp, and rsync (which assumes ssh), user@host:path/to/file.

more details:

Since we lean heavily on ~/.ssh/config, it's more practically, host:path/to/file. This is both easier to parse and can be used directly during backup. For restore, I just awk -F: to get host and file since the command to get the files using ssh necessarily uses the parts in a deconstructed way. See #1277 for more context to the snippets below,

BKPTGZ=${1:-$HOME/ASGS-SAVE-${PWD}.tgz
#... looks for ":" to indicate remote file path
if [[ $BKPTGZ =~ ":" ]]; then
  USE_SSH=yes
fi

...

if [ "$USE_SSH" == "yes" ]; then
  HOST=$(echo "$BKPTGZ"|awk -F: '{print $1}')
  FILE=$(echo "$BKPTGZ"|awk -F: '{print $2}')
  tar zcvf - -T ./ASGS-MANIFEST | ssh $HOST "cat > $FILE"
else         
  tar zcvf $BKPTGZ -T ./ASGS-MANIFEST
fi

The command to get a file and pipe it to tar zcvf is,

  ssh $HOST "cat $FILE" | tar zxvf -

[jasonfleming]

The usual transport for hotstart files is https ... maybe if the ":" is missing then we assume the transport is https?

[jasonfleming]

Need to include a fix for #1002 for this to be useful.

[wwlwpd]

@jasonfleming I have a few things left to do, but you can do an initial review here. I recommend we only test what we normally use. e.g., eye-ball the gz support but don't worry about testing it. Similarly, do not worry about testing the (upcoming) INPUTINIT script support I am adding to support @enikidis's asgs-lfs.sh script.

I will need you to closely look or test if you have time that the URL schemes are working for the original intent of #715, which was to allow hotstart files to be specified via scp:// or ssh:// URIs (like we can do with http).

[wwlwpd]

@jasonfleming - Documentation could be an endless process, I stub'd out all the major parts, I think. Anything major that's obviously missing? We should develop some of the smaller sections at some point - but I am shooting for "good enough to remind us" :-) - what do you think? https://github.com/StormSurgeLive/asgs/wiki/Adding-a-New-Mesh

[wwlwpd]

I am going to recreate teh PR.