Support for file creation from git repositories.

[hjoliver]

At NIWA we still use an in-house deployment system (now in special deployment tasks) to install suite components at the start of a suite run. The main reason for this is that our operational system runs a lot of code stored in various small git repositories (none of these are "UM-related", and some even pre-date our use of FCM - they were initially darcs repositories). This works well enough, but our suites would be a bit cleaner if we could just install these files via rose app configs. Potentially I could have a crack at this, following lib/python/rose/loc_handlers/svn.py? In which case, any pointers or issues to be aware of?

[matthewrmshin]

Yes, it is possible to implement support by following rose.loc_handlers.svn or rose.loc_handlers.rsync. All you need is to create a class with a similar interface.

The main issue is how to write down the location of a Git clone + a revision + a path as a source.

[matthewrmshin]

See also metomi/fcm#48.

[matthewrmshin]

@benfitzpatrick you expressed an interest in this issue this morning.

[benfitzpatrick]

I did! My new team is allergic to Subversion, having had experience of Git. Our main scripts are in internal Git repositories.

[matthewrmshin]

I hope this allergy is contagious...

[dpmatthews]

Related discussion: https://cylc.discourse.group/t/check-out-single-file-from-github-at-suite-run-time

[wxtim]

I hope this allergy is contagious...

Yes...

[oliver-sanders]

Easy to do, however, we will probably want to implement multiple methods and let the user choose which to use:

Method	Overview	GitHub	Refs	Branches	Tags	Selected Files
Shallow Clone	git clone -b <branch|tag> --depth=1 <repo>	:ok:	:no_entry: (1)	:ok:	:ok:	:no_entry:
Archive	mkdir <path>; git -C <repo> archive <branch|tag|ref> <path> | tar -x -C <path>	:no_entry:	:ok:	:ok:	:ok:	:ok:

[1] Allowed if uploadArchive.allowUnreachable=true

[edmundhenley-mo]

@oliver-sanders, re selected files, might an additional method here be the raw.githubusercontent.com links?

At least for public repos - looks like tokens on private repo links have an expiry date (default 7 days, extendable to only-if-unused-in-1-year...); not sure whether rules for private-repo-under-organisation tokens differ.

E.g. I now use this public-raw-link based hack ~often for getting a handy ".svnignore" file for Python-based svn repos:

# Get the Python-based .gitignore from raw GH url
wget https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore --output-document=.svnignore --quiet

# Remove trailing slashes on patterns for dirs, which svn can't handle
sed --expression="s|/$||" --in-place .svnignore

# Generate a usage message
usage=$(cat<<"EOF"
# Adapted .gitignore for svn (no trailing \ for dirs)
# Source .gitignore: https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore
# Usage to mimic .gitignore: svn propset svn:global-ignores -F .svnignore .
EOF
)

# Prepend usage message onto .svnignore
printf "%s\n\n%s\n" "$usage" "$(cat .svnignore)" > .svnignore

# Apply it!
svn propset svn:global-ignores -F .svnignore .

# Commit this reference ignore file; any updates will need reapplying
svn add .svnignore