gitman
gitman copied to clipboard
jacebrowning
Gitman ignores groups on nested gitman install runs
We have just decided to switch from an inhouse developed, almost identical tool, to gitman. But I've stumbled across what seems to be a bug, during my initial testing:
Given a gitman.yml file that looks like this:
location: .gitman
sources:
# Production sources
- name: tf-module-production
link: tf-module
repo: git@githost:tf-module.git
rev: 1.2.0
# Dev sources
- name: tf-module-dev
link: tf-module
repo: git@githost:tf-module.git
rev: master
groups:
- name: production
members:
- tf-module-production
- name: dev
members:
- tf-module-dev
I expect that I end up with a directory looking like this, if I do a gitman install dev:
├── .gitman
├── .gitman.yml
└── tf-module -> .gitman/tf-module-dev
And if I do a gitman install production:
├── .gitman
├── .gitman.yml
└── tf-module -> .gitman/tf-module-production
This works - so far so good.
But if my tf-module contains a gitman.yml file with a similar structure and I run gitman install production, gitman actually installs all the sources, linking for each source, and it seems like I end up with the dev sources (so I end up with the master branches). I guess that gitman doesn't honour the groupname in the subsequent gitman install calls...?
Thanks for the report!
if my tf-module contains a gitman.yml
By this do you mean that git@githost:tf-module.git has its own set of dependencies to be recursively managed by gitman? Could you share the output of that second gitman install production -vv run?
Yes - sure! Here it is (some output is obfuscated):
[DEBUG ] Running install command...
[INFO ] Installing dependencies: production
[DEBUG ] Searching for config...
[DEBUG ] Looking for config in: /home/username/develop/gitman-test
[DEBUG ] Found config: /home/username/develop/gitman-test/.gitman.yml
[INFO ] No locked sources, using latest...
[INFO ] $ mkdir -p /home/username/develop/gitman-test/.gitman
[INFO ] $ cd /home/username/develop/gitman-test/.gitman
[INFO ] Updating source files...
[DEBUG ] Creating a new repository...
[INFO ] $ git clone --reference /home/username/.gitcache/gitman-subsub.reference [email protected]:username/gitman-subsub.git module-production
[DEBUG ] > Cloning into 'module-production'...
[INFO ] $ cd module-production
[DEBUG ] Checking for a valid working tree...
[DEBUG ] $ git rev-parse --is-inside-work-tree
[DEBUG ] > true
[DEBUG ] Checking for a valid git top level...
[DEBUG ] $ git rev-parse --show-toplevel
[DEBUG ] > /home/username/develop/gitman-test/.gitman/module-production
[DEBUG ] $ cwd /home/username/develop/gitman-test/.gitman/module-production
[DEBUG ] Confirming there are no uncommitted changes...
[DEBUG ] $ git update-index -q --refresh
[DEBUG ] $ git diff-index --quiet HEAD
[DEBUG ] $ git ls-files --others --exclude-standard
[DEBUG ] $ git rev-parse --abbrev-ref HEAD
[DEBUG ] > master
[DEBUG ] $ git rev-parse HEAD
[DEBUG ] > fc6f3ed3c7f8ab297d22af0eda76e4bfd53d5676
[DEBUG ] $ git describe --tags --exact-match
[DEBUG ] > 0.0.1
[INFO ] $ git remote set-url origin [email protected]:username/gitman-subsub.git
[INFO ] $ git fetch --tags --force --prune origin 2.0.0
[DEBUG ] > From githost.example:username/gitman-subsub
[DEBUG ] > * tag 2.0.0 -> FETCH_HEAD
[DEBUG ] $ git stash
[DEBUG ] > No local changes to save
[INFO ] $ git checkout --force 2.0.0
[DEBUG ] > HEAD is now at fc6f3ed Update gitman.yml
[DEBUG ] $ git branch --set-upstream-to origin/2.0.0
[DEBUG ] > fatal: could not set upstream of HEAD to origin/2.0.0 when it does not point to any branch.
[DEBUG ] Ignored error from call to 'git'
[INFO ] Creating a symbolic link...
[DEBUG ] Looking for config in: /home/username/develop/gitman-test/.gitman/module-production
[DEBUG ] Found config: /home/username/develop/gitman-test/.gitman/module-production/gitman.yml
[INFO ] No locked sources, using latest...
[INFO ] $ mkdir -p /home/username/develop/gitman-test/.gitman/module-production/.gitman
[INFO ] $ cd /home/username/develop/gitman-test/.gitman/module-production/.gitman
[INFO ] Updating source files...
[DEBUG ] Creating a new repository...
[INFO ] $ git clone --reference /home/username/.gitcache/gitman-submodule.reference [email protected]:username/gitman-submodule.git module-production
[DEBUG ] > Cloning into 'module-production'...
[INFO ] $ cd module-production
[DEBUG ] Checking for a valid working tree...
[DEBUG ] $ git rev-parse --is-inside-work-tree
[DEBUG ] > true
[DEBUG ] Checking for a valid git top level...
[DEBUG ] $ git rev-parse --show-toplevel
[DEBUG ] > /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[DEBUG ] $ cwd /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[DEBUG ] Confirming there are no uncommitted changes...
[DEBUG ] $ git update-index -q --refresh
[DEBUG ] $ git diff-index --quiet HEAD
[DEBUG ] $ git ls-files --others --exclude-standard
[DEBUG ] $ git rev-parse --abbrev-ref HEAD
[DEBUG ] > master
[DEBUG ] $ git rev-parse HEAD
[DEBUG ] > c2efb7cfb2766c8b67d0a027fa97b281ff4cea49
[DEBUG ] $ git describe --tags --exact-match
[DEBUG ] > fatal: no tag exactly matches 'c2efb7cfb2766c8b67d0a027fa97b281ff4cea49'
[DEBUG ] Ignored error from call to 'git'
[INFO ] $ git remote set-url origin [email protected]:username/gitman-submodule.git
[INFO ] $ git fetch --tags --force --prune origin 1.0.0
[DEBUG ] > From githost.example:username/gitman-submodule
[DEBUG ] > * tag 1.0.0 -> FETCH_HEAD
[DEBUG ] $ git stash
[DEBUG ] > No local changes to save
[INFO ] $ git checkout --force 1.0.0
[DEBUG ] > HEAD is now at 8d84107 Release 1.0.0
[DEBUG ] $ git branch --set-upstream-to origin/1.0.0
[DEBUG ] > fatal: could not set upstream of HEAD to origin/1.0.0 when it does not point to any branch.
[DEBUG ] Ignored error from call to 'git'
[INFO ] Creating a symbolic link...
[INFO ] $ mkdir -p /home/username/develop/gitman-test/.gitman/module-production/modules
[DEBUG ] Looking for config in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[DEBUG ] No config found in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[DEBUG ] $ cd /home/username/develop/gitman-test/.gitman/module-production/.gitman
[INFO ] Updating source files...
[DEBUG ] Creating a new repository...
[INFO ] $ git clone --reference /home/username/.gitcache/gitman-submodule.reference [email protected]:username/gitman-submodule.git module-sandbox
[DEBUG ] > Cloning into 'module-sandbox'...
[INFO ] $ cd module-sandbox
[DEBUG ] Checking for a valid working tree...
[DEBUG ] $ git rev-parse --is-inside-work-tree
[DEBUG ] > true
[DEBUG ] Checking for a valid git top level...
[DEBUG ] $ git rev-parse --show-toplevel
[DEBUG ] > /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[DEBUG ] $ cwd /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[DEBUG ] Confirming there are no uncommitted changes...
[DEBUG ] $ git update-index -q --refresh
[DEBUG ] $ git diff-index --quiet HEAD
[DEBUG ] $ git ls-files --others --exclude-standard
[DEBUG ] $ git rev-parse --abbrev-ref HEAD
[DEBUG ] > master
[DEBUG ] $ git rev-parse HEAD
[DEBUG ] > c2efb7cfb2766c8b67d0a027fa97b281ff4cea49
[DEBUG ] $ git describe --tags --exact-match
[DEBUG ] > fatal: no tag exactly matches 'c2efb7cfb2766c8b67d0a027fa97b281ff4cea49'
[DEBUG ] Ignored error from call to 'git'
[DEBUG ] $ git stash
[DEBUG ] > No local changes to save
[INFO ] $ git checkout --force master
[DEBUG ] > Already on 'master'
[DEBUG ] > Your branch is up to date with 'origin/master'.
[DEBUG ] $ git branch --set-upstream-to origin/master
[DEBUG ] > Branch 'master' set up to track remote branch 'master' from 'origin'.
[INFO ] Creating a symbolic link...
[DEBUG ] Looking for config in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[DEBUG ] No config found in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[DEBUG ] $ cd /home/username/develop/gitman-test/.gitman/module-production/.gitman
[DEBUG ] $ cd /home/username/develop/gitman-test/.gitman
[INFO ] Skipped dependency: module-sandbox
[INFO ] No locked sources, using latest...
[INFO ] $ cd /home/username/develop/gitman-test/.gitman
[INFO ] Running install scripts...
[INFO ] $ cd module-production
[DEBUG ] Checking for a valid working tree...
[DEBUG ] $ git rev-parse --is-inside-work-tree
[DEBUG ] > true
[DEBUG ] Checking for a valid git top level...
[DEBUG ] $ git rev-parse --show-toplevel
[DEBUG ] > /home/username/develop/gitman-test/.gitman/module-production
[DEBUG ] $ cwd /home/username/develop/gitman-test/.gitman/module-production
[INFO ] (no scripts to run)
[DEBUG ] Looking for config in: /home/username/develop/gitman-test/.gitman/module-production
[DEBUG ] Found config: /home/username/develop/gitman-test/.gitman/module-production/gitman.yml
[INFO ] No locked sources, using latest...
[INFO ] $ cd /home/username/develop/gitman-test/.gitman/module-production/.gitman
[INFO ] Running install scripts...
[INFO ] $ cd module-production
[DEBUG ] Checking for a valid working tree...
[DEBUG ] $ git rev-parse --is-inside-work-tree
[DEBUG ] > true
[DEBUG ] Checking for a valid git top level...
[DEBUG ] $ git rev-parse --show-toplevel
[DEBUG ] > /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[DEBUG ] $ cwd /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[INFO ] (no scripts to run)
[DEBUG ] Looking for config in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[DEBUG ] No config found in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-production
[DEBUG ] $ cd /home/username/develop/gitman-test/.gitman/module-production/.gitman
[INFO ] Running install scripts...
[INFO ] $ cd module-sandbox
[DEBUG ] Checking for a valid working tree...
[DEBUG ] $ git rev-parse --is-inside-work-tree
[DEBUG ] > true
[DEBUG ] Checking for a valid git top level...
[DEBUG ] $ git rev-parse --show-toplevel
[DEBUG ] > /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[DEBUG ] $ cwd /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[INFO ] (no scripts to run)
[DEBUG ] Looking for config in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[DEBUG ] No config found in: /home/username/develop/gitman-test/.gitman/module-production/.gitman/module-sandbox
[DEBUG ] $ cd /home/username/develop/gitman-test/.gitman/module-production/.gitman
[DEBUG ] $ cd /home/username/develop/gitman-test/.gitman
[INFO ] Installed 3 dependencies
[DEBUG ] Command succeeded
The config looks like this:
location: .gitman
sources:
- name: module-production
repo: [email protected]:username/gitman-subsub.git
rev: 2.0.0
link: modules/module0
- name: module-sandbox
repo: [email protected]:username/gitman-subsub.git
rev: master
link: modules/module0
groups:
- name: production
members:
- module-production
- name: sandbox
members:
- module-sandbox
Does gitman-subsub.git have it's own gitman config file? Can you share that as well?
Yes, sure:
location: .gitman
sources:
# Production sources
- name: module-production
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: 1.0.0
# Dev sources
- name: module-sandbox
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: master
groups:
- name: production
members:
- module-production
- name: sandbox
members:
- module-sandbox
Thanks, that's helpful! You're correct that groups are only processed once at the top level, so I believe this issue is a request to pass that information to each nested level.
FWIW, I'm not sure if installing different versions of the same dependency is the intended use case for groups.
You should be able to accomplish what you're trying to do with locked sources: https://gitman.readthedocs.io/en/latest/use-cases/branch-tracking/
location: .gitman
sources:
- name: module-production
repo: [email protected]:username/gitman-subsub.git
rev: master
link: modules/module0
sources_locked:
- name: module-production
repo: [email protected]:username/gitman-subsub.git
rev: 2.0.0
link: modules/module0
where your "dev" setup is gitman update --skip-lock and your "production" setup is gitman install.
Hi Jace
Thank you for your answer. You’re right that I can used locked_sources for that simple example. Truth is that I have 3 environments - production, preprod and prod - which requires the group functionality, together with different named versions of the same sources. You could probably argue that I could/should use locked_sources for both preprod and prod, but it would be nice to differentiate those two, until I feel confident that the version is ready for production.
Do you have any idea if it can be implemented..? I took a look at the code yesterday, but I can’t seem to find where it descends into the sub repo and calls itself again..
(As a side note: I use this for a Infrastructure As Code repo, where the sources that Gilman is managing, is the terraform modules and sub modules used for managing the infrastructure. So I can have situations where I’m doing a new release of the infrastructure, but still need to be able to ensure that the production environment is running the current version (be able to run terraform plan without seeing any changes). Unfortunately, our in-house tool is having the same exact ‘bug’ (not honoring the groups in the sub repos) :-( - together with other bugs ).
Sent from my iPad
On 20 Aug 2020, at 16.46, Jace Browning [email protected] wrote:
Thanks, that's helpful! You're correct that groups are only processed once at the top level, so I believe this issue is a request to pass that information to each nested level.
FWIW, I'm not sure if installing different versions of the same dependency is the intended use case for groups.
You should be able to accomplish what you're trying to do with locked sources: https://gitman.readthedocs.io/en/latest/use-cases/branch-tracking/
location: .gitman sources:
- name: module-production repo: [email protected]:username/gitman-subsub.git rev: master link: modules/module0 sources_locked:
- name: module-production repo: [email protected]:username/gitman-subsub.git rev: 2.0.0 link: modules/module0 where your "dev" setup is gitman update --skip-lock and your "production" setup is gitman install.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.
This is the spot where "production" is translated to ["tf-module-production"]: https://github.com/jacebrowning/gitman/blob/5acc0e4b1d5c15683213563901acd0349a1fa155/gitman/models/config.py#L87
This is the spot where we recurse into nested repositories: https://github.com/jacebrowning/gitman/blob/5acc0e4b1d5c15683213563901acd0349a1fa155/gitman/models/config.py#L117
You'll notice that names are not forwarded to recursive installation calls. To achieve the behavior you're looking for, we would need to determine that "production" is the name of a group (and not just the name of a single source), then forward that as *names to all recursive calls. In hindsight, having a separate --groups argument may have been a better design to keep these lists of names separate.
Another thing to consider is how to handle nested gitman configs that don't have groups defined, but are passed a group name.
Hi Jace
Thank you for the pointers - I’ll have a look at it.
Gitman files without groups: Maybe just treat that as a normal gitman install, and install the repos listed under sources...?
On 21 Aug 2020, at 16.55, Jace Browning [email protected] wrote:
This is the spot where "production" is translated to ["tf-module-production"]: https://github.com/jacebrowning/gitman/blob/5acc0e4b1d5c15683213563901acd0349a1fa155/gitman/models/config.py#L87
This is the spot where we recurse into nested repositories: https://github.com/jacebrowning/gitman/blob/5acc0e4b1d5c15683213563901acd0349a1fa155/gitman/models/config.py#L117
You'll notice that names are not forwarded to recursive installation calls. To achieve the behavior you're looking for, we would need to determine that "production" is the name of a group (and not just the name of a single source), then forward that as *names to all recursive calls. In hindsight, having a separate --groups argument may have been a better design to keep these lists of names separate.
Another thing to consider is how to handle nested gitman configs that don't have groups defined, but are passed a group name.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.
We have a similar requirement solved by using seperate branches in the top level repository (like dev and prod) to maintain seperate gitman configs because of our many repositories and parallel dev enviroments. Switching between environments is just accomblished by switching between branches.
Furthermore we realized that the recursive dependency resolution increases the complexity of the config management at a certain amount of nested dependency levels. We have a very complex dependency graph with more than 80 repositories. To simplify the config management we introduced a distinct top level "product" repository which contains one huge flat gitman.yml and some product related global stuff (configs/build scripts/jenkins groovy scripts). We don't use recursive dependency resolution which can also produce hard to find problems when referencing different versions of the same repository.
We resolve all product related dependencies over a global gitman config. Each developer and also our build system needs to use this product repository to build all or parts of the product. This helps to avoid workspace clashes. Because of the many different software developer roles resp. software modules we defined groups which represents sub-workspaces (like app_a, app_b, hal, drivers and so on). This works quite well.
In a release phase we create a new release branch in this toplevel repository and pining each of the dependencies one by one. At the end in each rev-field is a tag referenced. To see what changed over several releases or between dev activities is quite easy by comparing the one and only gitman.yml between the different branches,
Maybe this approach is also helpful for your application.
@daniel-brosche Thank you for your input - although I have a hard time figuring out what you are doing. Can you maybe paint the picture a little bit clearer with a simple example..?
Maybe this helps, I have tried to illustrate it with an abstract example.
The product repository here is product_xy which contains the global product related gitman.yml besides some other stuff.
Mostly we continously build the mainline (master) of all repositories during the common development phase. Here the product repository master branch references the needed master branches of all repository dependencies. As mentioned, this product repository is used by all developers and the buildsystem itself (jenkins here).
In this global gitman.yml there are groups defined which represents sub-workspaces for the main developer domains.
In the release phase I create a release branch in the product repository and reference tags over time. At the end of stabilization only tags are referenced in this product repository release branch.
For very complex features I mix these approaches and create a distinct development environment that reference feature branches and tags.
Over this way, I can ensure a stable feature environment where only the related feature repositories are mutable over time (e.g. 75% of all repositories are immutable and 25% are mutable).
This is also very helpful to exchange feature related workspaces between developers and buildsystem to build reproducible feature related builds. At the end of the feature development all depedent feature repositories are merged in the corresponding master branch and the feature branch in the toplevel repository will be deleted.
Some parts of this worklow can also be achieved by locking sources but in my experience this approach is more explicit and pretty easy in terms of config management.
This is almost how I do it. I have a directory structure like this one:
tf-live/
├── gitman.yml
├── tf-bootstrap
│ └── modules
│ └── tf-aws
├── tf-network
│ ├── gitman.yml
│ └── modules
│ ├── tf-asg
│ └── tf-aws-network
│ └── modules
│ └── tf-vpn-gateway
└── tf-services
└── modules
└── tf-aws-service
tf-live is just a shell, containing a gitman.yml file and a script to wrap the calls to terraform in the right order. I have 3 environments, sandbox, preprod and prod. I want the toplevel gitman.yml file to check out the same tf-* repos, but it different versions, depending on the environment:
tf-live/
├── gitman.yml
├── tf-bootstrap
├── tf-network
└── tf-services
-
Master branch, when it's sandbox -
v1.0.0if it's production -
v1.1.0if it's preprod and I'm testing a new release
I was hoping that groups could do that for me, and it works fine in the top level. But when fx tf-network contains modules, these can also vary depending on the environment. And they can also have submodules, with differing versions depending on the environment (which means another gitman.yml file):
tf-live/tf-network/
├── gitman.yml
└── modules
├── tf-asg
└── tf-aws-network
├── gitman.yml
└── modules
└── tf-vpn-gateway
If gitman supported sending the groupname to the subsequent gitman install command, it would work fine, but it doesn't.
Unfortunately I'm not a python programmer, so I'm not exactly sure how to solve this in the code, but I'll try to take a stab at it.
I might end up using a dedicated wrapper tool like terragrunt for this instead, as it handles environments with dependencies of differing versions just fine. But we also use this functionality in other code projects, and would like to use gitman if at all possible...
Just a quick thought: Would it be possible/nice to structure the config as follows, instead of the separate groups section?:
location: .gitman
sources:
default:
# Production sources
- name: tf-module-production
link: tf-module
repo: git@githost:tf-module.git
rev: 1.2.0
dev:
# Dev sources
- name: tf-module-dev
link: tf-module
repo: git@githost:tf-module.git
rev: master
Then always have a default "group" or section, which get's installed when just running gitman install.
But when fx tf-network contains modules, these can also vary depending on the environment.
According to my described approach, there is no nested gitman.yml and then I would arrange it like this:
In the master Branch of tv-life:
location: .gitman
sources:
- name: module0
repo: [email protected]:username/gitman-subsub.git
rev: master
link: modules/module0
- name: module1
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: master
In the branch v1.0.0
location: .gitman
sources:
- name: module0
repo: [email protected]:username/gitman-subsub.git
rev: 2.0.0
link: modules/module0
- name: module1
link: modules/module1
repo: [email protected]:username/gitman-submodule.git
rev: 1.0.0
You could also change the gitman location to modules then the links are not needed.
I do import everything in defined imports directory also with subdirectories like infra\component1 directly over the name field (-name: infra/component1). All components look into this directory to resolve their own dependencies (by convention).
I've come up with an alternative approach that I like a bit better.
First we add a gitman config file for each environment:
gitman-production.yml gitman-preprod.yml gitman-sandbox.yml
Then a simple Makefile:
.PHONY: install clean uninstall
all: .gitman.yml
@echo "Preparing for deploying to ${ENV}..."
gitman install
.gitman.yml::
@ln -sf gitman-${ENV}.yml .gitman.yml
clean:
gitman uninstall --force
-rm -f .gitman.yml
uninstall:
gitman uninstall
-rm -f .gitman.yml
All it does is symlinking the gitman-${ENV}.yml file to .gitman.yml and run gitman install (besides the clean and uninstall targets)
In all subrepos we add a similar structure (if it has dependencies on anything, of course):
gitman-production.yml gitman-preprod.yml gitman-sandbox.yml Makefile
Each gitman-env.yml file looks like this (with multiple different repos in different revisions of course):
location: .gitman
sources:
- name: testhest
repo: [email protected]:username/gitman-test.git
rev: master
link: testhest
scripts:
- ENV=${ENV} make
As you can see it runs make in the end, with the $ENV var preprended. This makes the sub repos’ git install command use the correct gitman config file. The .gitman.yml (which is just a symlink) should be ignored in .gitignore.
So to sum it up: With a setup like this, we have a workflow looking like this (deploying to production):
- Check out main repo
- Run ENV=production make to install production sources
- Run deployment scripts
- Run make clean to clean up (this forces a gitman uninstall)