submodule fails

[magne]

Hi.

I'm trying to add vundle as a submodule for my vim repository, but when I try to execute vcsh vim submodule add https://github.com/gmarik/vundle.git ~/.vim/bundle/vundle, it fails with:

fatal: working tree '/home/magne' already exists.
Clone of 'https://github.com/gmarik/vundle.git' into submodule path '/home/magne/.vim/bundle/vundle' failed

Is using submodules impossible with vcsh, or is there something I'm missing?

Magne Rasmussen

[RichiH]

I have never used git submodule before, I didn't test it.

Do I assume correctly that, normally, git submodule would check out into a subdirectory of another source tree? If yes, there's an easy explanation: $GIT_WORK_TREE forces it to not act on a subdirectory but in $HOME directly.

Possible approaches to fix this:

• Parse command being passed to vcsh and do magic if it sees 'submodule'. This is likely to fail with custom aliases, new commands, etc
• Try to get a concept of relative paths into git. I am not sure how that would work.
• Repo-specific config files that set GIT_WORK_TREE

For now, as a workaround, please try:

VCSH_BASE='/home/magne/.vim/bundle/vundle' vcsh vim submodule add https://github.com/gmarik/vundle.git

as an aside, you can always

export VCSH_DEBUG='1'

to see what is going on.

[smarter]

Using VCSH_BASE doesn't seem to help: http://pastebin.com/YF4HegKw

[RichiH]

How did you call it, exactly? This info is missing, but important.

[smarter]

VCSH_BASE='/home/smarter/.oh-my-zsh' vcsh zsh submodule add https://github.com/sorin-ionescu/oh-my-zsh.git > vcsh.log 2>&1

[RichiH]

OK. I can't promise to be able to look at it as I will leave for three weeks tomorrow morning...

[RichiH]

Can you run it again, but set

export VCSH_DEBUG='1'

before doing it and paste full input and output? Thanks!

[ioanrogers]

I've put some debug logs here: https://gist.github.com/2992239

I never managed to get submodules to work with a detached WORK_DIR before I started using vcsh, either. A git restriction/problem?

[RichiH]

Thanks.

Yes, it's a git limitation and I came to the conclusion that I can not safely work around it within the scope of vcsh. git submodule would normally use a sub-directory to $PWD for data storage. In your case, that would be $PWD/.ls-colors.

What happens is that it uses $GIT_WORK_TREE when it should use $GIT_WORK_TREE/.ls-colors, instead. I understand why it's implemented in this way, but it's still a problem.

I will poke the git ML about this.

[RichiH]

http://permalink.gmane.org/gmane.comp.version-control.git/200640

[RichiH]

After some more talking on the ML, are you even certain using submodules will help you in any way? With vcsh, you may be able to just use it as a normal repo and everything works automagically.

In case you want automagic pulls etc, mr can do that.

Can you try this and get back to me?

[ioanrogers]

Personally, I use submodules as a way to bring in 3rd-party rcfiles in a plugin-like fashion. Another submodule I was using checks out under ~/.zsh/plugins/zsh-syntax-highlighting-filetypes. It's a lot more convenient to have e.g. all the 3rd party zsh plugins as submodules of my rc-zsh.git repo.

To use them as standalone, I have to alter the $VCSH_BASE as the submodules need to be in subdirs, not $HOME. e.g:

VCSH_BASE=$HOME/.ls_colors vcsh clone git://github.com/trapd00r/LS_COLORS.git rc-zsh-ls_colors

This has the side effect of dumping a gitignore.d dir in ~/.ls_colors, and giving permission errors when I mr push.

So far it's usable, but not as convenient as submodules could be, with git support.

I was thinking of a post-update hook that checks for a .gitmodules file and clones or pulls all the listed submodules; but then what to do if more than one repo has submodules?

[RichiH]

This should be fixed since commit be8779f7ac9a3be9aa783df008d59082f4054f67 , the test case should work since 95e7705310bce3b2721ebf789776c170cf424abb .

git v1.7.12 contains both commits; verification by someone who actively uses both submodules and vcsh would be appreciated.

[RichiH]

Those commits are in git, not vcsh; something that seems to confuse github's parsing magic.

[ioanrogers]

Looks good!

machine1: vcsh rc-zsh submodule add -f git://github.com/trapd00r/LS_COLORS.git .ls_colors && vcsh commit && mr push

machine2: mr up && vcsh rc-zsh submodule update --init

does what I expect.

Perhaps a warning in the docs about only having one .gitmodules file?

Thanks!

[RichiH]

What do you mean, only one .gitmodules file?

I assume those dumped into $HOME and thus conflict with each other. Can you force/set up a different name?

[ioanrogers]

Sorry, yes, that's what I meant.

I can't find any mention in the git docs of being able to specify an alternative file for gitmodules

[sebastiankricner]

Hi,

checking out submodules is possible.

According to this information on http://stackoverflow.com/questions/7446681/how-do-i-init-update-a-git-submodule-in-a-working-tree-after-pushing-to-a-bare-w the main problem is that the GIT_WORK_TREE is set. But a workaround is possible like this:

[.config/vcsh/repo.d/vim.git] checkout = VCSH_BASE=~/.vim vcsh clone ssh://xxxxxxxxx/vim.git vim fixups = cd ~/.vim;git --git-dir=$HOME/.config/vcsh/repo.d/vim.git/ submodule update --init

I think it is also possible to add such fix into vcsh. First vcsh would need to change to the GIT_WORK_TREE and call git just with the GIT_DIR with the according parameters for submodules. This means GIT_WORK_TREE must be unset in environment.

The same also applies for adding submodules and other possible tasks on submodules.

[RichiH]

There's now a hook called post-clone-retired in the retire branch: https://github.com/RichiH/vcsh/compare/retire which allows running arbitrary clone commands after a clone.

I am not sure how feasible that approach is for this use case as I don't use submodules; advice would be appreciated.

Also, I hear that git subtree is the new cool kid on the block and git submodule will be deprecated. Not saying that I won't support submodules, merely noting that I would appreciate feedback on this, as well.

[sebastiankricner]

Indeed subtree looks nice, must have a further look at it.

[RichiH]

Any update? Does subtree work with vcsh? Is it a viable replacement for submodules?

[sebastiankricner]

Not done anything yet, due lack of time.

[tobru]

git subtree seems to work together with vcsh. Example:

vcsh vim remote add vundle [email protected]:gmarik/vundle.git
vcsh vim fetch vundle
vcsh vim subtree add -P .vim/bundle/vundle/ -m "vundle subtree" vundle/master

[txomon]

Seems like this isn't doing any progress, is it?

[lenormf]

EDIT: I didn't see @tobru's comment, as my original message stated, subtrees do work.

[RichiH]

@lenormf as I found your comment to be useful on its own and don't see a need to edit it away, I am quoting you in full below:

I got everything to work with a bit of tweaking around the subtrees thing, assuming you're interested in not merging plugins into your configuration repository.

Create a regular vcsh repository, and add the directory that will hold the plugins
vcsh init vim && mkdir -p ~/.vim/bundle && touch ~/.vim/bundle/.keepdir
Add a remote to the repo that will point to the plugin's files
git remote add -f vim-pathogen https://github.com/tpope/vim-pathogen.git
Commit the changes to the repository, and push to a central repository of your own (I use gitolite for that)
On the machine you want to deploy your configuration on, clone the repository, and create a new branch for the plugin's files to be merged on
git clone … && git branch plugins && git checkout plugins
Add the subtree (the squash option merges the history of the remote repo in one commit)
git subtree add --prefix .vim/bundle/pathogen vim-pathogen master --squash
I use an additional branch so that I can keep the commits I make on the master branch seperate from all the merges made by the subtrees (I have a few scripts to automate the process of adding plugins/creating the subtrees).

Submodules keep adding metadata to the repo and an ugly .gitmodules file in my $HOME, subtrees are much more convenient ;)

HTH.

[RichiH]

To answer @txomon 's question: Is progress on submodules still needed?

I tried several times to make it work, but things keep breaking. I talked to the two main devs of git submodule and they told me part of the problems are design-wise and that changing its behavior at this stage is unlikely due to momentum and lack of manpower.

[lenormf]

@RichiH The problem with my method is that the remotes you add to your local repository will not be saved to the one you push to. You lose all references to the plugins' remote repositories when you clone from the central repo (the one that supposedly hosts your configuration files) on another system, which means that you have to use subtrees and merge the plugin files from the beginning. My original message assumed that I could leave the plugin files on their own remote repository and clone them when I needed to (just like with submodules), but that's not the case, so @tobru had it right in the first place ☺

The other solution I thought about was to make my own submodules system (which would not create files/directories in $HOME), but that's too much hassle compared to the ease of use of subtrees (although I have to integrate them completely into the parent repository).

[txomon]

I achieved having this working, but is horrible the integration at the beginning, mainly when deploying on a new machine...

[RichiH]

@txomon Can you share your hack, no matter how horrible?

Michael McCracken has asked for ways to make this work. He may be on GitHub as @mccracken ; I am not sure.

[txomon]

It's not really a hack, is just using submodules.

I started from a home in which I had omz, and vim plugins, and added them as submodules. The problem on updates is that you have to register them one by one, saying which submodules there are and which is the clone url. I created a .gitmodules file for that.

I must say it really takes some time to get it deployed in a computer, but after that, everything goes smoother.

My dotfiles https://github.com/txomon/dotfiles