Versioning Dotfiles with Git

It’s become quite common to see ‘dotfiles’ repositories across GitHub, and for good reason. Versioning your home directory is a great way to maintain backups of your dotfiles, share with others, learn from others, and makes configuring a new machine a bit easier. Many users, however, don’t actually version their home directory. A common approach is to have a dotfiles repo somewhere (let’s say ~/dotfiles) and use a number of different ways to get the actual dot-files into the home directory itself: manually copy, manually symlink, or scripting the copy/symlink.

I’m not a fan of the copy approach. Any changes made to the actual dotfiles must to be manually replicated in the repo. I’m also not a fan of the symlink approach. It solves the problem of modifications but doesn’t account for new files. So why don’t we version the home directory itself?

I’m sure there are some good reasons not to version the home directory, and I’d love to hear them. I can’t think of concrete examples where having a ~/.git directory is a bad idea, but I’m sure there are a few. Regardless, here’s my current setup.

My Setup

My dotfiles repo is cloned under ~/dotfiles. Git has a setting (core.worktree) that configures where the working tree should be checked out. It expects a path that can be either absolute or relative to the ‘.git’ directory. To accomplish this, from my home directory:

$ git clone --no-checkout https://github.com/jasonkarns/dotfiles.git
$ git config core.worktree="../../"
$ git checkout master

This will do a normal git checkout, but instead of checking out the master branch to ‘~/dotfiles’, it checks out to the home directory itself. The repo itself (‘~/dotfiles’) is completely empty, except for the ‘.git’ directory.

Benefits

  1. Any changes to my dotfiles are known to git
  2. New files are known to git
  3. The home directory itself is not a git repo, yet it’s fully versioned.
    • git commands cannot be run under ~
    • git status is not displayed in my command prompt ($PS1) under ~
  4. To manage the dotfiles repo and run git commands, I must be in ‘~/dotfiles’. (I like the forced context switch.)

This setup has worked well for me. I have the benefits of a versioned home directory, without the annoyance of it being an actual repo (like seeing git status info in my command prompt).

Complications

There is one major complication: submodules. I use Vundle to manage my Vim plugins. Vundle itself is a git submodule under ‘dotfiles/.vim/bundle’. In order to run git submodule update, git requires that I be in the repo (duh) but also that I be in the working tree root. Since my working tree root is not in the repo, I get an error:

$ git submodule update
fatal: Not a git repository (or any of the parent directories): .git

To get around this, git has a feature wherein a plaintext file named ‘.git’ is placed in the root of the working tree. It contains just a single line: gitdir: /path/to/actual/repo/.git. While this file exists, the home directory itself becomes, for all intents and purposes, a proper git repo. I can run git commands directly from the home directory and even my command prompt picks up the git status info.

So, with ‘~/.git’ containing gitdir: /Users//dotfiles.git, I am able to properly run git submodule update and everything works! Of course, as long as this ‘.git’ file exists, my home directory is essentially a proper git repo, so once I’ve run any necessary commands, I simply delete the ‘.git’ file, and now I’m back to a plain, non-repo home directory!

Subdirectory Checkouts with git sparse-checkout

If there is one thing I miss about SVN having switched to git (and trust me, it’s the only thing), it is the ability to checkout only a sub-tree of a repository. As of version 1.7, you can check out just a sub-tree in git as well! Now not only does git support checking out sub-directories, it does it better than subversion!

New Repository

There is a bit of a catch-22 when doing a sub-tree checkout for a new repository. In order to only checkout a sub-tree, you’ll need to have the core.sparsecheckout option set to true. Of course, you need to have a git repository before you can enable sparse-checkout. So, rather than doing a git clone, you’ll need to start with git init.

  1. Create and initialize your new repository:

    mkdir  && cd 
    git init
    git remote add –f  
  2. Enable sparse-checkout:

    git config core.sparsecheckout true
  3. Configure sparse-checkout by listing your desired sub-trees in .git/info/sparse-checkout:

    echo some/dir/ >> .git/info/sparse-checkout
    echo another/sub/tree >> .git/info/sparse-checkout
  4. Checkout from the remote:

    git pull  

Existing Repository

If you already have a repository, simply enable and configure sparse-checkout as above and do git read-tree.

  1. Enable sparse-checkout:

    git config core.sparsecheckout true
  2. Configure sparse-checkout by listing your desired sub-trees in .git/info/sparse-checkout:

    echo some/dir/ >> .git/info/sparse-checkout
    echo another/sub/tree >> .git/info/sparse-checkout
  3. Update your working tree:

    git read-tree -mu HEAD

Modifying sparse-checkout sub-trees

If you later decide to change which directories you would like checked out, simply edit the sparse-checkout file and run git read-tree again as above.

Be sure to read the documentation on read-tree/sparse-checkout. The sparse-tree file accepts file patterns similar to .gitignore. It also accepts negations—enabling you to specify certain directories or files to not checkout.

Now there isn’t anything that svn does better than git!

Merge Two Git Repositories Into One

A few weeks ago I tweeted: “Just did 2 subtree merges in order to combine 2 partially-related git repos into a single repo and still maintain history. #gitrocks” Wanna learn how to do it? Here we go…

TL;DR

# create new project as the parent
$ mkdir new_parent_project
$ cd new_parent_project
$ git init
$ touch .gitignore
$ git ci -am "initial commit"

# merge project A into subdirectory A
$ git remote add -f projA /path/to/projA
$ git merge -s ours --no-commit projA/master
$ git read-tree --prefix=subdirA/ -u projA/master
$ git ci -m "merging projA into subdirA"

# merge project B into subdirectory B
$ git remote add -f projB /path/to/projB
$ git merge -s ours --no-commit projB/master
$ git read-tree --prefix=subdirB/ -u projB/master
$ git ci -m "merging projB into subdirB"

The most common use case for sub-tree merges, that I’m aware of, is to merge another git repository into a subdirectory in an existing repository. There are quite a few tutorials which cover these steps. In fact, the second two sets of commands above do exactly that. However, I discovered that in order to merge two repositories into a new repository, the new repository must already have a prior commit in it. Otherwise, the sub-tree merges will not work as planned. So, as in the first set of commands above, be sure to create at least one initial commit prior to doing the sub-tree merges. It can be as trivial as committing an empty file. But without it, the merges will not work correctly. Armed with this knowledge, you can follow the tutorial on GitHub on merging sub-trees. Or you can follow along with me here.

Create Parent Repo

First, create a new, empty project to act as the parent project for our two existing repositories.

Jason@BRUTUS ~/dev
$ mkdir parent

Jason@BRUTUS ~/dev
$ cd parent/

Jason@BRUTUS ~/dev/parent
$ git init
Initialized empty Git repository in ~/dev/parent/.git/

Now we need to create the initial commit. This is essential.

Jason@BRUTUS ~/dev/parent (master #)
$ touch .gitignore

Jason@BRUTUS ~/dev/parent (master #)
$ git ci -am "initial commit"
[master (root-commit) fc6f5ad] initial commit
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 .gitignore

Merge Project A Into Subdirectory

Next, we add a remote to the first project we’d like to import. We’ll give the remote a name (projectA) and pass the -f option so that it will fetch the contents of this remote immediately.

Jason@BRUTUS ~/dev/parent (master)
$ git remote add -f projectA /path/to/projectA
Updating projectA
warning: no common commits
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 16 (delta 7), reused 0 (delta 0)
Unpacking objects: 100% (16/16), done.
From /path/to/projectA
 * [new branch]      master     -> projectA/master

Now, let’s run a merge but not commit the result (--no-commit flag). We also need to specify the merge strategy ours with the -s switch.

Jason@BRUTUS ~/dev/parent (master)
$ git merge -s ours --no-commit projectA/master
Automatic merge went well; stopped before committing as requested

Now that we are in merging mode, we’ll read in the tree from the remote, taking care to provide a subdirectory into which the subproject will go. This is specified with with --prefix switch. Also, add the -u flag to update the working tree with our changes.

Jason@BRUTUS ~/dev/parent (master|MERGING)
$ git read-tree --prefix=projA/ -u projectA/master

The remote has been merged into its own subdirectory and the changes have been staged. Now we can simply commit them.

Jason@BRUTUS ~/dev/parent (master +|MERGING)
$ git ci -m "merging project A into subdirectory"
[master 4d2d50d] merging project A into subdirectory

Merge Project B Into Subdirectory

At this point, we have Project A merged into its own subdirectory within our new parent project. Merging in Project B uses the same simple steps as above.

Jason@BRUTUS ~/dev/parent (master)
$ git remote add -f projectB /path/to/projectB
Updating projectB
warning: no common commits
remote: Counting objects: 47, done.
remote: Compressing objects: 100% (47/47), done.
remote: Total 47 (delta 23), reused 0 (delta 0)
Unpacking objects: 100% (47/47), done.
From /path/to/projectB
 * [new branch]      master     -> projectB/master

Jason@BRUTUS ~/dev/parent (master)
$ git merge -s ours --no-commit projectB/master
Automatic merge went well; stopped before committing as requested

Jason@BRUTUS ~/dev/parent (master|MERGING)
$ git read-tree --prefix=projB/ -u projectB/master

Jason@BRUTUS ~/dev/parent (master +|MERGING)
$ git ci -m "merging project B into subdirectory"
[master 8f41792] merging project B into subdirectory

Pulling In Updates

If the original repositories (Projects A and B in this example) continue to live on elsewhere as separate projects, you can easily pull in updates to your new parent repo. Using the sub-tree merge strategy, the updates will be applied properly to the applicable subdirectory.

Jason@BRUTUS ~/dev/parent (master)
$ git pull -s subtree projectA master

However, if you no longer have any need for the original repositories, they can be deleted and the remotes in your new parent project can safely be removed.

Below is a screenshot of a repository after two sub-tree merges. The repositories that I merged were two separate userstyles: one for using tab color for notifications (gtab) and another for adding S/MIME icons to gmail’s inbox (gmail). Each of these two projects have their own history that was maintained after the merge. Now they are each in their own subdirectory in a common userstyles git repository. As you can see, the two projects each have their own lines of development that do not include any common ancestry until the merge point.