Git worktree - clone but not quite

Have you ever wanted another workarea for the current repository you're working in?  Maybe you're running some tests and need the normal workarea to stay unchanged, so you can't rebase or tweak another branch in the meantime.

Options up to now:

• $ git clone [current-repo] [temp-repo]; cd temp-repo  # too heavy-weight, have to push changes back
• $ sleep 120; check email  # interrupts flow

In Git 2.5.0, you can do this easily:

• $ git worktree add ../temp master

This creates a new workarea in ../temp with all the current branches.  It's the same repository!  Any git command you do in the new workarea is applied to (and uses the database of) the original repository: commit, rebase, push, etc.

NOTE: If you leave off the branch name, 'git worktree add' creates a branch named after the new worktree directory.

NOTE: If you want to operate on the same branch as the original repository, it is disallowed by default.  In order to operate on the same branch, you have to say $ git worktree add ../temp master -f'.  Also if you ever move off of the branch and want to switch back when another worktree has the same branch checked out, you have to use an annoyingly-long option: $ git checkout master --ignore-other-worktrees.  You could put that in an alias like so: $ git config --global alias.co "checkout --ignore-other-worktrees".

Why Instead of What

Writing a good commit message is an important collaborative skill on a software project.

See:
- http://who-t.blogspot.de/2009/12/on-commit-messages.html
- http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
- https://github.com/torvalds/linux/pull/17#issuecomment-5661185

While all the formatting advice can seem pretty nitpicky, there are a couple principles that stand out to me:

1. Above all, clearly and concisely state WHY you are making the change
2. Also, include all the considerations that went into this change and why you are making THIS change instead of any others you considered

Even if the formatting isn't perfect or wouldn't meet the linux kernel mailing list standards, just having the context later when you are scratching your head wondering "why?" - that is invaluable.

Rewriting Git History for Fun

For starters, if you're looking for a good JDBC connection pool, look no farther than c3p0 [github, doc].

Back a few years commons-dbcp wasn't very stable under load, so I went looking for an alternative and landed on c3p0.  Ever since then, Steve Waldman has been making it better and better.  Except for a 2-year hiatus when he wasn't working on it, he's been very responsive and willing to accept feedback and make improvements.  I'm seriously impressed by the project.

Anyway, since I've been a long-time user and fan, when I saw Steve put his software on GitHub, I went for a look.

Here is what I saw:

But what about all those prior releases in source snapshot form, that I was used to seeing from SourceForge?

I realized that perhaps I could contribute a little to the project, so I went and grabbed all the source release zips from SourceForge, created a local git repo, created a commit for each release, tagged it, and spliced Steve's recent work on the top.

Then I submitted a GitHub issue to ask Steve what he thought -- wasn't really a pull request because it was a completely disconnected history.

Here is the top of the new history:

And here is the earliest part of that history:

The tools used were:

- git config author.name - to give credit where credit was due

- git config author.email

- curl - to download all the releases

- unzip | sed - to figure out what the commit date should be

- git commit --date "[release date]" - to create the commits

- vi .git/info/grafts - to temporarily splice the new history on the old

- git filter-branch --tag-name-filter - to rewrite the new history permanently on top

- git tag -f - to replace the existing release tags to point to the new rewritten history

- git push --tags - to push it up to github

If I could consult for projects / companies to do this kind of VCS conversion work and actually get paid for it -- wouldn't that be awesome!

Useful Git Tips

Every one of these tips was useful to me:

• http://mislav.uniqpath.com/2010/07/git-tips/

It just amazed me that these were written over a year ago, and I've been trying to learn as much about git as I could, but didn't even stumble across these in manpages.

I found this by looking for an option that would let "git remote update" only fetch a subset of the remotes I have attached.

It is rewarding to feel like I'm working with a rich toolset.

Thanks to @mislav for posting this.

BTW: Poking around in his tweet stream also yielded this gem.  I've always wondered if there was a lightweight XPath for JSON, and there it is.  Some background is in order, as much of the "XPath for Javascript" mentality was based on early JQuery thoughts.

Some Git/Ruby stuff of interest

While taking the git survey a while back, I encountered:

• git-annex: awesome git-enabled large file archive support

which led me to:

• git-bup: way cool! git-like backup, even serves as a remote for git-annex

also this morning ran into:

• rib: irb, except better, including rib-heroku
• rest-core: like rack, except for RESTful clients
• gemgem: easy gem building, without all the hoe dependency cruft

I just wish that gem-man read the README{,.md} in the root of a gem (like gemgem does) if that gem didn't have a manpage.  Sounds like a good patch to send.  That will help me understand how gems can access their own filesystem post-install.

Thought you might like to know.

And btw, rbenv is way nicer to interact with than rvm, at least that's been my experience over the last couple weeks.

srev-taming: Migrating Subversion to Git

After reading about Mediawiki's pending svn => git migration, it sounded very familiar to me, because I was proposing such a conversion at work about a year ago.

Mass svn => git migration tooling doesn't really exist in the way I wish.

The most popular tool is probably svn2git.  But even with all the conversion goodies baked into it, it's still based on git-svn.

Anything based on git-svn means that for an svn repo that has multiple projects in it, you have to take N passes over the repository, which is a huge hit for repos in the 20,000+ commit range.

The most performant conversion tool is svn-fe, but it doesn't do much more than just import at the repo boundary: 1 svn repo => 1 git repo.  And it doesn't even begin to deal with the situation of multiple svn histories as we have migrated from one repo to another.

The closest thing that exists is a set of scripts posted as part of this thread.

Here is a rough cut of a project I wish existed:

"srev-taming" ~~ "svn-migrate"
(anagram, in the spirit of "snerp-vortex" ~~ "svn-exporter")

• scan of SVN dump for projects & codelines => annotated projects & branches list
• easy invocation of svn-fe for mass-import => generate me a command-line
• editable auto-detected project tags & codelines => language that easily expresses projects, codelines & grafts
• easy clone/filter-branch invocation to extract & stitch codelines together (based on projects & branches list and configured grafts)
• post-import filter for author fixup, SVN & migration artifact removal => generation of starting author list & svn URL minifier

Unless I get explicitly assigned to do the migration, I guess it's not going to prioritize high enough for me to do anything about it.  At least svn-fe is being maintained and enhanced.

But I'm posting it here to see if anything exists that I don't know about -- or in case it sparks some ideas for someone to create this thing.

Git as Menograph

Background

In Mar. 2010, Matz came and talked at MWRC.

The awesome talk focused around Ruby as a great invention that makes it possible for programmers to bring a new world into being that could only be imagined before.

He made reference to the inventor in Hugo Gernsback's book: Ralph 124c 41 +.  The name was a play on words "One to foresee for one", with "+" being reserved for the 10 or so greatest inventors currently living.

Although Matz's reference could be interpreted as prideful, anyone who has ever had anything to do with him would correct you -- he is a very humble programmer who does awesome stuff.

Because I wanted to understand Matz's reference very thoroughly, I went and read the book.  I was able to appreciate the scope of Matz's talk much more clearly after having read about Ralph, the great inventor.

Menograph, the invention

One thing that Ralph invented was a Menograph.  It was a mind-recording device, operated by pressing a button that started a mind wave recorder and operated a scroll of fabric on which the waves were traced in ink, similar to an old-style seismograph.

The Menograph was one of Ralph 124C 41 +'s earliest inventions, and entirely superseded the pen and pencil.  It was only necessary to press the button when an idea was to be recorded and to release the button when one merely reflected and did not wish the thought-words recorded.

Those thoughts could then be replayed again verbatim by any person by use of a device called a Hypnobioscope.

After reading an account of Ralph using this invention, which is more detailed than I am able to reproduce here, I had a vague sense that I had experienced a similar feeling before while programming.

Git as Menograph

After thinking about it for a while, I realized that how Ralph used his Menograph is how I use git.

While coding, I work on stuff, then I record commits into git.  Then I work on stuff some more, then record commits.

Then I take passes back over what I've recorded and get rid of the bad stuff and keep the good stuff.  Then I push it out to the team.  Sometimes, a pair or triple on the team do this as a group exercise, working history over a few time until it won't break the world.

The parallel to git was eye-opening.  In fact, after having noticed this behavior in myself, and seeing the behavior tendency grow over time, I'd say that git encourages multi-pass thinking styles.

P.S.  If you care at all about Ruby, listen to Matz's talk.  Some of the future stuff he was talking about is already in Ruby 1.9.2.  NOTE: I'm on linux, and I had to use the flash player to get the video to show.

Core Goals of Version Control

After having a couple of years of experience with Git, I was finally able to clearly articulate the core value that version control brings to software development. This article will be written with a bias toward Git, but I've at least tried to express the core value in abstract terms.

The value is expressible in 4 goals.

Core Goals

1. Capture source artifacts
2. Isolate stable codelines from WIP
3. Enable concurrent WIP
4. Make it easy to leave sensible history behind

WIP = Work In Progress

The continuous delivery crowd says that WIP should be basically zero. I think that there is space for a fairly short development pipeline (3-5d) that can free developers up to capture ideas and then wrangle them into shape in a second or third pass.

Capture Source Artifacts

As a developer, I feel happy when I can push a save button when I've taken a small step -- and then never have think about it again. I also feel happy when I can save, even if I'm based on a slightly out-of-date base. I also feel happy when I don't have to worry about binary vs. text -- as long as its a true source file of a reasonable size.

I feel happiest when I don't have to be distracted by superfluous concerns, and can focus on the task at hand and save the results easily.

Main Point: A good version control system makes it easy to capture source artifacts.

Isolate Stable Codelines From WIP

Software is hard to get right, and once things get stable, the only way to keep them there is to avoid making risky changes. But stability is always balanced against the common desire for enhancement and restructuring.

A typical bug fix is one or two isolated commits that fix a specific problem. Most version control systems more or less support cherry-picking a small change over into another codeline.

I feel happy when I'm able to easily take an isolated series of commits from the development codeline into a stable one, even if the fix turns out to be a little larger than a couple commits.

Main Point: A good version control system makes it easy to cherry-pick changes, and even longer patch sequences, from one codeline to another with easy reconciliation of overlapping edits.

Enable Concurrent WIP

For software developed by a team larger than 2-3 people, it is useful to be able to work in parallel with each other, even on the same feature. Sometimes you can arrange your work so as not to overlap at all, but there are real situations where you want to track someone else's development within the fairly short development pipeline (3-5d) before things have gotten stable.

I feel happy when I don't have to catch people up on what I've been doing, and when they don't have to catch me up on what they've been doing, and we can get to the point easily and quickly. I also feel happy when we can leap-frog each other with ideas and real implementation.

Main Point: A good version control system makes it easy to track other team members' work, and easy to tentatively integrate with that work.

Make It Easy To Leave Sensible History Behind

After stepping away from a piece of software, it is easy to lose context and forget the concerns that shaped the development of that software. It doesn't take long. For me, about 2 weeks is enough for me to start forgetting details and motivations.

Of course, it is important to leave a properly structured artifact behind. Proper naming means a lot; proper factoring is important. Once you comprehend the structure, it is possible to change things without introducing big problems.

But often, full comprehension isn't practical, and a flat 2D view of the artifact is insufficient for reasoning about the thing itself. Often, it is often very useful to be able to get a vector on where the software came from, and interpolate from that vector where it was headed.

The creative process is messy, filled with double steps and backtracking. Don't confuse me with all the noise. Give me a history that, given all the knowledge you had when it got stable, at least looks well-reasoned.

I feel happy when I'm able to take an easy editing pass following capture/review that lets me intentionally craft history as an email to the future about the vector of change that is inherent in a certain patch sequence. I also feel very happy to not have to think about this during the creative, messy, focus-draining capture/review pass.

Main Point: A good version control system includes tools to both: 1) enable the developers to easily leave behind meaningful history, and 2) extract focused history for all or part of a piece of software.

Starting a box with git

In The Creative Habit Twyla Tharp talks about "starting a box" to keep track of all the creative ideas that come on a project.

I've used git to just start tracking ideas in digital form. A brand new git repo is my digital box. Usually, the ideas start out as a whiteboard diagram or scratching on paper, which I then scan to PDF and put in the folder. The organization varies, but chronological file naming has worked for me, as has thematic subfolder organization.

Being able to just capture the work frees me up to try new things and not worry about whether that stuff I just did is safe or whether I can get back to it if I try something new out.

Git does such great cross-file compression that even after several revisions across binary files, the resulting repo sizes are surprisingly small.

An entire sprint without an SVN branch

A typical development flow has been:

• request an SVN task branch (wait for CM to help)
  
• do stuff (1-3 weeks)
  
• run all the regression tests (not on the latest stable)
  
• hope that nobody did anything that conflicted with us
  
• merge to the pending release (and pray we resolve any conflicts correctly)
  

Now we do the following:

• set up a team git repo (able to do this without CM's help)
  
• clone off a repo for each team member
• set up a daily cron job to get the latest stable from the pending release
• set up a daily hudson build to merge that in and push to team if it passes unit tests
• pull from team as we go (fast!)
  
• run all the regression tests (on the latest stable)
  
• use git to merge on top of the pending release and commit back to SVN (with minimal conflict windows)
  

Well, we finally got through a whole sprint without an SVN task branch. Maybe that seems like a small victory, but it saved us a bunch of late-breaking integration risk.

Nice benefits of git

I use git now and it saved my bacon last week. Two things made a really hard, late-breaking change possible:

1. Ease of integration enabled parallel development.
2. Smallness of commits meant easy bug isolation.

Ease of Integegration => Parallel Development
Because of the ease of integrating small commits, my coding companion and I were able to develop two complementary sets of changes in parallel. This saved us quite a bit of time because we didn't have to wait for each other. The changes were developed as a last-minute response to a problem right before release, and without the ease of integration, it would have been hard to get the problem right on a compressed schedule.

Small Commits => Easy Bug Isolation
Also, because commits are so small (because it's easy to commit without any side-effects, because your repo is isolated from everything else), it was easy to isolate the exact changes that caused a subtle problem. I could have used bisect, but I had an inkling which change caused it, so that was faster. With svn or a mongo patch accrued over multiple changes, it would have been much more confusing and disorienting.

Revert by Commit

Here is a sequence that happens to me regularly:

1. Have a theory about how to fix something
2. Try it out
3. Find out it was a stupid theory
4. Realize that there is a kernel of learning in what you tried
5. Realize that you don't want to throw away that kernel of learning quite yet
6. But want to get the bad changes out of the way
7. But there are some good uncommitted changes you don't want to throw away
8. Ack, what should I do now?

Here is a recipe that makes it really easy to save the bad experiment *and* get it out of the way in ONE step.

• git checkout -b experiment-that-stunk
• git add -i (pick the one or two files that contain the changes you want to get rid of)
• git commit -m "tried to do something stupid, but it didn't work because ..."
• git checkout stuff-i-was-working-on

Revert by commit. Now the changes are gone, saved, explained, pickled for fermentation for more ideas or conversation.

That is something that was NOT part of my experience before git.

Actually, I take that back, I did this once or twice before, but I wasted lots of time because I resisted give up on the experiment at the first sign of stupidity -- I subconciously knew it would be painful to save a whole patch, partially revert, hope I got it right, forget what I was doing, mess it up, revert all the way, re-apply the patch, repeat. So it took longer to abort the bad experiment because of the hesitation because of the perceived expense of partial revert.

UPDATE

I just realized this must be how "git stash" is implemented, and that it's definitely not a new idea.

Tracking work to be done

My team had the task of porting code off of an old API onto a new (hopefully-mostly-equivalent) API.

Diomidis Spinellis wrote an article in IEEE Software titled "Start with the Most Difficult Part" (on his blog here, official reference here). Now seemed like a good time to apply the advice.

So I wanted to measure: What was the most difficult part. I wanted to know how many usages of the old API were present and where I should focus my efforts first.

So I wrote macker rules to detect usages to port. And I wrote a ruby script that drove the ant macker target and pulled the usage counts into a list of counts. Then I published the current counts and started a graph so we could get the usages to zero.

Here is the chart we ended up with:


This drove us to two useful behaviors:

1. Getting our tests converted first, the most complex one first (exposed a lot of issues early).
2. Giving our work a metrics-driven feel -- all we have to focus on is getting this to zero.

There were also two things I missed:

1. The hardest task to port was one in which the new API didn't have anything near what we needed, and the usage measurement didn't catch that. ==> A quick pass over the tasks looking for especially risky ones (where the port would be really hard) would have exposed that earlier.
2. Another team was doing other work that intersected with ours, and we were unaware of the impact of their efforts on what we were doing. ==> Being able to get their changes quickly and easily may have helped expose the problem earlier.
   

Now that we've started using git, I'm going to suggest that we do two things:

• Get an SVN task branch so we can integrate as a team during the sprint
  
• Get an SVN task branch right before merging to release and re-apply all our git patches onto that task branch so the merge is really smooth

Or maybe if we can get done in time, we could volunteer to do that for the team that has the sprint-long task (getting them on everyone else's changes with a minimum of svnmerge.py headache).

Helpful git aliases

I know you can go all out with git aliases. My favorite complex alias was "Use graphviz for display", documented here.

Here are ones that are useful for me:
[alias]
b = branch
co = checkout
ci = commit -a -s
st = status
stat = status
l = !git-log --pretty --date=local --abbrev=10 --abbrev-commit | sed -e 's/^\\(commit .*\\)...$/\\1/' | less
k = !gitk
patch-apply = am
patch-gen = format-patch

Alias "b":
Alias "co":
Alias "ci":
Alias "st":
Alias "stat":
- so I don't have to type as much

Alias "l":
- so that I can see real dates, not UTC
- so that the commit numbers are clickable for copy/paste

Alias "k":
- in case I type "git k" instead of "gitk"

Alias "patch-apply":
Alias "patch-gen":
- in case I can't remember what "am" and "format-patch" mean (common occurrence)