Git worktree - clone but not quite

Have you ever wanted another workarea for the current repository you're working in?  Maybe you're running some tests and need the normal workarea to stay unchanged, so you can't rebase or tweak another branch in the meantime.

Options up to now:

• $ git clone [current-repo] [temp-repo]; cd temp-repo  # too heavy-weight, have to push changes back
• $ sleep 120; check email  # interrupts flow

In Git 2.5.0, you can do this easily:

• $ git worktree add ../temp master

This creates a new workarea in ../temp with all the current branches.  It's the same repository!  Any git command you do in the new workarea is applied to (and uses the database of) the original repository: commit, rebase, push, etc.

NOTE: If you leave off the branch name, 'git worktree add' creates a branch named after the new worktree directory.

NOTE: If you want to operate on the same branch as the original repository, it is disallowed by default.  In order to operate on the same branch, you have to say $ git worktree add ../temp master -f'.  Also if you ever move off of the branch and want to switch back when another worktree has the same branch checked out, you have to use an annoyingly-long option: $ git checkout master --ignore-other-worktrees.  You could put that in an alias like so: $ git config --global alias.co "checkout --ignore-other-worktrees".

Paying Down Learning Risk

I've heard: "Solve the hardest problem first."  As a rule of thumb, that works great to reduce risk early on a software project.  But I found myself saying something different to a co-worker recently:

Sometimes I start with the hardest problem, but sometimes I like to start with a really easy problem.  Why do I do that?

Why would it be a good idea to start with the easiest problem?  What kind of risk are you trying to pay down in a given situation?

Here are some reasons that would justify breaking the "hardest problem first" rule:

• If you need to gain experience in a new domain, starting with something easy can help you get experience before tackling the harder problems.
• If the world has changed out from underneath an existing system in an unpredictable way, starting with changing something easy or predictable can help you observe the source of the chaos.
• If you are sharing work, handing the easy work items out to others based on their learning goals can help them learn better.
• If tackling a hard problem will take a very long time, and others are waiting for you, then picking an easier part of the problem can help ease integration while still letting you engage on the hard problem.

The kind of risk you want to pay down first is important.  Here are the kinds of risk that would be payed down by the above behaviors:

• risk of getting lost while learning
• risk of being unable to bring order to a chaotic system
• risk of assigning impossible tasks to someone who just wants to ramp up
• risk of high integration costs because of trying to change too much at once

Most of the time, the risk caused by the uncertainty inherent in solving a hard problem is the most important risk to pay down first.  But sometimes, there are other factors at play, and other subtle variables that need to be managed to achieve a successful group outcome.

Thank you to Michael Nelson for his instructive collaboration on this topic.

Painless localhost demos

Quick demo?  Easier than heroku?  Look at @ngrok.

I have periodically needed something that lets me painlessly set up a demo from my laptop that I could just email a link to anyone on the internet.

I guess that ngrok.com would be a pretty valuable target to pwn.  Maybe it wouldn't be too hard to install on my own host.

Thanks to @lmonson for retweeting about @ngrok.

Map-reduce-friendly graph traversal

You may be well acquainted with this, but I thought I'd pass it along.

    https://giraph.apache.org/intro.html

It does iterative graph processing.  The cool thing is that it lets you break down your graph traversal into substeps that are distributed.

Here is some sample code:

  public void compute(Iterable<DoubleWritable> messages) {
    double minDist = Double.MAX_VALUE;
    for (DoubleWritable message : messages) {
      minDist = Math.min(minDist, message.get());
    }
    if (minDist < getValue().get()) {
      setValue(new DoubleWritable(minDist));
      for (Edge edge : getEdges()) {
        double distance = minDist + edge.getValue().get();
        sendMessage(edge.getTargetVertexId(), new DoubleWritable(distance));
      }
    }
    voteToHalt();
  }

Anyway, it looks like a very interesting project.

If I can do map reduce over change history, then it would then be possible to do contribution graph processing -- deep inspection of the graph of users and records they edit, possibly overlaid with the graph of users and which records they are connected to by ancestry.

For example, given a defined group of users, it would be interesting to see how their edits overlap with each other.  Another example would be to determine which top 1000 editing users made the most "distant" edits, based on some kind of ancestral distance measure.

Kent Beck about steering code via tests

I was about to send this to my team, but then I realized this is a blog post.

Here is Kent's article:

    https://www.facebook.com/notes/kent-beck/driving-development-tests-as-a-steering-wheel/719644314735102

And here is a related tweet from him on that topic:

    "selecting the next test is an act of design"

    https://twitter.com/KentBeck/status/438709708887560192

I've always thought about "design" as choosing which classes exist and how they relate to each other, and what methods look like, etc.  Or at a system level, deciding which services exist and what their core of responsibility should be.

However, I never thought about gaming my own mind (like applying genetic algorithms to my own thoughts), and choosing to introduce tests in different orders -- in order to produce a different neurophysical response from myself.

Perhaps "design" is not just about a static system and what to include.  Perhaps "design" is about acknowledging the dynamism of the human/computer programming environment, and leveraging that dynamic nature in order to get a more optimum response.

Modular Apps

In the last 12 months, it seems like apps are getting a lot more modular.

I discovered news.me before Christmas last year -- which felt modular at an app level -- RSS => social, and the reciprocal daily digest.  In fact, I've been reading their daily Twitter summary basically ever since, even though I am effectively a read-only non-citizen on Twitter.

It's old news to a lot of people, but when I saw IFTTT and Wappwolf, my mind was blown.  This is modularity at an app level, not just a code level.

Here is a good outline of what kind of things are possible with this new breed of app-level integrator apps:

http://www.readwriteweb.com/hack/2012/04/4-cool-things-you-can-do-with

Minus the fairly major hole of lack of HTTPS support from IFTTT, it looks awesome.  When that's fixed I'll be investing a fair amount of time/automation into it.

Awesome CLI gem: Commandable

Ever wanted to drive your ruby code with a CLI?  Well, the commandable gem makes it easy.

At first, I struggled with OptionParser in the stdlib, but I had to duplicate my option handling across CLI code and method invocation.  Then OptionParser didn't handle required parameters at all, requiring me to do required parameters in a non-DRY way.   Also, OptionParse doesn't do sub-commands, git-style, which I needed for the new git plugin I'm working on.

When I saw the pickled_optparse gem (from Mike Bethany), it appeared to solve the missing 'required parameter' feature.  And the subcommand gem seemed to allow for subcommands with their own options, git-style.  However, the duplication across option definition and method invocation was still present.  And even subcommand required you to hook up the command to the class/method that needed to be called.

Finally I found that the author of pickled_optparse had totally rethought CLI in ruby with his commandable gem.  And I really like what he did.

Now there is no real additional logic AT ALL in order to hook your ruby class up to the command line.  Now my option handling code will be in the place it belongs, in the code that is doing the command that I invoked from the command line.

The main awesomeness here is that commandable pushes the CLI code into the appropriate place in your ruby class -- it makes it hard to scatter CLI code all over the place.

My only beef with the gem are the following points:

• it comes with colors enabled by default (unusual for a CLI)
• it clears the screen by default when displaying help (when colors are enabled, very unusual and annoying for a CLI)
• it appears to be hard to do global options like: "cmd --global subcommand [args]"
• there is no support for parsing command CLI-style options into ruby hashes, but this is a minor nit, because that's what OptionParser is for, and it's easy to use inside the method

I fixed a problem where the screen was cleared even if you disabled colors, and I've pushed it up to my fork.

Here is my hello world that shows commandable in action:

And here's the output for a few representative cases:

Intrusion Detection through Stackable Filesystems

I've always wondered what exploit might be running on my system, and never had any time to devise/install a detection system that would have the right balance of useful detection (maximize) and performance impact (minimize).

When I stumbled upon unionfs a couple weeks ago, I thought that was an interesting idea from a change-logging perspective.  It's sometimes useful to be able to keep a filesystem-based diff of what a certain operation does to a system, and then bake it onto the system when I know it did what I wanted to.  The takeaway for me was that unionfs's performance profile had the opportunity to be so low because it was so thin and so baked into the kernel.

This compounded with my recent discovery and fiddling with fusefs (in user-land) and I wondered about what kind of useful logic I could put underneath a filesystem.  The GlusterFS feature set, the recent LessFS GC stuff, the bup-fuse stuff, and the S3FS stuff is all just *really* cool.  I ended up gazing longingly at the big backlog of fuse filsystem suggestions on their wiki, and wandered back into the unionfs space.

So, when I saw the I3FS paper (linked from here) about the modular application of this technique to intrusion detection, this really triggered the Useful *AND* Performant neurons and I got really excited.

Unfortunately, on first glance, the stackable filesystems stuff seemed pretty cryptic to set up in a lightweight, just-works kind of way (think custom mount command-lines complete with arcane stacking options).

It would be soooo awesome to have an easy-to-compose ruby DSL for doing some kind of rack-like filesystem mashup with a kernel-level unionfs layer underneath a user-land fusefs layer, but all expressed in the same DSL.

This would be an awesome tool to put on top of the Arch-derived clone I want to put up for people at work.  There are folks who care about living more on the edge of linux stuff, but that don't care to install from scratch, and also might not care that much about not having a full Gnome stack if things just work.  And if I could give them the same tuned IDS-on-the-desktop solution (or upgrade their developer stack by letting them pull a filesystem delta over), that would be really cool.

The cheap development observation [PDF, linked from] because of modularity is one of attributes valued highly by the Ruby community as well.  It is one the key things that makes Ruby as a community awesome.

These kinds of ideas really matter, and making them so cheap and stable that you don't have to think about them really matters even more.

NOTE: The I3FS paper is really pretty old (2004), and the whole unionfs stack is older than that.  The fuse stack came into the kernel a long time ago too (2005).  So while this is new to me, it's been around for quite a long time.  I'm playing catch-up.

Semantic Versioning

After having skimmed the semantic versioning proposal/spec, I really like it, and I'm going back for a deep read.

The most notable violator of this that has bit me in the past has been the jersey framework, and maybe earlier versions of commons-collections.

Some Git/Ruby stuff of interest

While taking the git survey a while back, I encountered:

• git-annex: awesome git-enabled large file archive support

which led me to:

• git-bup: way cool! git-like backup, even serves as a remote for git-annex

also this morning ran into:

• rib: irb, except better, including rib-heroku
• rest-core: like rack, except for RESTful clients
• gemgem: easy gem building, without all the hoe dependency cruft

I just wish that gem-man read the README{,.md} in the root of a gem (like gemgem does) if that gem didn't have a manpage.  Sounds like a good patch to send.  That will help me understand how gems can access their own filesystem post-install.

Thought you might like to know.

And btw, rbenv is way nicer to interact with than rvm, at least that's been my experience over the last couple weeks.

Git as Menograph

Background

In Mar. 2010, Matz came and talked at MWRC.

The awesome talk focused around Ruby as a great invention that makes it possible for programmers to bring a new world into being that could only be imagined before.

He made reference to the inventor in Hugo Gernsback's book: Ralph 124c 41 +.  The name was a play on words "One to foresee for one", with "+" being reserved for the 10 or so greatest inventors currently living.

Although Matz's reference could be interpreted as prideful, anyone who has ever had anything to do with him would correct you -- he is a very humble programmer who does awesome stuff.

Because I wanted to understand Matz's reference very thoroughly, I went and read the book.  I was able to appreciate the scope of Matz's talk much more clearly after having read about Ralph, the great inventor.

Menograph, the invention

One thing that Ralph invented was a Menograph.  It was a mind-recording device, operated by pressing a button that started a mind wave recorder and operated a scroll of fabric on which the waves were traced in ink, similar to an old-style seismograph.

The Menograph was one of Ralph 124C 41 +'s earliest inventions, and entirely superseded the pen and pencil.  It was only necessary to press the button when an idea was to be recorded and to release the button when one merely reflected and did not wish the thought-words recorded.

Those thoughts could then be replayed again verbatim by any person by use of a device called a Hypnobioscope.

After reading an account of Ralph using this invention, which is more detailed than I am able to reproduce here, I had a vague sense that I had experienced a similar feeling before while programming.

Git as Menograph

After thinking about it for a while, I realized that how Ralph used his Menograph is how I use git.

While coding, I work on stuff, then I record commits into git.  Then I work on stuff some more, then record commits.

Then I take passes back over what I've recorded and get rid of the bad stuff and keep the good stuff.  Then I push it out to the team.  Sometimes, a pair or triple on the team do this as a group exercise, working history over a few time until it won't break the world.

The parallel to git was eye-opening.  In fact, after having noticed this behavior in myself, and seeing the behavior tendency grow over time, I'd say that git encourages multi-pass thinking styles.

P.S.  If you care at all about Ruby, listen to Matz's talk.  Some of the future stuff he was talking about is already in Ruby 1.9.2.  NOTE: I'm on linux, and I had to use the flash player to get the video to show.

Attention to Error vs Attention to Detail

After having read Talent Code, a friend pointed me to the following article:

Why Do Some People Learn Faster?

Although the author of that article had gotten some lackluster reviews on some of his book-length work on Amazon, this article brought a distinct idea to the forefront for me:

Attention to Error

I've always thought that attention to detail was an important tendency in myself that made me a good learner.  But I realized that there is a distinction between paying attention to detail and paying attention to error.

The constant reconciliation between what you believe to be correct and what turns out to be correct is the fuel for my personal learning.

Some people might say this is what the scientific method is all about.  I'm less methodical than a scientist would be -- I find it nevertheless useful to constantly put myself in situations where I can learn informally from comparing what I expect to what actually happens.

Keeping a Beginner's Mind

As an experienced software developer, I care deeply about retaining my ability to remain flexible in my habits and learning style.  That's the only way I got good, and if I ossify in my learning habits, I'll end up the equivalent of a COBOL programmer.  Certainly not the way to live up to the broad possibilities that exist to make the world a better place.

I appreciated the wider perspective that I got from glancing through these slides by Patrick Kua:

The Beginner's Mind

Although I had seen and skimmed Pragmatic Thinking & Learning, this presentation was a gentle and useful introduction to the whole idea of the Dreyfus model of skill aquisition.

Another very useful idea that this presentation expressed was the contrast between "skill-acquiring apprentice" vs. "closed-minded expert".  Patrick said that the "skill-acquiring" attribute can also apply to highly-skilled practitioners.

Among the tips Patrick gave, the following were useful to me:

• You can't be an expert on everything. [so don't even try]
• How can I try this safely?
• How does this fit in my world?
• Remain curious.
• Mix with diverse groups.
• Beware of built-in biases.
• Avoid judging early.

The reason I decided to compare/contrast some of the ideas in this presentation was actually because of my forays into the Pragmatic Thinking & Learning book, which was one of three "Further Resources" offered at the end.

Patrick recommended the following books for further learning:

• Pragmatic Thinking & Learning
• Apprenticeship Patterns
• Changing Software Development

Because of the context in which the Apprenticeship Patterns book appears, I want to read that as a next step for learning how to make a real difference at FamilySearch.

Core Goals of Version Control

After having a couple of years of experience with Git, I was finally able to clearly articulate the core value that version control brings to software development. This article will be written with a bias toward Git, but I've at least tried to express the core value in abstract terms.

The value is expressible in 4 goals.

Core Goals

1. Capture source artifacts
2. Isolate stable codelines from WIP
3. Enable concurrent WIP
4. Make it easy to leave sensible history behind

WIP = Work In Progress

The continuous delivery crowd says that WIP should be basically zero. I think that there is space for a fairly short development pipeline (3-5d) that can free developers up to capture ideas and then wrangle them into shape in a second or third pass.

Capture Source Artifacts

As a developer, I feel happy when I can push a save button when I've taken a small step -- and then never have think about it again. I also feel happy when I can save, even if I'm based on a slightly out-of-date base. I also feel happy when I don't have to worry about binary vs. text -- as long as its a true source file of a reasonable size.

I feel happiest when I don't have to be distracted by superfluous concerns, and can focus on the task at hand and save the results easily.

Main Point: A good version control system makes it easy to capture source artifacts.

Isolate Stable Codelines From WIP

Software is hard to get right, and once things get stable, the only way to keep them there is to avoid making risky changes. But stability is always balanced against the common desire for enhancement and restructuring.

A typical bug fix is one or two isolated commits that fix a specific problem. Most version control systems more or less support cherry-picking a small change over into another codeline.

I feel happy when I'm able to easily take an isolated series of commits from the development codeline into a stable one, even if the fix turns out to be a little larger than a couple commits.

Main Point: A good version control system makes it easy to cherry-pick changes, and even longer patch sequences, from one codeline to another with easy reconciliation of overlapping edits.

Enable Concurrent WIP

For software developed by a team larger than 2-3 people, it is useful to be able to work in parallel with each other, even on the same feature. Sometimes you can arrange your work so as not to overlap at all, but there are real situations where you want to track someone else's development within the fairly short development pipeline (3-5d) before things have gotten stable.

I feel happy when I don't have to catch people up on what I've been doing, and when they don't have to catch me up on what they've been doing, and we can get to the point easily and quickly. I also feel happy when we can leap-frog each other with ideas and real implementation.

Main Point: A good version control system makes it easy to track other team members' work, and easy to tentatively integrate with that work.

Make It Easy To Leave Sensible History Behind

After stepping away from a piece of software, it is easy to lose context and forget the concerns that shaped the development of that software. It doesn't take long. For me, about 2 weeks is enough for me to start forgetting details and motivations.

Of course, it is important to leave a properly structured artifact behind. Proper naming means a lot; proper factoring is important. Once you comprehend the structure, it is possible to change things without introducing big problems.

But often, full comprehension isn't practical, and a flat 2D view of the artifact is insufficient for reasoning about the thing itself. Often, it is often very useful to be able to get a vector on where the software came from, and interpolate from that vector where it was headed.

The creative process is messy, filled with double steps and backtracking. Don't confuse me with all the noise. Give me a history that, given all the knowledge you had when it got stable, at least looks well-reasoned.

I feel happy when I'm able to take an easy editing pass following capture/review that lets me intentionally craft history as an email to the future about the vector of change that is inherent in a certain patch sequence. I also feel very happy to not have to think about this during the creative, messy, focus-draining capture/review pass.

Main Point: A good version control system includes tools to both: 1) enable the developers to easily leave behind meaningful history, and 2) extract focused history for all or part of a piece of software.

Revert by Commit

Here is a sequence that happens to me regularly:

1. Have a theory about how to fix something
2. Try it out
3. Find out it was a stupid theory
4. Realize that there is a kernel of learning in what you tried
5. Realize that you don't want to throw away that kernel of learning quite yet
6. But want to get the bad changes out of the way
7. But there are some good uncommitted changes you don't want to throw away
8. Ack, what should I do now?

Here is a recipe that makes it really easy to save the bad experiment *and* get it out of the way in ONE step.

• git checkout -b experiment-that-stunk
• git add -i (pick the one or two files that contain the changes you want to get rid of)
• git commit -m "tried to do something stupid, but it didn't work because ..."
• git checkout stuff-i-was-working-on

Revert by commit. Now the changes are gone, saved, explained, pickled for fermentation for more ideas or conversation.

That is something that was NOT part of my experience before git.

Actually, I take that back, I did this once or twice before, but I wasted lots of time because I resisted give up on the experiment at the first sign of stupidity -- I subconciously knew it would be painful to save a whole patch, partially revert, hope I got it right, forget what I was doing, mess it up, revert all the way, re-apply the patch, repeat. So it took longer to abort the bad experiment because of the hesitation because of the perceived expense of partial revert.

UPDATE

I just realized this must be how "git stash" is implemented, and that it's definitely not a new idea.