Frame heap idea for Ruby

After listening to @srawlins great talk about new allocation stats features from @tmm1 in ruby 2.1, I got thinking about garbage collection in ruby.  BTW, MWRC 2014 is awesome so far!

I was wondering if there is some hybrid model that takes advantage of the call stack patterns that exist in server-side apps, or in apps that do a lot of iterative processing work.

If I could annotate a certain stack frame as a collection barrier, then I could write something that would modify collection behavior at that call boundary.  On the way in, I could pre-allocate per-call heaps that would be used up to capacity by all code from that point in the call graph on down.  On the way out, collection could be triggered to get rid of the garbage.

The benefits I can see here are as follows:

• there is greater locality of reference in the heap for objects allocated within a certain call graph
• objects that survive the frame heap collection would be merged into the global heap
• objects that overflow the frame heap space would either cause another linked frame heap to be allocated, or would overflow to the global heap
• there is greater opportunity for compacting the heap, per-call-graph collections imply compaction
• different collection barriers can be collected in parallel, because unless the call graphs won't overlap, even if they happen concurrently, so concurrent collection logic is potentially lock-free

Possible points of annotation as collection barrier:

• rack request
• http client request
• json parse / save
• File.readlines w/ block
• long-running loop{}s early in the stack

My thinking about heap and garbage collection is formed by what I've seen happen over the years in the HotSpot JVM heap structures.  First there was mark/sweep, with large pauses.  Then generational garbage collection came along, optimized for collecting new garbage, and only doing mark/sweep when an object has survived many early collection cycles.  Then there was the garbage-first collector which did memory region analysis, prioritizing entire regions of heap that are likely to contain the most garbage.

Ruby has seen some garbage collection innovation recently, and there is more to come in ruby 2.2.  However, maybe this is a novel idea that can help out.   Or maybe this idea isn't new and has been tried and has failed, but it was at least worth a try to get this out there.

Unanswered Questions

In my experience, programmers vary in their ability to tolerate ambiguity, or in their ability to proceed without an answer to a critical question.

For myself, I've had the sense that I have advanced in my ability to tolerate not knowing the answer to an important question.  However, it's only because of some coping mechanisms I've built up over time.  And without those coping mechanisms, I still basically stink at dealing with ambiguity at a core human level.

There is a sequence that I go through all the time:

1. No explanation yet
2. What is the real question?
3. How to file open questions while I'm working to find the real question?
4. What is the TTL on open questions?
5. How do I review open questions?
6. How do I forget to revisit something important?

And I always feel uneasy when it gets to #4-#6.  I realize that GTD is all about managing a fixed-size attention span, and keeping track of things that fall outside that attention span.  However, I stink at the paperwork part of GTD, but I try to apply some of the principles in the context of open questions.

Here is a list of ideas that relate to each other and are related to this overall theme:

• Ambiguous results
• Anomalous results
• Open question
• Loss reflex
• Disorientation cost
• Orientation rate
• Orientation ability
• Orientation cost
• Learning pipeline
• Fixed buffer size of open questions
• Open question LRU/LN cache eviction (least recently used, least needed)
• Isolating open questions in code

Here are some articles that relate to this theme:

• http://dailykibitz.blogspot.com/2007/04/tolerance-of-ambiguity.html
• http://www.cs.virginia.edu/~robins/YouAndYourResearch.html (search: "that trait is ambiguity")
• http://www.edcbmj.ir/browse.php?a_id=286&sid=1&slc_lang=en
• PragProg: Good Enough Software
• PragProg: Not Until You Are Ready
• PragProg: The Specification Trap
• PragThinkLearn: Embed Failing in Practice (p. 189)
• PragThinkLearn: Create an Exploratory Environment

This post is totally alpha and I don't even know where to go with it, but I wanted to get it out there to think about it some more, since I always think better after pressing "Post" than before.

Why Instead of What

Writing a good commit message is an important collaborative skill on a software project.

See:
- http://who-t.blogspot.de/2009/12/on-commit-messages.html
- http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
- https://github.com/torvalds/linux/pull/17#issuecomment-5661185

While all the formatting advice can seem pretty nitpicky, there are a couple principles that stand out to me:

1. Above all, clearly and concisely state WHY you are making the change
2. Also, include all the considerations that went into this change and why you are making THIS change instead of any others you considered

Even if the formatting isn't perfect or wouldn't meet the linux kernel mailing list standards, just having the context later when you are scratching your head wondering "why?" - that is invaluable.

Code doc: "Responsible for ..."

Instead of writing class documentation like:

/**
 * This class does X.
 */

Instead of that, please start writing class documentation like:

/**
 * Responsible for X.
 */

The difference seems subtle at first, however the latter is better because it sends you down the path of the Single Responsibility Principle as a default way of thinking about things.

If it's hard to write a sentence fragment "responsible for X" because X is really X, Y, and Z, then that is a code smell that can point to a need for more refined subdivision of responsibility.

Feeling Good with Creative Tension

This blog went dark after last summer because I had to focus on Boy Scout summer camp and moving my family to a new location.  But now those projects are largely taken care of, and I have some mental space to speak here again.

I have been reading two books recently:

• Creating
• Feeling Good

In Creating, the author says that it is more important to acknowledge the tension between your creative vision and current reality, despite the pain that comes from realizing how far away from your vision you might currently be.  This has been very hard to internalize for me.  I feel depressed when things that I care about look so impossible.

Well, I gave up on Creating for a while, and one of the books that I hauled to my new house was Feeling Good.  I found the core of the book in the description of the interaction between "feelings" and "thoughts".

Chapter 3 - Understanding Your Moods: You Feel the Way You Think

There, the author lays out a list of common mental errors that present themselves in people who suffer depressive symptoms.  The premise of the book is that cognitive therapy focused on correcting the mental errors is helpful to relieve depressive symptoms.

Reading these books together put the two thoughts in juxtaposition in my mind, and the result was interesting.  I realized that I was making some of the common mental errors in the context of creating my vision of a distributed versioned family tree.

So the principle I want to put into practice now personally is "Feeling Good with Creative Tension" by recognizing the mental errors and correcting them as quickly as is practical.

Despite the hope that these thoughts bring, I have had to refocus my efforts after being assigned to a project at work that deals with innovation on the entire corpus of pedigree data.  The things I have been learning at a whole dataset level are valuable to me in thinking about how to make a distributed tree even work.

Teaching Software

Today, I read "The Science in Computer Science" from Peter J. Denning in CACM (May 2013).  Very interesting article.

The most interesting part to me were the references to how CS ideas are beginning to be taught in high school and elementary school.  Here are some references that I found after looking into this:

• http://csunplugged.org/open-source-edition-ms-word (fun, offline approach to teaching basics, CC licensed)
• http://www.code.org/teach (great intro videos to kick off the idea, 1m, 5m, and 9m versions)
• http://www.code.org/learn (links to sites to learn coding)
• http://www.csprinciples.org/home/about-the-project (pilot project to reinvigorate AP CS course)

Here is one of those videos linked to above:

I need to volunteer to teach at my kids' school, and in a community education setting.

Awesome CLI gem: Commandable

Ever wanted to drive your ruby code with a CLI?  Well, the commandable gem makes it easy.

At first, I struggled with OptionParser in the stdlib, but I had to duplicate my option handling across CLI code and method invocation.  Then OptionParser didn't handle required parameters at all, requiring me to do required parameters in a non-DRY way.   Also, OptionParse doesn't do sub-commands, git-style, which I needed for the new git plugin I'm working on.

When I saw the pickled_optparse gem (from Mike Bethany), it appeared to solve the missing 'required parameter' feature.  And the subcommand gem seemed to allow for subcommands with their own options, git-style.  However, the duplication across option definition and method invocation was still present.  And even subcommand required you to hook up the command to the class/method that needed to be called.

Finally I found that the author of pickled_optparse had totally rethought CLI in ruby with his commandable gem.  And I really like what he did.

Now there is no real additional logic AT ALL in order to hook your ruby class up to the command line.  Now my option handling code will be in the place it belongs, in the code that is doing the command that I invoked from the command line.

The main awesomeness here is that commandable pushes the CLI code into the appropriate place in your ruby class -- it makes it hard to scatter CLI code all over the place.

My only beef with the gem are the following points:

• it comes with colors enabled by default (unusual for a CLI)
• it clears the screen by default when displaying help (when colors are enabled, very unusual and annoying for a CLI)
• it appears to be hard to do global options like: "cmd --global subcommand [args]"
• there is no support for parsing command CLI-style options into ruby hashes, but this is a minor nit, because that's what OptionParser is for, and it's easy to use inside the method

I fixed a problem where the screen was cleared even if you disabled colors, and I've pushed it up to my fork.

Here is my hello world that shows commandable in action:

And here's the output for a few representative cases:

Unicode Support Better Now in Ruby

Unicode support in Ruby has historically been one of Ruby's major problems for me.

With the advent of Ruby 1.9 of course, Unicode support started being added to the language.  However, it's not as straightforward as Java, which supported some version of Unicode from the beginning.

Even though it's been a rough decade, things are finally looking up.  In fact, I actually like the way things got factored after all of the mess.

The thing I like is that the minimal amount of support is included in the standard library, and it's easy to compose things in non-standard ways for weird scenarios or data in improperly encoded formats.

The core library has support for strings of codepoints and bytes and a flexible set of encoding facilities.

In addition, there are two libraries of interest:

• unicode_utils - includes implementations of word-break functionality, grapheme boundaries, etc.
• jcode (if you're stuck on ruby 1.8.x)

This series of posts gives you a full understanding of the topic.  Highly recommended!

This post gives a high-level view of where things were at around 2006, much of which is valuable background.

This post has a good summary of unicode-related resources, as does this stackoverflow question.

Semantic Versioning

After having skimmed the semantic versioning proposal/spec, I really like it, and I'm going back for a deep read.

The most notable violator of this that has bit me in the past has been the jersey framework, and maybe earlier versions of commons-collections.

Using dot from graphviz

The whole graphviz package is an amazingly useful piece of software.  Especially the dot program.

I've watched other people hand-craft Visio documents that go out of date really fast.

I've also watched developers try to use design tools for high-level sketch kind of diagrams and get bogged down with superfluous code-sync features.

When I want to illustrate a point, just plain old boxes and arrows work wonders.  Especially when I can commit the source and be able to tweak it afterwards.

I've found that dot meets and exceeds the my common use goals.  I would guess that 80% of the time, the diagram communicates what I want to say without any tweaking at all.  About 15% of the time, it takes some layout/shape/font/color tweaking to get the message across in a clear & direct manner.  About 5% of the time, I have to output the graph as SVG and load it into Inkscape for further slight tweaking.

Here are some especially helpful links:

• official site
• gallery of examples, also 
• up-to-date language reference, and other documentation
• outdated, but very well-organized Users Guide
• dot-functions.sh

The last piece of software is intended as a helper for invoking dot quickly from the command line as part of a REPL authoring flow.

To use dot-functions.sh, first download it.  The script defines a bash function that can be used from a bash CLI.  Source it into the current shell by doing: source dot-functions.sh, followed by dot somefile.dot.


Here is the source of dot-functions.sh:

Another very helpful article was written on this topic by Diomidis Spinellis.  Now that I think about it, there is another very helpful article about how to get a hand-written sketch into digital form.

Keeping a Beginner's Mind

As an experienced software developer, I care deeply about retaining my ability to remain flexible in my habits and learning style.  That's the only way I got good, and if I ossify in my learning habits, I'll end up the equivalent of a COBOL programmer.  Certainly not the way to live up to the broad possibilities that exist to make the world a better place.

I appreciated the wider perspective that I got from glancing through these slides by Patrick Kua:

The Beginner's Mind

Although I had seen and skimmed Pragmatic Thinking & Learning, this presentation was a gentle and useful introduction to the whole idea of the Dreyfus model of skill aquisition.

Another very useful idea that this presentation expressed was the contrast between "skill-acquiring apprentice" vs. "closed-minded expert".  Patrick said that the "skill-acquiring" attribute can also apply to highly-skilled practitioners.

Among the tips Patrick gave, the following were useful to me:

• You can't be an expert on everything. [so don't even try]
• How can I try this safely?
• How does this fit in my world?
• Remain curious.
• Mix with diverse groups.
• Beware of built-in biases.
• Avoid judging early.

The reason I decided to compare/contrast some of the ideas in this presentation was actually because of my forays into the Pragmatic Thinking & Learning book, which was one of three "Further Resources" offered at the end.

Patrick recommended the following books for further learning:

• Pragmatic Thinking & Learning
• Apprenticeship Patterns
• Changing Software Development

Because of the context in which the Apprenticeship Patterns book appears, I want to read that as a next step for learning how to make a real difference at FamilySearch.

Lazy Developer plus No Slack equals Funding Guilt

I have enough hubris to consider myself a lazy software developer. I also work in an organization where there is not a lot of slack. Overall, I consider myself very blessed to be a part of the software group in which I work. There are many intangible rewards that I receive personally by working on the projects I'm assigned to work on.

Most of what I wanted to say in this article has already been said, so I'll spare you the effort if you're already well-read in this area.

However, the lazy developer / no slack combination often produces an interesting set of emotions within me. In particular, I repeatedly feel a certain ugly kind of frustration.

Here is my shot at defining this certain kind of frustration:

Funding Guilt: n. The feeling of having a sense that you could solve the problem at hand in a better way and more conveniently by inventing a solution to a more general problem, but being in a position where the investment in that more general solution is not specifically funded by the person paying for the specific assignment and might be more costly in the short term.

A lazy developer's answer to this is often to ignore the context and solve the more general problem anyway, with an eye toward paring the problem down to its core and avoiding gold-plating.

I remember working with a very smart developer who told me he kept an open critical bug as cover for getting real work done. If asked what he was spending his time on, he would point to his efforts to solve the critical bug. He would make enough progress on the bug over a period of time that people would continue to leave him alone, because they really wanted the bug fixed as soon as possible. But he devoted a significant amount of his time each day toward his real work. And when that bug got fixed, he'd pick up another serious bug that looked like it could be of use. Maybe this approach could be called "Bug Cover".

Although that approach to avoiding funding guilt is deceptive at its core, it worked for him. I'd prefer a way to avoid funding guilt that was more above board.

And perhaps funding guilt only appears in hyper-responsible people, of which I am one.

For those of us who feel it, acknowledgment of this emotion sometimes helps create a sense of understanding and perspective in ourselves and others, including possibly sponsors of our work. It is important to be able to forgive oneself for funding guilt if solving the more general problem is really the right thing to do. But this is not a resolution to the core question. How can funding guilt be avoided as a matter of course?

Maybe it's a matter of choosing very carefully which projects you work on, or at least aborting unfruitful projects very quickly.

Maybe the proper response to expressed funding guilt is at the core of the art of great software management. Perhaps great managers are those who can create the sense of slack that is needed to allow developers to get their juices flowing and create great stuff.

Even if it does involve a little cover.

P.S. FWIW, many of the ideas in this post come from having read and digested the following:

• Pragmatic Thinking & Learning (personal learning)
• Driving Technical Change (organizational rollout affordances)
• Mythical Man-month (multi-pass thinking)
• using Git on a daily basis (multi-pass thinking)
• Pay to Learn when playing the game of software (optimizing for learning)
• Sooner Not Faster (optimizing for learning)
• Reductionist Agile Manifesto (paring things down to the core)

Core Goals of Version Control

After having a couple of years of experience with Git, I was finally able to clearly articulate the core value that version control brings to software development. This article will be written with a bias toward Git, but I've at least tried to express the core value in abstract terms.

The value is expressible in 4 goals.

Core Goals

1. Capture source artifacts
2. Isolate stable codelines from WIP
3. Enable concurrent WIP
4. Make it easy to leave sensible history behind

WIP = Work In Progress

The continuous delivery crowd says that WIP should be basically zero. I think that there is space for a fairly short development pipeline (3-5d) that can free developers up to capture ideas and then wrangle them into shape in a second or third pass.

Capture Source Artifacts

As a developer, I feel happy when I can push a save button when I've taken a small step -- and then never have think about it again. I also feel happy when I can save, even if I'm based on a slightly out-of-date base. I also feel happy when I don't have to worry about binary vs. text -- as long as its a true source file of a reasonable size.

I feel happiest when I don't have to be distracted by superfluous concerns, and can focus on the task at hand and save the results easily.

Main Point: A good version control system makes it easy to capture source artifacts.

Isolate Stable Codelines From WIP

Software is hard to get right, and once things get stable, the only way to keep them there is to avoid making risky changes. But stability is always balanced against the common desire for enhancement and restructuring.

A typical bug fix is one or two isolated commits that fix a specific problem. Most version control systems more or less support cherry-picking a small change over into another codeline.

I feel happy when I'm able to easily take an isolated series of commits from the development codeline into a stable one, even if the fix turns out to be a little larger than a couple commits.

Main Point: A good version control system makes it easy to cherry-pick changes, and even longer patch sequences, from one codeline to another with easy reconciliation of overlapping edits.

Enable Concurrent WIP

For software developed by a team larger than 2-3 people, it is useful to be able to work in parallel with each other, even on the same feature. Sometimes you can arrange your work so as not to overlap at all, but there are real situations where you want to track someone else's development within the fairly short development pipeline (3-5d) before things have gotten stable.

I feel happy when I don't have to catch people up on what I've been doing, and when they don't have to catch me up on what they've been doing, and we can get to the point easily and quickly. I also feel happy when we can leap-frog each other with ideas and real implementation.

Main Point: A good version control system makes it easy to track other team members' work, and easy to tentatively integrate with that work.

Make It Easy To Leave Sensible History Behind

After stepping away from a piece of software, it is easy to lose context and forget the concerns that shaped the development of that software. It doesn't take long. For me, about 2 weeks is enough for me to start forgetting details and motivations.

Of course, it is important to leave a properly structured artifact behind. Proper naming means a lot; proper factoring is important. Once you comprehend the structure, it is possible to change things without introducing big problems.

But often, full comprehension isn't practical, and a flat 2D view of the artifact is insufficient for reasoning about the thing itself. Often, it is often very useful to be able to get a vector on where the software came from, and interpolate from that vector where it was headed.

The creative process is messy, filled with double steps and backtracking. Don't confuse me with all the noise. Give me a history that, given all the knowledge you had when it got stable, at least looks well-reasoned.

I feel happy when I'm able to take an easy editing pass following capture/review that lets me intentionally craft history as an email to the future about the vector of change that is inherent in a certain patch sequence. I also feel very happy to not have to think about this during the creative, messy, focus-draining capture/review pass.

Main Point: A good version control system includes tools to both: 1) enable the developers to easily leave behind meaningful history, and 2) extract focused history for all or part of a piece of software.

Playing catch-up

The world is changing all the time, which implies the requirement of constant learning to keep current. In other words, you are always behind.

If you accept the fact that you are always going to be behind, in one way or another, the problem then becomes more tractable. How, in a limited amount of time, can you bootstrap yourself into a learning environment where you can catch up enough to get something working?

The core questions are:

1. What to learn? (because of the limited time, you know you have to be selective)
   
2. How to go about it? (because of the limited time, and the constant churn, you have to be a quick learner/applier)
   

Andy Hunt wrote a book about pragmatic learning. Clayton Christensen wrote several books about disruptive innovation. The Wikipedia contributors wrote an article about the term "learning curve". The ideas in these books can be instructive.

I have my own opinion about the matter.

My answers to the core questions are:

1. Look around and get creative about how you can apply about-to-be-stable newer technology to the software problem at hand.
2. Climb the dynamic learning curve by becoming an "early adopter".

Being an "early adopter" is a productive approach to bootstrapping yourself into a rich learning environment. The key to quality learning is keeping it real & experiential. And trying new technology out and trying to apply it to the task at hand is certainly real & experiential.

The part that makes this whole learning equation possible is that the "innovators" actually need the "early adopters" in order to gain traction and stability. In an open world, that means that you can use early adoption as a means by which it is always possible to inject yourself into a rich and productive learning environment.

After I play this game for a while and become skilled at it, I'm guessing there will be a point at which I will want to have a talk with Paul Graham about a startup. Or maybe I will care about being an innovator in my family more than being famous in a technical sphere. Who knows.

Counterpoint

Donald Knuth is the classic example of someone whose life mission specifically excludes playing month-to-month catch-up. Oh, and by the way, that page returns the following HTTP header (in 2010):

Last-Modified: Fri, 23 Sep 2005 04:39:22 GMT

Even the innovation-encouraging Paul Graham wrote an article about addiction that cautions about blanket acceptance of technical improvements.

How to rewrite a complex test

An Integrated Test is suboptimal for asserting basic functionality and basic sub-component collaboration behavior.

So if you have a massive Integrated Test, how is it possible to rewrite that test into some number of the following kinds of tests?

• focused unit test
• focused collaboration test (how one class collaborates with another)
  
• systems-level integration test (load balancer behavior, queuing system behavior)
  

I think it comes down to the following activities:

• enumerate the different permutations of state
• enumerate the different permutations of flow
• for each permutation of state: create one focused unit test
• for each permutation of flow: decide whether 1) the permutation of flow devolves into a sub-component collaboration test, or 2) into a systems-level integration test
  
• create the required focused collaboration tests
  
• create any required systems-level integration tests (usually very rare)

There is an interesting smell that comes from the activity of creating tests. There may be an existing test that is responsible for asserting the focused behavior, but it isn't in the right place, so it is hard to find out whether it exists. In this case, the act of "create focused test" implies the act of "move focused test into its rightful home" (so others, including yourself, can find it later).

In a meeting in Feb. 2010, I wrote the following about problems I've experienced with a test suite at work:

• Inability to run a test context-free => high re-run costs and downstream delays
• Too much custom test infrastructure => high maintenance costs
• Risk of centralized integration => waiting on central integration before shipping

The approach I suggested was to find the top 20% costliest tests, and focus on those.

I suggested measuring "costliest tests" using a combination of the following criteria:

• How many superfluous assertions in this test?
• How many superfluous historical failures has this test generated in the last 6 months?
• How long does it take to run this test?
• How many "permutations of state" is this test trying to cover?
• How many "permutations of flow" is this test trying to cover?
• How far away from the code is this test?
• Is there a place closer to the code where those "permutations of state and flow" can be adequately tested?
• Are there ways to ensure all the "permutations of flow" can be covered without having to mix the test with trying to test all the "permutations of state" at the same time?

The whole idea is to simulate expensive parts of our tests in a way that still gives us the confidence that the test is valid and covers the desired integration case.

Where and how to test what

J.B. Rainsberger wrote in the fall of 2009 about why he thinks typical programmer use of "Integrated Tests" leads to a vicious cycle of denial and suboptimal behavior.

His overall ideas were summarized well by Gabino Roche, Jr.

And there are good uses of integration tests.

I was about to hit the delete button on this post, because I thought all I had to say had already been said. But there was still something to say: How do I personally work in a way that avoids the Vortex of Doom?

The key idea that has me personally is to pause, and ask the following question:

What is the responsibility of this test?

and then to consider the answer to the related question:

What is the responsibility of the class being tested?

Of course, those are fairly basic OO questions. However, when you're writing tests along with the code, there is a situation that is easy to get stuck in: having so many things in mind at once, that you get confused about the purpose of the test, and even the software you are working to create.

There are at least three things that tend to compete for mind space:

1. What observable behavior do you want out of your software?
2. How do you think you might implement that?
3. How does what you are building interacts with the rest of your system?

And, when #2 gets the top spot in my mind, I find myself forgetting about #3, and resorting to copy/paste for #1 (from other similar tests). However, when I focus on #1 and, by extension, #3, I find myself getting new ideas about how to actually implement the new behavior.

In addition, I find that these new ideas are reorienting in nature. The new stuff I'm working on ends up either modifying an existing concept in a novel way, or introducing a new concept that collaborates with the existing system in a certain way. Then the test I thought I was going to write ends up being a straightforward unit test on a new class, or new methods of an existing class. And a couple of collaboration tests that make sure the new behavior actually gets used.

In the end, there are a few questions that need to get answered:

1. Does the new behavior work? (unit tests will tell you this, 80% of tests)
2. Is the new behavior hooked up? (collaboration tests will tell you this, 15% of tests)
3. Does the whole thing hold together? (automated deploy to a production-style test site with system-level smoke tests will tell you this, 5% of tests)

And the system-level smoke tests are only responsible for making sure that certain itemized flows work, not all permutations of state that go through those flows.

Hopefully this is a useful addition to the already-posted conversations started in 2009.

Reality Quotient

There is a fairly subjective measure I've only recently been able to give a name to:

Reality Quotient = ability to keep context while working toward a specific goal

This has to do with how deep you allow your stack to be, which if you allow it to get too deep, this affects your net throughput on Cockburn's "unvalidated decisions" or Demarco's "Total Useful Mental Discriminations (TUMD)". If the stack is too deep, you end up wasting a lot of time making useless decisions about things that have ceased having anything to do with the REAL task at hand.

The tendency to accept decisions that pin you into a corner is closely related to lowering the Reality Quotient. There is a whole book about the attitude of Getting Real, and I equate that attitude with a high Reality Quotient.

I did a search to see whether anyone else had published a writeup under the "Reality Quotient" heading. Although I found a lot of stuff on the web, none of it really matched what I wanted to say. The topic of this post is NOT:

• whether you are an honest person in general, or in a specific social situation
• a manager trying to find out what's REALLY going on in his/her org
• whether the actions of time travelers will have any effect
• some kind of measure of whether a politician is telling the truth or producing spin
• some measure of how "real" a reality show really is

The measurement I wanted to talk about is how capable you are of focusing on the problem you set out to solve until 1) it is truly solved and published to the world, or until 2) you have redefined the problem into another solvable one and published that transformation to the world.

In short, a high Reality Quotient requires a short stack, with tight feedback loops, focused on publishing real stuff to real people.

IntelliJ patches don't seem to install

When I started IntelliJ (9.0.1, community edition) today, it popped a dialog up with an option to download a "patch" to install 9.0.2 release.

I've clicked "Download Patch" before, but nothing ever happened, and I ended up resorting to a full install.

But today, I came across a comment that described how to apply the patch:

http://blogs.jetbrains.com/idea/2010/01/intellij-idea-901-released/#comment-128607

And it worked great for me.

Others on my team said their patches installed just fine. To be clear, I'm using the Community Edition, and it may only be an issue with that edition.

BYU Conference on Computerized Family History & Genealogy

I attended BYU's Conference on Computerized Family History & Genealogy Monday and Tuesday this week.

It was really informative and disorienting in a good way. I've been working on New FamilySearch for a long while and not realized fully what kind of a target audience I've been developing for. When I was one of the only 30-somethings in the room, and we were talking about a more-than-a-decade technology generation gap, it finally sunk in that the world was different than I had thought.

The main topics I came away with were:

• The technology generation gap is NOT ok, and needs to be bridged (by both the younger and the older generations).
• Among serious genealogists, there is a very real drive to publish, similar to the scientific community's reputation-based drive, but motivated also by a desire to preserve research that would otherwise be obliterated by death or lack of interest on the part of direct descendants.
• There is also a gaping technological hole in the genealogical community: Lack of Internet-style content-driven collaboration on genealogical research. Even serious genealogists seem to be content with this state of things.
• The Next Generation (TNG) looks really nice. I want to use it for the Sumsion family tree on sumsion.org/genealogy.
  

The idea that I can make a difference here is VERY motivating.

An entire sprint without an SVN branch

A typical development flow has been:

• request an SVN task branch (wait for CM to help)
  
• do stuff (1-3 weeks)
  
• run all the regression tests (not on the latest stable)
  
• hope that nobody did anything that conflicted with us
  
• merge to the pending release (and pray we resolve any conflicts correctly)
  

Now we do the following:

• set up a team git repo (able to do this without CM's help)
  
• clone off a repo for each team member
• set up a daily cron job to get the latest stable from the pending release
• set up a daily hudson build to merge that in and push to team if it passes unit tests
• pull from team as we go (fast!)
  
• run all the regression tests (on the latest stable)
  
• use git to merge on top of the pending release and commit back to SVN (with minimal conflict windows)
  

Well, we finally got through a whole sprint without an SVN task branch. Maybe that seems like a small victory, but it saved us a bunch of late-breaking integration risk.