PTSD After Large Release

I don't like releasing large chunks of software at once, but there are situations where it's easier to swap out a large piece of interdependent pieces than it is to replace them piecemeal.

Except you have to make it through the release.

I worked on replacing the Family Tree database on familysearch.org with a new one.  Old was Oracle, new was Cassandra.

It started at 12:30am and extended, with various emergencies that were handled more or less gracefully by amazing people, until 6:21am.  And then it was done, and we had to start keeping it up.

After a few surprises that none of our simulations exposed, I have enough rest again and can function more or less normally.

However, the anxiety I felt during the release and the uncertainty I felt in the days just afterwards all added up to a feeling I don't remember feeling before.

Then I realized that the flashbacks and the irrational worry about keeping things working - probably some mild form of post-traumatic stress.

Certainly only a taste of what others go through who were in danger of losing their life and barely survived.  Not trying to imply that my experience is anywhere near that sort of thing.

I'm just trying to process my emotions and am hopeful this helps someone know they're not alone.

Heartbleed Reaction Part 2

A particularly relavent statement from http://heartbleed.org (server side):

"Fortunately many large consumer sites are saved by their conservative choice of SSL/TLS termination equipment and software. Ironically smaller and more progressive services or those who have upgraded to latest and best encryption will be affected most."

There doesn't appear to be any up-to-the-minute current registry that I could find of sites that are affected on the server.  The scan posted on github is fairly out of date at this point, and from what I can tell only takes the homepage into consideration, not sites that only forward to https for things like login / checkout.

Here is the best one-off checker I could find (server side):
- https://www.ssllabs.com/ssltest/

Also, it may not be necessary to update Chrome/Firefox, based on the following language on the security stackexchange site:
- http://security.stackexchange.com/questions/55119/does-the-heartbleed-vulnerability-affect-clients-as-severely

"Chrome (all platforms except Android): Probably unaffected (uses NSS)"
"Chrome on Android: 4.1.1 may be affected (uses OpenSSL). Source. 4.1.2 should be unaffected, as it is compiled with heartbeats disabled."
"Mozilla products (e.g. Firefox, Thunderbird, SeaMonkey, Fennec): Probably unaffected, all use NSS"

The potential vulnerability of clients is discussed here:
- http://security.stackexchange.com/questions/55119/does-the-heartbleed-vulnerability-affect-clients-as-severely
- https://www.openssl.org/news/secadv_20140407.txt (language: "client or server")
- http://heartbleed.com

"Furthermore you might have client side software on your computer that could expose the data from your computer if you connect to compromised services."

My guess is that curl going to an https site would be affected, or other programs that use OpenSSL.  Maybe a chat client or if programs are downloading their own "auto-updates" over SSL.  Those are the only kinds of things that come to mind right now.

Reacting to Heartbleed

It's 2:37am and I can't sleep.  It feels like the internet fell down around my ears.

What I am doing:

1. Got educated at http://heartbleed.com
2. Updated Chrome to the 34.x version manually (promoted to stable yesterday)
• look at https://support.google.com/chrome/answer/95414 and search for "Check for update manually"
3. Checked for vulnerability in sites I use
• dnsimple.com (patched as of 8 Apr 14:22 GMT)
• github.com (patched and sessions invalidated somewhere between 8 Apr 18:34 GMT and 9 Apr 01:08 GMT, based on atom feed post timestamps)
• familysearch.org (not vulnerable as of 8 Apr around 16:00 GMT)
• amazon.com (not vulnerable as of 8 Apr around 16:00 GMT)
4. Completely clearing cookies and cache on ALL my computers, family & work, including phones
5. Installing LastPass and resetting ALL my passwords as I become confident that each site is patched
• I am assuming that all my user/passwords are either already known at this point, or can be discovered by anyone who recorded SSL traffic in the past 2 years
6. Wondering what will happen because of this

UPDATE: Chrome update seems to be not strictly necessary as stated here.  But I'm upgrading anyway, because the Chrome stable release on 8 Apr. 2014 has a lot of other security fixes in it.

UPDATE: More details that I've learned are here in a follow-up post.

Paying Down Learning Risk

I've heard: "Solve the hardest problem first."  As a rule of thumb, that works great to reduce risk early on a software project.  But I found myself saying something different to a co-worker recently:

Sometimes I start with the hardest problem, but sometimes I like to start with a really easy problem.  Why do I do that?

Why would it be a good idea to start with the easiest problem?  What kind of risk are you trying to pay down in a given situation?

Here are some reasons that would justify breaking the "hardest problem first" rule:

• If you need to gain experience in a new domain, starting with something easy can help you get experience before tackling the harder problems.
• If the world has changed out from underneath an existing system in an unpredictable way, starting with changing something easy or predictable can help you observe the source of the chaos.
• If you are sharing work, handing the easy work items out to others based on their learning goals can help them learn better.
• If tackling a hard problem will take a very long time, and others are waiting for you, then picking an easier part of the problem can help ease integration while still letting you engage on the hard problem.

The kind of risk you want to pay down first is important.  Here are the kinds of risk that would be payed down by the above behaviors:

• risk of getting lost while learning
• risk of being unable to bring order to a chaotic system
• risk of assigning impossible tasks to someone who just wants to ramp up
• risk of high integration costs because of trying to change too much at once

Most of the time, the risk caused by the uncertainty inherent in solving a hard problem is the most important risk to pay down first.  But sometimes, there are other factors at play, and other subtle variables that need to be managed to achieve a successful group outcome.

Thank you to Michael Nelson for his instructive collaboration on this topic.

Innovation Risks

Instead of responding to "innovation" as a buzzword, I want to make sure that I always just think about innovation in terms of social changes, large or small.

As software people, we probably tend to be much more change-tolerant on average than many people in the non-software population -- I believe that's one reason why we gravitate toward the soft-ware part of things.

But there are particular kinds of innovation risk, I think:
1) effort investment risk
2) future opportunity risk
3) legacy replacement risk
4) replacement rate risk

Business people often talk about the expected ROI of a particular proposal.  That is what I'd categorize under the category of #1.  If I expect a return, it'd better be worth the effort I put into it.  This is standard stuff for software developers.  We do estimates to establish expected ROI, we do the work and see the results.

What it comes down to the categories of risk #2 and #3, I thing there is a wide variation in risk tolerance in software developers.  Often this is because of variation in our perception of value, or varied backgrounds, and even in similar backgrounds, variation in our recall of the hard lessons of experience.  Some of the most experienced among us look farther ahead, and therefore avoid certain risks because they look similar to times when we got bitten in the past.

In addition, it seems that #4 is different than #3, because even though some people may be willing to absorb the cost of a significant change once or twice, they may not be willing to continue to absorb changes of the same magnitude on the same frequency.

I think that common responses to these different kinds of risk are as follows:
1) proper planning (mitigates effort investment risk)
2) proper deliberation (mitigates future opportunity risk by measuring which opportunity to chase)
3) caution, loss aversion (enhances legacy replacement risk by clouding judgement)
4) apathy, rejection (enhances replacement rate risk by inhibiting trust and hurting relationships)

The difference is that most humans have a disproportionate amount of loss aversion for things they perceive that they own.  That's what distinguishes #2 from #3.

Where we fall into a trap is if we over-deliberate or let loss aversion dictate our learning environment, and if the world changed in a way that causes us to mis-predict failure based on our prior experience.  Sometimes it's more valuable to walk away from something of value even when we don't know what we're looking for in its replacement, because we have a distinct feeling that non-linear improvement is needed, or because we trust someone else's concept of where we can eventually end up.  Sometimes, something we failed at earlier is now possible, but only possible in a way are ignorant of, and therefore only possible in a way we cannot predict.

Ignorant and highly-motivated young blood (or adventurous veterans) in our field is what keeps us continue taking inordinate risks and learning from the experiences that come from them.

Does experience and capability make us better innovators?  Does our level of context make us more capable of effecting positive change?  Not necessarily, I think.

Maybe, to a point.  Once we've achieved a certain level of experience, I believe our efforts have higher overall effectiveness only if we are capable of avoiding the expert trap, and are able to forget & re-learn appropriate parts of our experience in the current context.

Some material that is relevant to this topic:
- http://matt.might.net/articles/programmers-resolutions/
- http://blog.8thlight.com/uncle-bob/2013/03/05/TheStartUpTrap.html (warn: unfortunate language)
- http://www.lessonsoffailure.com/developers/habits-kill-career/ (warn: contains a crude analogy)
- http://tcagley.wordpress.com/tag/zen/
- http://pragprog.com/book/trevan/driving-technical-change

Just remember the following pragmatic rallying cry:

"If it's not broke, let's not invite the UN to fix it." (heard on Linux Radar podcast)

Intrusion Detection through Stackable Filesystems

I've always wondered what exploit might be running on my system, and never had any time to devise/install a detection system that would have the right balance of useful detection (maximize) and performance impact (minimize).

When I stumbled upon unionfs a couple weeks ago, I thought that was an interesting idea from a change-logging perspective.  It's sometimes useful to be able to keep a filesystem-based diff of what a certain operation does to a system, and then bake it onto the system when I know it did what I wanted to.  The takeaway for me was that unionfs's performance profile had the opportunity to be so low because it was so thin and so baked into the kernel.

This compounded with my recent discovery and fiddling with fusefs (in user-land) and I wondered about what kind of useful logic I could put underneath a filesystem.  The GlusterFS feature set, the recent LessFS GC stuff, the bup-fuse stuff, and the S3FS stuff is all just *really* cool.  I ended up gazing longingly at the big backlog of fuse filsystem suggestions on their wiki, and wandered back into the unionfs space.

So, when I saw the I3FS paper (linked from here) about the modular application of this technique to intrusion detection, this really triggered the Useful *AND* Performant neurons and I got really excited.

Unfortunately, on first glance, the stackable filesystems stuff seemed pretty cryptic to set up in a lightweight, just-works kind of way (think custom mount command-lines complete with arcane stacking options).

It would be soooo awesome to have an easy-to-compose ruby DSL for doing some kind of rack-like filesystem mashup with a kernel-level unionfs layer underneath a user-land fusefs layer, but all expressed in the same DSL.

This would be an awesome tool to put on top of the Arch-derived clone I want to put up for people at work.  There are folks who care about living more on the edge of linux stuff, but that don't care to install from scratch, and also might not care that much about not having a full Gnome stack if things just work.  And if I could give them the same tuned IDS-on-the-desktop solution (or upgrade their developer stack by letting them pull a filesystem delta over), that would be really cool.

The cheap development observation [PDF, linked from] because of modularity is one of attributes valued highly by the Ruby community as well.  It is one the key things that makes Ruby as a community awesome.

These kinds of ideas really matter, and making them so cheap and stable that you don't have to think about them really matters even more.

NOTE: The I3FS paper is really pretty old (2004), and the whole unionfs stack is older than that.  The fuse stack came into the kernel a long time ago too (2005).  So while this is new to me, it's been around for quite a long time.  I'm playing catch-up.

Attention to Error vs Attention to Detail

After having read Talent Code, a friend pointed me to the following article:

Why Do Some People Learn Faster?

Although the author of that article had gotten some lackluster reviews on some of his book-length work on Amazon, this article brought a distinct idea to the forefront for me:

Attention to Error

I've always thought that attention to detail was an important tendency in myself that made me a good learner.  But I realized that there is a distinction between paying attention to detail and paying attention to error.

The constant reconciliation between what you believe to be correct and what turns out to be correct is the fuel for my personal learning.

Some people might say this is what the scientific method is all about.  I'm less methodical than a scientist would be -- I find it nevertheless useful to constantly put myself in situations where I can learn informally from comparing what I expect to what actually happens.

Lazy Developer plus No Slack equals Funding Guilt

I have enough hubris to consider myself a lazy software developer. I also work in an organization where there is not a lot of slack. Overall, I consider myself very blessed to be a part of the software group in which I work. There are many intangible rewards that I receive personally by working on the projects I'm assigned to work on.

Most of what I wanted to say in this article has already been said, so I'll spare you the effort if you're already well-read in this area.

However, the lazy developer / no slack combination often produces an interesting set of emotions within me. In particular, I repeatedly feel a certain ugly kind of frustration.

Here is my shot at defining this certain kind of frustration:

Funding Guilt: n. The feeling of having a sense that you could solve the problem at hand in a better way and more conveniently by inventing a solution to a more general problem, but being in a position where the investment in that more general solution is not specifically funded by the person paying for the specific assignment and might be more costly in the short term.

A lazy developer's answer to this is often to ignore the context and solve the more general problem anyway, with an eye toward paring the problem down to its core and avoiding gold-plating.

I remember working with a very smart developer who told me he kept an open critical bug as cover for getting real work done. If asked what he was spending his time on, he would point to his efforts to solve the critical bug. He would make enough progress on the bug over a period of time that people would continue to leave him alone, because they really wanted the bug fixed as soon as possible. But he devoted a significant amount of his time each day toward his real work. And when that bug got fixed, he'd pick up another serious bug that looked like it could be of use. Maybe this approach could be called "Bug Cover".

Although that approach to avoiding funding guilt is deceptive at its core, it worked for him. I'd prefer a way to avoid funding guilt that was more above board.

And perhaps funding guilt only appears in hyper-responsible people, of which I am one.

For those of us who feel it, acknowledgment of this emotion sometimes helps create a sense of understanding and perspective in ourselves and others, including possibly sponsors of our work. It is important to be able to forgive oneself for funding guilt if solving the more general problem is really the right thing to do. But this is not a resolution to the core question. How can funding guilt be avoided as a matter of course?

Maybe it's a matter of choosing very carefully which projects you work on, or at least aborting unfruitful projects very quickly.

Maybe the proper response to expressed funding guilt is at the core of the art of great software management. Perhaps great managers are those who can create the sense of slack that is needed to allow developers to get their juices flowing and create great stuff.

Even if it does involve a little cover.

P.S. FWIW, many of the ideas in this post come from having read and digested the following:

• Pragmatic Thinking & Learning (personal learning)
• Driving Technical Change (organizational rollout affordances)
• Mythical Man-month (multi-pass thinking)
• using Git on a daily basis (multi-pass thinking)
• Pay to Learn when playing the game of software (optimizing for learning)
• Sooner Not Faster (optimizing for learning)
• Reductionist Agile Manifesto (paring things down to the core)