PTSD After Large Release

I don't like releasing large chunks of software at once, but there are situations where it's easier to swap out a large piece of interdependent pieces than it is to replace them piecemeal.

Except you have to make it through the release.

I worked on replacing the Family Tree database on familysearch.org with a new one.  Old was Oracle, new was Cassandra.

It started at 12:30am and extended, with various emergencies that were handled more or less gracefully by amazing people, until 6:21am.  And then it was done, and we had to start keeping it up.

After a few surprises that none of our simulations exposed, I have enough rest again and can function more or less normally.

However, the anxiety I felt during the release and the uncertainty I felt in the days just afterwards all added up to a feeling I don't remember feeling before.

Then I realized that the flashbacks and the irrational worry about keeping things working - probably some mild form of post-traumatic stress.

Certainly only a taste of what others go through who were in danger of losing their life and barely survived.  Not trying to imply that my experience is anywhere near that sort of thing.

I'm just trying to process my emotions and am hopeful this helps someone know they're not alone.

Dystopia - Fallacies Conquered

Dystopia - from NetflixOSS's Adrian Cockcroft:

We have spent years striving to build perfect apps running on perfect kernels on perfect CPUs connected by perfect networks, but this utopia hasn't really arrived.

Instead we live in a dystopian world of buggy apps changing several times a day running on JVMs running on an old version of Linux running on Xen running on something I can't see, that only exists for a few hours, connected by a network of unknown topology and operated by many layers of automation.

Reminds me of Fallacies of Distributed Computing, but from the perspective of people who have actually vanquished the beast with weapons that are usable by others.

Innovation Risks

Instead of responding to "innovation" as a buzzword, I want to make sure that I always just think about innovation in terms of social changes, large or small.

As software people, we probably tend to be much more change-tolerant on average than many people in the non-software population -- I believe that's one reason why we gravitate toward the soft-ware part of things.

But there are particular kinds of innovation risk, I think:
1) effort investment risk
2) future opportunity risk
3) legacy replacement risk
4) replacement rate risk

Business people often talk about the expected ROI of a particular proposal.  That is what I'd categorize under the category of #1.  If I expect a return, it'd better be worth the effort I put into it.  This is standard stuff for software developers.  We do estimates to establish expected ROI, we do the work and see the results.

What it comes down to the categories of risk #2 and #3, I thing there is a wide variation in risk tolerance in software developers.  Often this is because of variation in our perception of value, or varied backgrounds, and even in similar backgrounds, variation in our recall of the hard lessons of experience.  Some of the most experienced among us look farther ahead, and therefore avoid certain risks because they look similar to times when we got bitten in the past.

In addition, it seems that #4 is different than #3, because even though some people may be willing to absorb the cost of a significant change once or twice, they may not be willing to continue to absorb changes of the same magnitude on the same frequency.

I think that common responses to these different kinds of risk are as follows:
1) proper planning (mitigates effort investment risk)
2) proper deliberation (mitigates future opportunity risk by measuring which opportunity to chase)
3) caution, loss aversion (enhances legacy replacement risk by clouding judgement)
4) apathy, rejection (enhances replacement rate risk by inhibiting trust and hurting relationships)

The difference is that most humans have a disproportionate amount of loss aversion for things they perceive that they own.  That's what distinguishes #2 from #3.

Where we fall into a trap is if we over-deliberate or let loss aversion dictate our learning environment, and if the world changed in a way that causes us to mis-predict failure based on our prior experience.  Sometimes it's more valuable to walk away from something of value even when we don't know what we're looking for in its replacement, because we have a distinct feeling that non-linear improvement is needed, or because we trust someone else's concept of where we can eventually end up.  Sometimes, something we failed at earlier is now possible, but only possible in a way are ignorant of, and therefore only possible in a way we cannot predict.

Ignorant and highly-motivated young blood (or adventurous veterans) in our field is what keeps us continue taking inordinate risks and learning from the experiences that come from them.

Does experience and capability make us better innovators?  Does our level of context make us more capable of effecting positive change?  Not necessarily, I think.

Maybe, to a point.  Once we've achieved a certain level of experience, I believe our efforts have higher overall effectiveness only if we are capable of avoiding the expert trap, and are able to forget & re-learn appropriate parts of our experience in the current context.

Some material that is relevant to this topic:
- http://matt.might.net/articles/programmers-resolutions/
- http://blog.8thlight.com/uncle-bob/2013/03/05/TheStartUpTrap.html (warn: unfortunate language)
- http://www.lessonsoffailure.com/developers/habits-kill-career/ (warn: contains a crude analogy)
- http://tcagley.wordpress.com/tag/zen/
- http://pragprog.com/book/trevan/driving-technical-change

Just remember the following pragmatic rallying cry:

"If it's not broke, let's not invite the UN to fix it." (heard on Linux Radar podcast)

Developing Organization Change Skill

When I was trying to describe what it takes to do a good rollout of something new across a larger software development organization, I came up with a chant that made a lot of sense:

Here's the old

Here's the new

Here's the difference

Here's what you can do

Yesterday, I was faced with frustration that came from feeling incapable of doing the rollout tasks that were my lot. And I wanted other people to be capable of rolling new stuff across the development organization, too.

I asked myself:

How am I going to get other people to be capable of rolling new stuff out?

That is when the idea came.

So I think that a successful rollout formula is:

1. presenting all 4 things in sequence, and
2. making it easy for people who are affected by the rollout to take the next step

Published with Blogger-droid v1.6.5