Intractable Problems

Getting into the state of mental flow leads to enjoyment for me.  Whether it's playing a lively game of Monopoly with my kids, or playing/coaching an engaging soccer match, or developing the next feature for FamilySearch -- all of the above activities are more enjoyable when I spend a lot of time in the state of mental flow.

There's a book about this, and I still would like to read it at some point.  But today's post is not primarily about flow.

There are times when I know that a state of mental flow is possible, and desirable, but I can't seem to enter that state.  Maybe I'm meaning to write in my journal but the thing I want to write about is an unresolved problem of some importance to me, and I go into immediate problem solving instead of writing about it.  Maybe I'm trying to get going on a project outside but I'm exhausted from not getting enough sleep the night before.  Maybe I'm attempting to learn something new at work and there is too much new all at once.

Today I realized that when I have problems entering the state of mental flow, it's because I'm stuck trying to solve intractable problems.  See https://en.wikipedia.org/wiki/Computational_complexity_theory

Here are a few "intractable" problems I've faced recently:

• Which task would be most helpful for me to work on first this morning?
• What is important to deal with after a trip, and what can be dropped?
• Who can I get to help set up flags on Memorial Day?
• How can I get uninterrupted time to spend with my wife?
• What can I do to increase the likelihood of my son wanting to work with me outside?

The truth is the above problems are only "intractable" when attempted in a certain mental state that treats these decision problems as personal tasks.

Hint: It takes social skill & action to answer all of the above questions effectively.  And while the social skill can be learned, if you approach the above problems by thinking that they are personal tasks to be completed, they will by definition be intractable.

A useful word is "stall".  Some definitions, glosses, examples:

- https://www.google.com/search?q=define%3Astall

- http://wordnetweb.princeton.edu/perl/webwn?s=stall

- https://en.wikipedia.org/wiki/Stall_(fluid_mechanics)

So when you feel yourself get into a stall, think what intractable problem you are trying to solve.  And if it's a problem that needs social skills to solve, reach out for help.  Figure out what skills you need to develop and improve your abilities.

Worry is a Signal, Not an Activity

"You look down today, what's going on?" my wife says when I get home from work.  I answer, "I don't know, I was just worrying about this project at work." What's wrong with this picture?

The mental error is that I was treating worry like an activity instead of treating it like a signal.  It's all about the self talk.  When there is some outstanding issue that needs attention, it is easy to jump straight to thinking about the issue, even though you can't really do anything about it at the moment.  The reality is that if I'm going to resolve the outstanding issue, I need my computer open, I need to talk with a team member to figure things out.  I need to write some code or run a query to see where things stand.

But if I attempt to sort things out mentally while I don't have everything I need to make progress on an issue, it's easy to spin my wheels and fall helplessly into a non-productive mental loop.

On the other hand, if the "outstanding issue" thought comes into my mind, and I call it worry (which it is), and instead of holding onto that thought, if I treat it like a signal, like an alarm bell, like a red light, then that frees me up to act on the signal.  Instead of treating the "outstanding issue" thought as an activity waiting to be engaged in, if I treat it as an self-alert, then I can move to deal with it at a later, more appropriate time.

The question becomes: "What action can I take right now to make sure that the outstanding issue is dealt with at the appropriate time and place?"  Maybe make a reminder on an index card and put it in my work pants pocket.  Maybe make a reminder on my phone.  Maybe send myself a memory jogger in email.  Maybe write a card and stick it in the Trello / Getting Things Done inbox.  It just needs to be something that I am confident will get my attention and lead me down the right mental path at the time when I know I will have resources to deal with the issue.

Any time spent on worry as an activity (beyond dealing with the reminder for the future time/place) I now believe to be worse than a total waste.  Not just worthless time spent, but also a drag on the rest of my life.  Any unnecessary, anxiety-provoking activity drags me down, makes me less capable of living my life in a worthwhile, enjoyable way.

Why did I not see this earlier in my life?  What could I have done to have learned this earlier in my adolescent / adult experience?

Map-reduce-friendly graph traversal

You may be well acquainted with this, but I thought I'd pass it along.

    https://giraph.apache.org/intro.html

It does iterative graph processing.  The cool thing is that it lets you break down your graph traversal into substeps that are distributed.

Here is some sample code:

  public void compute(Iterable<DoubleWritable> messages) {
    double minDist = Double.MAX_VALUE;
    for (DoubleWritable message : messages) {
      minDist = Math.min(minDist, message.get());
    }
    if (minDist < getValue().get()) {
      setValue(new DoubleWritable(minDist));
      for (Edge edge : getEdges()) {
        double distance = minDist + edge.getValue().get();
        sendMessage(edge.getTargetVertexId(), new DoubleWritable(distance));
      }
    }
    voteToHalt();
  }

Anyway, it looks like a very interesting project.

If I can do map reduce over change history, then it would then be possible to do contribution graph processing -- deep inspection of the graph of users and records they edit, possibly overlaid with the graph of users and which records they are connected to by ancestry.

For example, given a defined group of users, it would be interesting to see how their edits overlap with each other.  Another example would be to determine which top 1000 editing users made the most "distant" edits, based on some kind of ancestral distance measure.

Living in the Future

What does it feel like to hoist yourself into the future, and start living in the future again after having drifted for a while?

I'm having to catch up on cloud deployment as part of a new team I'm on.  Wow, there is a lot of change in the last 5 years.  I feel swamped.

I remember I felt swamped like this a while ago.  And reading about taking crazy risks reminded me of the feeling, and makes me wonder what the risks are of introducing significant change into the group I'm working in.

I guess I could get fired for being too inconsiderate of people who don't like change or the associated risks.

I've also had a long-running idea about making genealogy data editable in a distributed version control way.  Although my ideas are undeveloped, due to my current lack of ability to focus, I've been working on how I could make the idea more viable.

So now that I'm feeling the need to get on the early adopter curve again, I saw How to Get Startup Ideas from Paul Graham, and realized that this article was about both things:

• living in the future
• developing new ideas

From Paul's article:

It takes time to come across situations where you notice something missing. And often these gaps won't seem to be ideas for companies, just things that would be interesting to build. Which is why it's good to have the time and the inclination to build things just because they're interesting.

Live in the future and build what seems interesting. Strange as it sounds, that's the real recipe.

I guess it's the process of shedding the natural loss aversion that I'm not used to, and accepting and realizing the innovation risks that may come.

Missed Tweets Through News.me

I've never really been a twitter nut.  There are already enough sources of distraction for me, I don't need to add another source.  It just seemed really noisy.

However, people have been saying and linking to important things on twitter for a while now.  I'm just not discerning enough and don't have enough time to filter the firehose in a useful way.

The most helpful summary of interesting news I skim regularly is the weekly LinkedIn emails.  In fact, I think it may have been through a link chain from one of those emails that I stumbled on unionfs, as reported in the last post.  If I could have the same thing for twitter, but personalized to the people I follow, that would be really helpful!

Ironically, as part of the effort to make this blog more stuble-upon-able, I was looking for a way to auto-post on twitter and facebook whenever I post on this blog, and I found twitterfeed.com, which does all of that and more.  And it's easy to set up.

After setting that up this morning, I wondered: "What is the company behind twitterfeed.com?", and found betaworks.  They run some very interesting sites/companies, some of which I knew about (bit.ly), some of which were new to me (findings.com, chartbeat.com).  The interesting one for me right now was another site called news.me.

Turns out that news.me is just the kind of thing that has the chance of making twitter useful to me.  If things go well, you'll probably hear more about it.

Attention to Error vs Attention to Detail

After having read Talent Code, a friend pointed me to the following article:

Why Do Some People Learn Faster?

Although the author of that article had gotten some lackluster reviews on some of his book-length work on Amazon, this article brought a distinct idea to the forefront for me:

Attention to Error

I've always thought that attention to detail was an important tendency in myself that made me a good learner.  But I realized that there is a distinction between paying attention to detail and paying attention to error.

The constant reconciliation between what you believe to be correct and what turns out to be correct is the fuel for my personal learning.

Some people might say this is what the scientific method is all about.  I'm less methodical than a scientist would be -- I find it nevertheless useful to constantly put myself in situations where I can learn informally from comparing what I expect to what actually happens.

Reality Vacuum

A place where people have chosen to ignore a certain part of reality, and where that choice to ignore is embedded in the culture of that place.

I know I've occupied such a space, and even contributed to the vacuum. However, once I realize what's going on, I always want out.

To some degree, each person in the world has to keep up in one way or another. Some just have a better feel for keeping in touch with current reality, and what to do when they're not.

Personally, I want to improve in my ability to close the loop better and faster.

Published with Blogger-droid v1.6.4

Playing catch-up

The world is changing all the time, which implies the requirement of constant learning to keep current. In other words, you are always behind.

If you accept the fact that you are always going to be behind, in one way or another, the problem then becomes more tractable. How, in a limited amount of time, can you bootstrap yourself into a learning environment where you can catch up enough to get something working?

The core questions are:

1. What to learn? (because of the limited time, you know you have to be selective)
   
2. How to go about it? (because of the limited time, and the constant churn, you have to be a quick learner/applier)
   

Andy Hunt wrote a book about pragmatic learning. Clayton Christensen wrote several books about disruptive innovation. The Wikipedia contributors wrote an article about the term "learning curve". The ideas in these books can be instructive.

I have my own opinion about the matter.

My answers to the core questions are:

1. Look around and get creative about how you can apply about-to-be-stable newer technology to the software problem at hand.
2. Climb the dynamic learning curve by becoming an "early adopter".

Being an "early adopter" is a productive approach to bootstrapping yourself into a rich learning environment. The key to quality learning is keeping it real & experiential. And trying new technology out and trying to apply it to the task at hand is certainly real & experiential.

The part that makes this whole learning equation possible is that the "innovators" actually need the "early adopters" in order to gain traction and stability. In an open world, that means that you can use early adoption as a means by which it is always possible to inject yourself into a rich and productive learning environment.

After I play this game for a while and become skilled at it, I'm guessing there will be a point at which I will want to have a talk with Paul Graham about a startup. Or maybe I will care about being an innovator in my family more than being famous in a technical sphere. Who knows.

Counterpoint

Donald Knuth is the classic example of someone whose life mission specifically excludes playing month-to-month catch-up. Oh, and by the way, that page returns the following HTTP header (in 2010):

Last-Modified: Fri, 23 Sep 2005 04:39:22 GMT

Even the innovation-encouraging Paul Graham wrote an article about addiction that cautions about blanket acceptance of technical improvements.

Reality Quotient

There is a fairly subjective measure I've only recently been able to give a name to:

Reality Quotient = ability to keep context while working toward a specific goal

This has to do with how deep you allow your stack to be, which if you allow it to get too deep, this affects your net throughput on Cockburn's "unvalidated decisions" or Demarco's "Total Useful Mental Discriminations (TUMD)". If the stack is too deep, you end up wasting a lot of time making useless decisions about things that have ceased having anything to do with the REAL task at hand.

The tendency to accept decisions that pin you into a corner is closely related to lowering the Reality Quotient. There is a whole book about the attitude of Getting Real, and I equate that attitude with a high Reality Quotient.

I did a search to see whether anyone else had published a writeup under the "Reality Quotient" heading. Although I found a lot of stuff on the web, none of it really matched what I wanted to say. The topic of this post is NOT:

• whether you are an honest person in general, or in a specific social situation
• a manager trying to find out what's REALLY going on in his/her org
• whether the actions of time travelers will have any effect
• some kind of measure of whether a politician is telling the truth or producing spin
• some measure of how "real" a reality show really is

The measurement I wanted to talk about is how capable you are of focusing on the problem you set out to solve until 1) it is truly solved and published to the world, or until 2) you have redefined the problem into another solvable one and published that transformation to the world.

In short, a high Reality Quotient requires a short stack, with tight feedback loops, focused on publishing real stuff to real people.