Command The Computer

After hearing hype around machine learning eating up tech jobs, I've wondered if I just have my head in the ground about what kind of work I'm doing.  So far, I haven't seen a straightforward way to apply machine learning to my work goals.

However, today I realized I can start a mind shift toward my work that will be both more healthy for me personally, and will allow me to see opportunities for machine learning that I've been missing.

There are 4 main levels of work I can see:

1. Initiative
2. Project
3. Task
4. Micro-task

As an individual contributor or team lead, the aspects of the work are scoped down to levels 2-4.  Most of the opportunities I've seen for applying machine learning have been at the Initiative level, where statistical methods can be applied to solve novel problems or automate whole classes of new/existing work.

Now, imagine you already work at levels 2-4.  Imagine you have a computer with infinite intelligence.  Imagine that if you describe a piece of work that needs to be done down to the 80% level of precision, that this intelligent computer could get it done and take care of the details.  This is what I've found impossible to imagine in advance.

The best way I've found to imagine this is as follows:

• do a Task the way I've always done it (level 3)
• break the task up into atomic pieces by creating self-standing git commits as I go (one commit per level 4 Micro-task)
• when I write each git commit, imagine that I had asked a computer to do what I just did
• write the git commit in imperative form, as if I had commanded the computer to accomplish the work at the 80% level of precision
• imagine that I had spent the last 30-60min doing something other than solving the problem, and ask myself, "What else would I have been able to do while the computer was spending 15-30min on this?"

This helps me wire up the neurons in my own brain to start thinking about the computer as an intelligent agent that can assist me.  And it helps me to imagine how I can use my attention and energy more effectively, rather than just solving micro-tasks all day.

It appears to me that applying machine learning effectively requires stepping back at least to the Project or Initiative level and delegating more work to an "intelligent" computer.  And if such an agent doesn't exist, perhaps it can be built.

In a weird sort of way, I'm starting to use the Tell, Don't Ask principle in my own thinking to enhance my ability to imagine solutions coming together more quickly.  I know that's taking the Law of Demeter way outside of its traditional scope.  But I'm trying to break out of the box here. ;)

If 2018 was a year I didn't write any blog posts, it was certainly a year of great personal growth.  I look forward to writing here more during 2019.

Dystopia - Fallacies Conquered

Dystopia - from NetflixOSS's Adrian Cockcroft:

We have spent years striving to build perfect apps running on perfect kernels on perfect CPUs connected by perfect networks, but this utopia hasn't really arrived.

Instead we live in a dystopian world of buggy apps changing several times a day running on JVMs running on an old version of Linux running on Xen running on something I can't see, that only exists for a few hours, connected by a network of unknown topology and operated by many layers of automation.

Reminds me of Fallacies of Distributed Computing, but from the perspective of people who have actually vanquished the beast with weapons that are usable by others.

Intrusion Detection through Stackable Filesystems

I've always wondered what exploit might be running on my system, and never had any time to devise/install a detection system that would have the right balance of useful detection (maximize) and performance impact (minimize).

When I stumbled upon unionfs a couple weeks ago, I thought that was an interesting idea from a change-logging perspective.  It's sometimes useful to be able to keep a filesystem-based diff of what a certain operation does to a system, and then bake it onto the system when I know it did what I wanted to.  The takeaway for me was that unionfs's performance profile had the opportunity to be so low because it was so thin and so baked into the kernel.

This compounded with my recent discovery and fiddling with fusefs (in user-land) and I wondered about what kind of useful logic I could put underneath a filesystem.  The GlusterFS feature set, the recent LessFS GC stuff, the bup-fuse stuff, and the S3FS stuff is all just *really* cool.  I ended up gazing longingly at the big backlog of fuse filsystem suggestions on their wiki, and wandered back into the unionfs space.

So, when I saw the I3FS paper (linked from here) about the modular application of this technique to intrusion detection, this really triggered the Useful *AND* Performant neurons and I got really excited.

Unfortunately, on first glance, the stackable filesystems stuff seemed pretty cryptic to set up in a lightweight, just-works kind of way (think custom mount command-lines complete with arcane stacking options).

It would be soooo awesome to have an easy-to-compose ruby DSL for doing some kind of rack-like filesystem mashup with a kernel-level unionfs layer underneath a user-land fusefs layer, but all expressed in the same DSL.

This would be an awesome tool to put on top of the Arch-derived clone I want to put up for people at work.  There are folks who care about living more on the edge of linux stuff, but that don't care to install from scratch, and also might not care that much about not having a full Gnome stack if things just work.  And if I could give them the same tuned IDS-on-the-desktop solution (or upgrade their developer stack by letting them pull a filesystem delta over), that would be really cool.

The cheap development observation [PDF, linked from] because of modularity is one of attributes valued highly by the Ruby community as well.  It is one the key things that makes Ruby as a community awesome.

These kinds of ideas really matter, and making them so cheap and stable that you don't have to think about them really matters even more.

NOTE: The I3FS paper is really pretty old (2004), and the whole unionfs stack is older than that.  The fuse stack came into the kernel a long time ago too (2005).  So while this is new to me, it's been around for quite a long time.  I'm playing catch-up.

Semantic Versioning

After having skimmed the semantic versioning proposal/spec, I really like it, and I'm going back for a deep read.

The most notable violator of this that has bit me in the past has been the jersey framework, and maybe earlier versions of commons-collections.

Software Architecture

This is what I hate about overreaching ideas and projects that accept them:


The whole "coming soon" idea is fine if it lasts for a couple days, but years and years?

Now I'm hanging my head in shame about www.sumsion.org. :)