Tag Archives: Craftsman

Agile development: A nasty hack for a HR-Process bug?

Unless you have been living under a rock for more than a decade, you will probably have picked up on the concept of agile software development that has grown in both popularity and in the often dogmatic, piousness of its advocates.

In agile software development, as the concept goes, we address the industries high failure rate at successfully completing software projects on schedule while remaining faithful to its promised specifications. The idea is that many decades of software development have shown our inability as an industry to handle a project phase horizon of more than a few weeks.

That is, where our ancestors managed to plan and execute large projects that spanned decades and used hundreds of skilled laborers, the software industry has often shown its inability to plan and successfully execute a relatively small project of a few years with only a few dozen craftsman.

The solution, according to the Agile advocates has been to start small. Build something that works but implements only a small fragment of the specs, and than use incremental enhancements to one day arrive at the desired goal.  If we would project this idea to our ancient history, it would be the equivalent of the ancient Egyptians building a gazebo, upgrade it to a shed, as to, in a few hundred two-week iterations finally finishing a full blown pyramid.  While this may sound ludicrous, there does seem to be empirical data that shows that for software engineering this approach is actually working relatively well. At least, that its working quite well as opposed to what was happening before whole the Agile hype started out.

So Agile might have fixed a problem that needed fixing, but it did in no way answer the question of ‘why’ software engineering had been suffering from that problem while apparently in other fields, the dreaded up-front planning and long project phases had been working out pretty darn good for the last four or five millennia at least.

I believe the root of the issue can be found in an essay by Edsger Wybe Dijkstra where he writes:

My third remark introduces you to the Buxton Index, so named after its inventor, Professor John Buxton, at the time at Warwick University. The Buxton Index of an entity, i.e. person or organization, is defined as the length of the period, measured in years, over which the entity makes its plans. For the little grocery shop around the corner it is about 1/2,for the true Christian it is infinity, and for most other entities it is in between: about 4 for the average politician who aims at his re-election, slightly more for most industries, but much less for the managers who have to write quarterly reports. The Buxton Index is an important concept because close co-operation between entities with very different Buxton Indices invariably fails and leads to moral complaints about the partner. The party with the smaller Buxton Index is accused of being superficial and short-sighted, while the party with the larger Buxton Index is accused of neglect of duty, of backing out of its responsibility, of freewheeling, etc.. In addition, each party accuses the other one of being stupid. The great advantage of the Buxton Index is that, as a simple numerical notion, it is morally neutral and lifts the difference above the plane of moral concerns. The Buxton Index is important to bear in mind when considering academic/industrial co-operation.

Lets look at this brilliant and important assertion by Dijkstra, and projecting it to our long-running project problem. For now lets look at them as long running-projects instead of long-running projects. We see that basically we have people who we could classify as marathon runners (high Buxton index) , people who we could classify as sprinters (low Buxton index) and the whole spectrum in between. From this we could raise the hypothesis that our earlier problem with completing long running project could well have been the result of having people with a low Buxton index assigned to projects that warranted a high Buxton team. That is we might have been trying to run a marathon with teams made up primary of sprinters.

Looking at some of the more popular Agile methodologies, we see that Scrum actually uses the term sprint to refer to a short iteration in witch work is done. Scrum divides a project into small, maybe two-week iterations called sprints. In this way, Scrum basically is dividing a marathon into small 400 meter dash distances suitable for sprinters.  That is, rather than fixing the problem of having the wrong type of people for the size of the project by trying to get the right kind of people, Scrum actually solves the problem by chopping up the project into byte-size chunks so our low-Buxton individuals can function in projects that used to be a bad fit for their low Buxton index.  As a result however, this has led to high-buxton index individuals that were well suited for the original problem, now feeling very unhappy and reduced to factory workers, having to do short repetitive tasks, as to them a two week sprint feels like being forced to work on an unimportant detail that is only distracting from the long term goals. This again may thus drive these high-Buxton individuals to seek a career outside of software-engineering.

Now an important question to ask is: how did we arrive at the point where software engineering teams for larger projects became so dominantly infested with low-Buxton individuals as to warrant the use of something like Scrum to get ourself producing output successfully again? My thesis is that our HR-Process has a bug. A major bug. The HR process in general and for software engineers in particular has devolved over the decades into a process that favors the hacker over the engineer, the sprinter over the marathon runner, the trivia over the deep understanding. In our earlier history we had the master apprentice system for many trades, and in many old trades remnants of this system are still present in the HR-process. Software engineering however has no roots in a master/apprentice system. Without these roots and with our fast passed HR-departments these days, the HR-Process for this relatively new trade has suffered most from a low-Buxton index favoring process.

Combining these facts and my thesis I would dare to pose the idea that the whole Agile movement as we are seeing today is the direct result of the lack of a master-apprentice culture in software engineering and the current day fast passed HR-process. That is, Agile has turned out to be a surprisingly effective but nasty hack for a bug in our HR-process. A bug that is likely going to show up in other trades as well as the master/apprentice roots of many old trades pass into oblivion.


Artist versus Craftsman.

If there is one thing I’ve learned the hard way with respect to software engineers, its to never trust first impressions. In many other aspects I have always been extremely good at sizing people up based on my first impression. To such an extend that some of my friends have jokingly accused me of witchcraft.
In contrast, with respect to software engineers and their skill and productivity level, I’ve turned out to have my first impressions proven wrong so many times that I stopped trusting them.

Sure, I know how to spot a standard mediocre software engineer based on first impression, but a good one, or one that generates an exceptional high percentage off the bugs for a project,  for some reason I’ve proved to be incapable of spotting the difference between those two. To me it seemed like the difference between being exceptional and being a major source of bugs was a subtle difference that I was unable to spot. Both the exceptional software engineer and the source of many bugs software engineers share major characteristics. Both are highly intelligent, highly creative and both have a strongly developed sense for aesthetics.

It took me many years of working in software engineering to notice a pattern emerge that seemed to point to what this difference might be. After working for a while with other software engineers, there were some software engineers that at some point felt the need to emphasize the creative aspects of their work as software engineer by referring to themselves as or comparing themselves with an ‘artist’.  The pattern that seemed to emerge was that for the group of highly  intelligent, highly creative software engineers with a strong sense for aesthetics, the eventual use of the word ‘artist’ and/or ‘artistic’  seemed to coincide with the sub group that instead of being exceptional ended up being a major source of bugs.

So, with all other things being seemingly equal between the exceptional software engineer and the software engineer that turns out to become a major source of bugs, does being an artist give us any pointers to what is different between these two groups? As someone who, before becoming a software  engineer, spent much of his time making drawings and air-brush paintings, I found this notion rather disagreeable.  For me, different from those ‘artist’ I talk about, software engineering isn’t an art but a craft. When we look back at the history of art, we notice an important paradox. Many of the most adored and most priceless pieces of paintings that were ever made, were created in an era when painting wasn’t considered an art and no painter would think of himself or refer to himself as an artist. A painter was a craftsman that took pride in his craft. He used his creativity in a way subordinate to his craftsmanship and the result, looking at the old masters was amazing and unsurpassed by most modern artist type painters.

So maybe the parallel between software engineering and artistic crafts like painting is indeed valid, but more than painting, choosing the  artist approach to software engineering rather than the craftsman approach is a road not without peril.

So basically the difference between being a major source of bugs and being exceptional at software engineering might just be a state of mind. Considering your craftsmanship as a subordinate tool to your artistic creativity  might be the thing that is holding you back from reaching exceptional heights as a software engineer. So you are an artist, fine, we’ll go with that. Take an example from the old masters. Make your artistic creativity subordinate to your craftsmanship. The old masters did it and produced some of the most beautiful paintings.If you are one of those highly intelligent, highly creative software engineers with  a strongly developed sense for aesthetics who considers himself an artist, this little shift in your state of mind might transform you to an exceptional software engineer without giving up on viewing software engineering as an artistic process.