Tag Archives: history

Agile development: A nasty hack for a HR-Process bug?

Unless you have been living under a rock for more than a decade, you will probably have picked up on the concept of agile software development that has grown in both popularity and in the often dogmatic, piousness of its advocates.

In agile software development, as the concept goes, we address the industries high failure rate at successfully completing software projects on schedule while remaining faithful to its promised specifications. The idea is that many decades of software development have shown our inability as an industry to handle a project phase horizon of more than a few weeks.

That is, where our ancestors managed to plan and execute large projects that spanned decades and used hundreds of skilled laborers, the software industry has often shown its inability to plan and successfully execute a relatively small project of a few years with only a few dozen craftsman.

The solution, according to the Agile advocates has been to start small. Build something that works but implements only a small fragment of the specs, and than use incremental enhancements to one day arrive at the desired goal.  If we would project this idea to our ancient history, it would be the equivalent of the ancient Egyptians building a gazebo, upgrade it to a shed, as to, in a few hundred two-week iterations finally finishing a full blown pyramid.  While this may sound ludicrous, there does seem to be empirical data that shows that for software engineering this approach is actually working relatively well. At least, that its working quite well as opposed to what was happening before whole the Agile hype started out.

So Agile might have fixed a problem that needed fixing, but it did in no way answer the question of ‘why’ software engineering had been suffering from that problem while apparently in other fields, the dreaded up-front planning and long project phases had been working out pretty darn good for the last four or five millennia at least.

I believe the root of the issue can be found in an essay by Edsger Wybe Dijkstra where he writes:

My third remark introduces you to the Buxton Index, so named after its inventor, Professor John Buxton, at the time at Warwick University. The Buxton Index of an entity, i.e. person or organization, is defined as the length of the period, measured in years, over which the entity makes its plans. For the little grocery shop around the corner it is about 1/2,for the true Christian it is infinity, and for most other entities it is in between: about 4 for the average politician who aims at his re-election, slightly more for most industries, but much less for the managers who have to write quarterly reports. The Buxton Index is an important concept because close co-operation between entities with very different Buxton Indices invariably fails and leads to moral complaints about the partner. The party with the smaller Buxton Index is accused of being superficial and short-sighted, while the party with the larger Buxton Index is accused of neglect of duty, of backing out of its responsibility, of freewheeling, etc.. In addition, each party accuses the other one of being stupid. The great advantage of the Buxton Index is that, as a simple numerical notion, it is morally neutral and lifts the difference above the plane of moral concerns. The Buxton Index is important to bear in mind when considering academic/industrial co-operation.

Lets look at this brilliant and important assertion by Dijkstra, and projecting it to our long-running project problem. For now lets look at them as long running-projects instead of long-running projects. We see that basically we have people who we could classify as marathon runners (high Buxton index) , people who we could classify as sprinters (low Buxton index) and the whole spectrum in between. From this we could raise the hypothesis that our earlier problem with completing long running project could well have been the result of having people with a low Buxton index assigned to projects that warranted a high Buxton team. That is we might have been trying to run a marathon with teams made up primary of sprinters.

Looking at some of the more popular Agile methodologies, we see that Scrum actually uses the term sprint to refer to a short iteration in witch work is done. Scrum divides a project into small, maybe two-week iterations called sprints. In this way, Scrum basically is dividing a marathon into small 400 meter dash distances suitable for sprinters.  That is, rather than fixing the problem of having the wrong type of people for the size of the project by trying to get the right kind of people, Scrum actually solves the problem by chopping up the project into byte-size chunks so our low-Buxton individuals can function in projects that used to be a bad fit for their low Buxton index.  As a result however, this has led to high-buxton index individuals that were well suited for the original problem, now feeling very unhappy and reduced to factory workers, having to do short repetitive tasks, as to them a two week sprint feels like being forced to work on an unimportant detail that is only distracting from the long term goals. This again may thus drive these high-Buxton individuals to seek a career outside of software-engineering.

Now an important question to ask is: how did we arrive at the point where software engineering teams for larger projects became so dominantly infested with low-Buxton individuals as to warrant the use of something like Scrum to get ourself producing output successfully again? My thesis is that our HR-Process has a bug. A major bug. The HR process in general and for software engineers in particular has devolved over the decades into a process that favors the hacker over the engineer, the sprinter over the marathon runner, the trivia over the deep understanding. In our earlier history we had the master apprentice system for many trades, and in many old trades remnants of this system are still present in the HR-process. Software engineering however has no roots in a master/apprentice system. Without these roots and with our fast passed HR-departments these days, the HR-Process for this relatively new trade has suffered most from a low-Buxton index favoring process.

Combining these facts and my thesis I would dare to pose the idea that the whole Agile movement as we are seeing today is the direct result of the lack of a master-apprentice culture in software engineering and the current day fast passed HR-process. That is, Agile has turned out to be a surprisingly effective but nasty hack for a bug in our HR-process. A bug that is likely going to show up in other trades as well as the master/apprentice roots of many old trades pass into oblivion.

Advertisements

Capibara.com and op.nu

Recently I received a message that my capibara.com domain required renewal. Capibara.com is the only remaining domain of the first 3 domains I owned and it has quite a history, but in recent years I basicaly forgot about this domain.

The renewal message made me remember the history of this domain, and made me think that I shouldn’t allow this domain that did play a modest but important role in the history of the world wide web to waste away.  So I decided to put wordpress up on this domain and this is my first blog. It seems only apropriate to dedicate my first blog post to the history of this domain.

The history of capibara.com (and the now long expired op.nu domain) started 14 years ago. At that time domain hosting wasn’t as cheap as it is today, in fact it was shamefully expensive.  With capibara.com I tried to change that. Fortunately in those days HTTP hosting had no virtual hosting, so my capibara.com server at that time was running on its own IP.  I created a simple index.cgi in Perl that looked at the requested domain, looked up an URL in an on-disk hashmap belonging with this domain, and generated a single frame page that hid the fact that the page wasn’t hosted at the requested domain. I created a script where people could register their domain and long url with capibara.com, and capibara.com would map the domain to the url. Capibara.com didn’t solve everything, but an other free service, granitecanyon that offered a free DNS service, and multiple providers of free web space (with long ugly url’s) combined to make it possible to get your own domain for just the domain registry costs.

For people for who this was still to much, I had the domain op.nu registered, and at granitecanyon had a wildcart domain entry for this domain. People could claim a free sub domain of op.nu.

For years this service helped people with litle money to be able to affort their own domain. At the end there were many thousands of domains mapper using the capibara.com url mapping service. But than things went badly. The hosting company that hosted capibara.com and tolerated the domain hosting service went bust, and as you may understand, other companies were unwilling to accept capibara.com on their servers.

At that same time ADSL was starting to pick up, but most ADSL providers didn’t deliver good quality of service on these lines, and Linux was getting prety popular already. The CDUCK project was born to fill the gap that the demise of the capibara free url cloacking service left. CDUCK was a distributed version of the services provided by granitecanyon and capibara, one that people could run at home.

With hosting programs now affordable to most and CDUCK orphaned as project on sourceforge, there is not much left of my original capibara.com initiative, neither is it needed anymore, but I do feel that capibara.com has played a modest yet important role in the history of the world wide web, one that makes this domain deserving of a second life as my personal blog space.