Unless you have been living under a rock for more than a decade, you will probably have picked up on the concept of agile software development that has grown in both popularity and in the often dogmatic, piousness of its advocates.
In agile software development, as the concept goes, we address the industries high failure rate at successfully completing software projects on schedule while remaining faithful to its promised specifications. The idea is that many decades of software development have shown our inability as an industry to handle a project phase horizon of more than a few weeks.
That is, where our ancestors managed to plan and execute large projects that spanned decades and used hundreds of skilled laborers, the software industry has often shown its inability to plan and successfully execute a relatively small project of a few years with only a few dozen craftsman.
The solution, according to the Agile advocates has been to start small. Build something that works but implements only a small fragment of the specs, and than use incremental enhancements to one day arrive at the desired goal. If we would project this idea to our ancient history, it would be the equivalent of the ancient Egyptians building a gazebo, upgrade it to a shed, as to, in a few hundred two-week iterations finally finishing a full blown pyramid. While this may sound ludicrous, there does seem to be empirical data that shows that for software engineering this approach is actually working relatively well. At least, that its working quite well as opposed to what was happening before whole the Agile hype started out.
So Agile might have fixed a problem that needed fixing, but it did in no way answer the question of ‘why’ software engineering had been suffering from that problem while apparently in other fields, the dreaded up-front planning and long project phases had been working out pretty darn good for the last four or five millennia at least.
I believe the root of the issue can be found in an essay by Edsger Wybe Dijkstra where he writes:
My third remark introduces you to the Buxton Index, so named after its inventor, Professor John Buxton, at the time at Warwick University. The Buxton Index of an entity, i.e. person or organization, is defined as the length of the period, measured in years, over which the entity makes its plans. For the little grocery shop around the corner it is about 1/2,for the true Christian it is infinity, and for most other entities it is in between: about 4 for the average politician who aims at his re-election, slightly more for most industries, but much less for the managers who have to write quarterly reports. The Buxton Index is an important concept because close co-operation between entities with very different Buxton Indices invariably fails and leads to moral complaints about the partner. The party with the smaller Buxton Index is accused of being superficial and short-sighted, while the party with the larger Buxton Index is accused of neglect of duty, of backing out of its responsibility, of freewheeling, etc.. In addition, each party accuses the other one of being stupid. The great advantage of the Buxton Index is that, as a simple numerical notion, it is morally neutral and lifts the difference above the plane of moral concerns. The Buxton Index is important to bear in mind when considering academic/industrial co-operation.
Lets look at this brilliant and important assertion by Dijkstra, and projecting it to our long-running project problem. For now lets look at them as long running-projects instead of long-running projects. We see that basically we have people who we could classify as marathon runners (high Buxton index) , people who we could classify as sprinters (low Buxton index) and the whole spectrum in between. From this we could raise the hypothesis that our earlier problem with completing long running project could well have been the result of having people with a low Buxton index assigned to projects that warranted a high Buxton team. That is we might have been trying to run a marathon with teams made up primary of sprinters.
Looking at some of the more popular Agile methodologies, we see that Scrum actually uses the term sprint to refer to a short iteration in witch work is done. Scrum divides a project into small, maybe two-week iterations called sprints. In this way, Scrum basically is dividing a marathon into small 400 meter dash distances suitable for sprinters. That is, rather than fixing the problem of having the wrong type of people for the size of the project by trying to get the right kind of people, Scrum actually solves the problem by chopping up the project into byte-size chunks so our low-Buxton individuals can function in projects that used to be a bad fit for their low Buxton index. As a result however, this has led to high-buxton index individuals that were well suited for the original problem, now feeling very unhappy and reduced to factory workers, having to do short repetitive tasks, as to them a two week sprint feels like being forced to work on an unimportant detail that is only distracting from the long term goals. This again may thus drive these high-Buxton individuals to seek a career outside of software-engineering.
Now an important question to ask is: how did we arrive at the point where software engineering teams for larger projects became so dominantly infested with low-Buxton individuals as to warrant the use of something like Scrum to get ourself producing output successfully again? My thesis is that our HR-Process has a bug. A major bug. The HR process in general and for software engineers in particular has devolved over the decades into a process that favors the hacker over the engineer, the sprinter over the marathon runner, the trivia over the deep understanding. In our earlier history we had the master apprentice system for many trades, and in many old trades remnants of this system are still present in the HR-process. Software engineering however has no roots in a master/apprentice system. Without these roots and with our fast passed HR-departments these days, the HR-Process for this relatively new trade has suffered most from a low-Buxton index favoring process.
Combining these facts and my thesis I would dare to pose the idea that the whole Agile movement as we are seeing today is the direct result of the lack of a master-apprentice culture in software engineering and the current day fast passed HR-process. That is, Agile has turned out to be a surprisingly effective but nasty hack for a bug in our HR-process. A bug that is likely going to show up in other trades as well as the master/apprentice roots of many old trades pass into oblivion.