Software Development and Complexity Theory

Some time ago, I have become interested in Complexity Theory. I do not remember the exact path that lead me to it, but as usual it was probably one thing dragging another until one of the links I clicked brought me to it and I was hooked.

Complexity Theory is a rather broad field of knowledge involving many different scientific areas and disciplines. It is far from coherent and complete, even some definitions of what complexity really means and how it can be measured are still being debated.

Once I have started reading on the subject, I have immediately become interested, among other things, in applications of Complexity Theory to software development, management and creativity. To understand how these fields might be connected, lets start with the following diagram taken from the Cynefin framework, created by Dave Snowden and his company "Cognitive Edge".



This diagram describes a framework for thinking about systems in four different categories: Simple, Complicated, Chaotic and Complex.

An example of a simple system is an old bicycle, the input of force to the pedal is transformed into a wheel rotation in an obvious and predictable way. A complicated system is a watch, which has a much less obvious structure but is still linear and predictable. An example of chaotic system is, say, gas molecules in a closed volume. We can not predict the movement of each individual molecule but we can statistically predict their behavior in general.

Complex systems, however, are something entirely different. They are non linear and dynamic, meaning that the same input may sometimes result in entirely different output. They are, thus, inherently unpredictable and can not be controlled directly requiring an entirely different type thinking and a new set of tools to deal with. (I will talk much more in next posts but for for now if you want more info please read the Wikipedia entry and watch this video of Dave Snowden explaining this concept and other related ideas)

This is all of course a monstrous oversimplification, for a great introduction I would recommend an excellent book "Complexity: A Guided Tour" by Melanie Mitchell.

What I will talk about here however, is specifically the implications of complexity in software development. While most software is complicated, the software development process however is complex. And this has a great impact on how the software development process should be organized and managed.

As obvious as this may sound, far too many managers in software companies have little or no understanding of this fact and keep trying to construct complicated rigid processes that should cover every little detail in the work flow, or use outdated methodologies that were originated in the large factories of the early industrial era. Since measuring complex systems in general, and software development process in particular is extremely difficult if not impossible they usually get away with it and move on to another project by blaming the failure on some external circumstance that is always out there.

I am convinced, however, that proper understanding of complexity issues and their impact on software development can improve the process as well as the resulting software and is important to study and understand. To kick off this track that I am hoping to continue researching in the near future I would really like to recommend this terrific presentation by Jurgen Appelo. It might sound a bit complicated for a beginner, but it is spot on, covering many issues and providing extensive list of references for future research and discussion.