Introducing Familinearity

Familinearity is when a person makes decisions in a new situation based on some other familiar situation while assuming that the relation between them is linear. This is also how software development planning is done in most companies.

The idea for this term came to me last Wednesday, when I have attended the FogBugz World Tour in Tel Aviv to finally see Joel in person and also learn a bit about version control and bug management. Both the organization of the event and the presentations were excellent and enjoyable completely making up for the apparent dryness of the subject. Even though I do not use FogBugz myself, I got an impression that it is an excellent and very powerful product, well integrated with version control systems and easy enough to use in both medium scale and large projects.

One feature in FogBugz that really caught my attention, however, was the Evidence Based Scheduling (EBS). The idea behind it was to match actual time of task performance to the estimated one for each developer over some period of time and then statistically process it to derive meaningful information about project planning and scheduling. Such information could, for example, include the developer timelines and velocity as well as estimated completion times that would allow you to determine a completion date for any subset of the project with definite probability.

A priori, I am extremely skeptical about the overall precision and usefulness of such methods. Not to say that evidence based approach in general is incorrect, it is certainly in fashion and gaining speed in medical community while also taking shots at other areas. It is other considerations, both practical and theoretical nature that cause my doubts and I will explain them in brief.

The practical concerns for the applicability of this tool are too long for this post, probably too long an entire book. Just a quick look at the article called "How to estimate software tasks" from FogBugz documentation can give a great example of the problems we face in this area. One paragraph towards the end of the article claims that giving people unrealistic schedules and hope they will be "motivated" by them is "brain-dead", a statement with which I completely agree. It is still a fact, however, that there are entire companies that live by this principle and lots of managers who did not read "A Mythical Man-Month" even though it is, as the article claims, a requirement. Joel Spolsky himself, of course, knows more than most people about those problems and have written numerous articles on the subject. These companies, by the way, can at times be very successful, and you will have a hard time trying to convince them that their management practices are flawed.

My point, in this case is that when a tool like EBS is deployed in an organization that has that specific management culture and later, one of the team leaders comes up to his boss to claim that based on previous experience his estimate has only 2% chance of being correct, it will more likely result in fireworks than in the manager amending his ways. Another problematic situation might occur when the manager will come to the above team leader and inquire why the feature that was a week ago 99% on time still late. All that, assuming the organization really deployed EBS and paid all the additional costs of constantly updating all estimates and drilling down all tasks without which the method will not work at all at best or produce highly inaccurate results at worst.

But even if we leave the practicalities aside, the theoretical aspects of using bootstrapping based statistical methods seems a bit problematic. As Joel himself would tell you, in software you rarely do the same things twice. It is design, and design is similar to an exploration of a highly fluid terrain, with high complexity. Using methods that use linear extrapolation based on precision of your previous estimates to provide precise answers seems at least problematic and might raise similar objections to the ones Nassim Taleb provided for the use of VAR formula in economics.

I am not saying that this tool is not useful, it might for example, reveal the "epistemic arrogance" of the management by showing the ill precision of their previous estimates and thus benefit tired and demotivated programmers. It might provide a good bottom margin for the prediction errors. But not taking into account the complexity and non linearity of the domain at hand it could hardly offset the existing "familinerarity" of the software development planning process.

Software Development and Complexity Theory

Some time ago, I have become interested in Complexity Theory. I do not remember the exact path that lead me to it, but as usual it was probably one thing dragging another until one of the links I clicked brought me to it and I was hooked.

Complexity Theory is a rather broad field of knowledge involving many different scientific areas and disciplines. It is far from coherent and complete, even some definitions of what complexity really means and how it can be measured are still being debated.

Once I have started reading on the subject, I have immediately become interested, among other things, in applications of Complexity Theory to software development, management and creativity. To understand how these fields might be connected, lets start with the following diagram taken from the Cynefin framework, created by Dave Snowden and his company "Cognitive Edge".



This diagram describes a framework for thinking about systems in four different categories: Simple, Complicated, Chaotic and Complex.

An example of a simple system is an old bicycle, the input of force to the pedal is transformed into a wheel rotation in an obvious and predictable way. A complicated system is a watch, which has a much less obvious structure but is still linear and predictable. An example of chaotic system is, say, gas molecules in a closed volume. We can not predict the movement of each individual molecule but we can statistically predict their behavior in general.

Complex systems, however, are something entirely different. They are non linear and dynamic, meaning that the same input may sometimes result in entirely different output. They are, thus, inherently unpredictable and can not be controlled directly requiring an entirely different type thinking and a new set of tools to deal with. (I will talk much more in next posts but for for now if you want more info please read the Wikipedia entry and watch this video of Dave Snowden explaining this concept and other related ideas)

This is all of course a monstrous oversimplification, for a great introduction I would recommend an excellent book "Complexity: A Guided Tour" by Melanie Mitchell.

What I will talk about here however, is specifically the implications of complexity in software development. While most software is complicated, the software development process however is complex. And this has a great impact on how the software development process should be organized and managed.

As obvious as this may sound, far too many managers in software companies have little or no understanding of this fact and keep trying to construct complicated rigid processes that should cover every little detail in the work flow, or use outdated methodologies that were originated in the large factories of the early industrial era. Since measuring complex systems in general, and software development process in particular is extremely difficult if not impossible they usually get away with it and move on to another project by blaming the failure on some external circumstance that is always out there.

I am convinced, however, that proper understanding of complexity issues and their impact on software development can improve the process as well as the resulting software and is important to study and understand. To kick off this track that I am hoping to continue researching in the near future I would really like to recommend this terrific presentation by Jurgen Appelo. It might sound a bit complicated for a beginner, but it is spot on, covering many issues and providing extensive list of references for future research and discussion.

Possibly the best (worst) poker beat I have ever seen

I have always thought that a Poker is a nice simplistic model of our reality, but this one beat is just plain awesome!

Prices of eBooks

The eBook reader market is on the rise. I have just got a brand new Sony Reader 600, and the Nook from Barnes & Noble, due in November, also looks fantastic. The prices of eBooks however, are still way to high, and for no apparent reason. After all, conventional logic shows that if you do not have to print, ship and store the book, it should make it much cheaper for the publishers. You can sell any amount of books and never run out of stock. You also don't take the risk of getting stuck with shelves of unsold books nobody wants. You do not need to rent a store, pay the cleaning lady and the clerk and the driver and the supervisor. Doesn't all this amount to something?

In order to understand why the prices are so high, I had to undertake a little investigation (read few blogs and do few searches). I have found several "logical" explanations why an eBook should cost 9.90 (as they mostly do) and why not printing the book does not really affect the cost too much. Some of the authors go to great length and provide some interesting numbers.

This guy, for example, shows a detailed breakdown of a book cost. He argues that this shows that printing is actually a minor part of expenses. And he is not alone. Here is another guy who is wrong and another. Let's take a look at this and see what is wrong with this picture.

"Based on a list price of $27.95

$3.55 - Pre-production - This amount covers editors, graphic designers, and the like
$2.83 - Printing - Ink, glue, paper, etc
$2.00 - Marketing - Book tour, NYT Book Review ad, printing and shipping galleys to journalists
$2.80 - Wholesaler - The take of the middlemen who handle distribution for publishers
$4.19 - Author Royalties - A bestseller like (John) Grisham will net about 15% in royalties, lesser known authors get less. Also the author will be paying a slice of this pie piece to his agent, publicist, etc.

This leaves $12.58, Money magazine calls this the profit margin for the retailer, however, when was the last time you saw a bestselling novel sold at its cover price."

He then suggests to count the costs from zero up, and suggesting that pre production, marketing and author fees are essentially the same, 9.90 is actually pretty cheap even for an electronic edition.

Everybody got that? Now let's see why this should clearly not be the case. In order to illustrate the point I will take Malcolm Gladwell as an author, and his three books, "The Tipping Point", "Blink" and "Outliers", last one published in 2009 and has a list price on Amazon of 27.99

First let's take a look at marketing. I first saw Malcolm at TED for free. Cause you see, where I live, there are no book tours, no New York Times and no New Yorker. So all that money spent on that advertising, for me, is money wasted. Now let's note, that Malcolm Gladwells first book sold about 2.5 million copies, which according to the above breakdown should amount to 5 million bucks in ads alone.

If this sounds strange, check this out. According to the above logic of breaking the book cost down to it's components, the cover editors and graphic designers of the above book earned little under 9 million dollars. Wow, I must be in the wrong business. Not to mention the author, who (once again according to the above calculation) got 10 million dollars in royalties.

All this leads me to the conclusion that the above breakdown, is a little bit problematic. The costs of producing, storing and selling books online, are close to nothing, taking into consideration the advantages of online distribution such as:

• Instant access to any book (30 seconds from seeing Malcolm on TED to buying his book on Amazon).
• No need to wait for delivery, start reading immediately.
• No forests are hurt in the process
• Availability of recommendation engines save marketing costs.
  
• Vast worldwide markets, not covered by paper book distribution.
  

So to conclude, since there are almost no necessary costs, save for author royalties, a fair price for an average eBook is 1.99$ US. What should be understood that in a modern networked world there should be an author and a reader and everything in between is irrelevant and not interesting. Period.

New Apple Mouse Looks Amazing

If you had not already go to apple.com and marvel the new Magic Mouse. What they basically did is put their superior mouse pad that was used in laptops on a regular mouse thus creating an all touch interface that supports gestures in combination with a regular convenient point and click experience.

The "wow" effect is guaranteed and it looks really slick and amazing.

PS: Just to balance things, they did a much poorer job on the new Apple Remote.

Michael Moore Movies Free Online

I have seen some interviews with Michael Moore on YouTube recently. I have heard about him before but only recently due to his Capitalism movie I have taken the time to study his work more closely.

The very first thing that struck me is why are not all of his movies available for direct download online and for free, I mean if he is such a democrat. After all, he has made a lot of money from those movies already, covering his costs many times over. What's the matter? Maybe he is greedy too? Does he really want to spread his ideas or just make some more bucks on the other part of our fears, the ones that has not been yet exploited by the government.

The only thing I found immediately is his movie "Slacker Uprising" and it was only available for download after you supply and email (what for?) and for citizens of US and Canada.

Well I guess it is much easier to talk about democracy and freedom than really practicing it.

Windows 7 Annoyances: Aero Peek

Are you annoyed by the Windows 7 Alt + Tab hiding your windows and behaving weirdly, want to go back to the old way, go to System Properties -> Advanced -> Performance Options and disable "Aero Peek" option. Downside, this will also disable full window preview from of multiple windows from the taskbar. Make your choice.