Using 'email' instead of 'username' in Passport.js

Just spent ah hour debugging why Passport.js would not forward my login request to the LocalStrategy. After putting some logs inside the strategy itself I have realized that it expect to see a 'username' field in the form, while I was sending an 'email'. Better answer (not mine) with code example here.

Good examples of Open Government implementation

It is not easy to come by really good implementations of systems for Open Government. For those who speak Russian, here is a really good example. It seems to be frozen at the moment but the idea and the implementation are very nice.

http://veche.duma.gov.ru

PYTHON: Reimplement imported methods for testing purposes

Spent some time figuring this out, though I'd share. Who knows someone might save some time.

I was trying to write a Django unit test for a method that performs some logic and then sends out several emails by calling the "send_email()" method. For the testing purposes, I wanted to replace the original "send_email()" method with "mock_send_email()". This would have been straightforward, if not for the fact that "send_email()" was not implemented in the tested module itself but in another module imported by it.

Here is what finally worked.

emailutils.send_email = mock_send_email # replace the method 

reload(testedmodule) # reload the module 

from testedmodule import testedmethod # import again 

testedmethod() # run the method 

Hope this helps.

Good answer

Check out this great answer to why software development time estimates are, to say it mildly, imprecise. Enjoy

No Country For Old Mann

I have been enjoying the Back To Work podcast with Merlin Mann for some time now and following the 13th episode, which I hope will not be the last one, I want to express some thoughts on some of the issues raised and covered in it, to be specific I want to talk about four topics: Advice, Inspiration, Honesty and Productivity. I am mostly going to talk about Merlin Mann because I think he is the heart of the show, with no offense to Dan Benjamin being the great host

On Advice.

When I think of good advice I have ever received or about good advice in general, I think my biggest criteria would be its timeliness. When you are at G and you want to get to H, last thing you can use is advice on how to get from A to B (cause I already know that) or from M to N (cause I have no idea what you are talking about). Even though Merlin really tries to make the show useful for everyone, I think most of the advice is really for people who are currently trying to get something started, are stuck somewhere or on something or are beating themselves over the head with doubt on whether they are doing the right thing or making enough progress and all sort of similar neurotic thoughts you could and should really do without. I liked when on one of the shows Merlin mentioned his wife telling him that most of 'those' people really need a hug and some therapy rather then another index card management system. I totally agree with that.

The types of people who should not listen to the podcast include those who are not doing anything and are happy as it is and those who are already happily doing something and making great stuff on their own. These types do just fine with regular porn, they don't need productivity porn in their lives as well. For years I was in fallacy of nagging people who are just happy with the ways things are either doing something or doing nothing with all sorts of helpful advice and was extremely irritated when they told me to buzz the fuck off. I totally get it now.

In some way, all Merlin is saying is to stop listening to him because he does not have any real answers. This, in a way, is the highest level of mastery, if you get this kind of thing. To those who are doing nothing, he says, stop listening and go do something, only when you start doing anything you will get what I am talking about. To those who are really doing great he says stop listening to me and do what ever that is you do. And to those who are 'trying' to do things (as himself) he says, listen I don't know what you should do and the most I can do is either tell you what someone else did or what you could do. But never what you should do or what you will do, this is your job, pal, to figure out. Which brings us to the second topic.

On Inspiration

In a way Merlin has put it brilliantly in the episode on inspiration by saying that you can not really make something out of inspiration alone. At most, it can serve as an initial push, after which you will still have to do this clackity noise, regardless. And do you really need an inspiration at all, it might, after all, be highly overrated. I really like Seth Godin's book called The Dip in this regard, and the metaphor is provides. You see this tiny hump in the beginning, you are lucky if the inspiration get you there but all the rest of the way is still yours to cover. If you want to go to a distant place, inspiration might get you out of the door, but you still have to work out through the entire way. And do you really need inspiration at all, just get out with or without it and this is what I think Merlin is trying to tell you there.

On Honesty

It is clear the Merlin Mann has decided on being absolutely and brutally honest in everything he does and talks about whether it works for him or not. He is not trying to make himself a guru or claim originality in any way on things that do not belong him. One of the commenters on the latest show called him a hypocrite for preaching the stuff he does not practice and I think this is a wrong way to go, not because I want to defend Merlin in any way, but because life does not work this way ever and you really should not punish the person telling you the truth, disappointing as it may sound. I could have said that you do not ask your dentist to open your mouth before the visit or for your shrink to tell you about himself first, but I really don't want to go there. Rather, having read a shitload of self help books over the years, what Merlin Mann has to offer deep inside is much much much more valuable that any of that crap if you only listen carefully. If you really listen in to this, you will realize that this is probably the most honest thing you will ever hear on this subject from a guy you basically don't even know on the internet, so treasure it.

On Productivity

In the morning my wife, who is building this toy house model from Styrofoam for a theater production, told me that she just discovered that if you add some color to the otherwise transparent glue you can work much faster because you immediately see which area is already covered with glue and which is not. This allowed her to improve her productivity twofold at least, but since it was the first time she did this kind of job she only discovered it in the middle of her work. If I knew this earlier, she said, it would be done even faster.

That is true, of course, but how much earlier. Imagine someone walking up to you in a street when you are like twelve and saying listen kid, you better mix some color into your glue, that would be the moment that you run like hell and ask the nearest adult you trust to dial nine one one. To sum this up I will paraphrase an famous quote by saying: "Productivity favors the prepared".

MZ.

Locked in Pandora's Box

It is coming up here and there, and I can totally relate, the idea that simplistic customization and adaptivity in information services can lead to you only seeing the same limited set of stuff you have been established to like. This topic is covered in the recent TED talk by Eli Pariser, who calls it 'personalization bubbles'.

One of the simple ways to demonstrate this is through Pandora, the best music radio site ever which I absolutely adore and which allows you to play music you like not by genre or artist but by its musical characteristics and does this extremely well. If you pick up some non mainstream music you can vote songs up and down until after a certain period of time you will be looping through a list of ten of ten songs or so.

This does have interesting implications in other areas as well, certainly when it goes into even more polarized domains such as, interestingly, politics.

Extraordinary as usual

Merlin Mann is being absolutely awesome.

If you feel down, unmotivated or just could use a boost of good spirit I would recommend this podcast, and this presentation (slides) from Webstock 2011.

Enjoy

Introducing Familinearity

Familinearity is when a person makes decisions in a new situation based on some other familiar situation while assuming that the relation between them is linear. This is also how software development planning is done in most companies.

The idea for this term came to me last Wednesday, when I have attended the FogBugz World Tour in Tel Aviv to finally see Joel in person and also learn a bit about version control and bug management. Both the organization of the event and the presentations were excellent and enjoyable completely making up for the apparent dryness of the subject. Even though I do not use FogBugz myself, I got an impression that it is an excellent and very powerful product, well integrated with version control systems and easy enough to use in both medium scale and large projects.

One feature in FogBugz that really caught my attention, however, was the Evidence Based Scheduling (EBS). The idea behind it was to match actual time of task performance to the estimated one for each developer over some period of time and then statistically process it to derive meaningful information about project planning and scheduling. Such information could, for example, include the developer timelines and velocity as well as estimated completion times that would allow you to determine a completion date for any subset of the project with definite probability.

A priori, I am extremely skeptical about the overall precision and usefulness of such methods. Not to say that evidence based approach in general is incorrect, it is certainly in fashion and gaining speed in medical community while also taking shots at other areas. It is other considerations, both practical and theoretical nature that cause my doubts and I will explain them in brief.

The practical concerns for the applicability of this tool are too long for this post, probably too long an entire book. Just a quick look at the article called "How to estimate software tasks" from FogBugz documentation can give a great example of the problems we face in this area. One paragraph towards the end of the article claims that giving people unrealistic schedules and hope they will be "motivated" by them is "brain-dead", a statement with which I completely agree. It is still a fact, however, that there are entire companies that live by this principle and lots of managers who did not read "A Mythical Man-Month" even though it is, as the article claims, a requirement. Joel Spolsky himself, of course, knows more than most people about those problems and have written numerous articles on the subject. These companies, by the way, can at times be very successful, and you will have a hard time trying to convince them that their management practices are flawed.

My point, in this case is that when a tool like EBS is deployed in an organization that has that specific management culture and later, one of the team leaders comes up to his boss to claim that based on previous experience his estimate has only 2% chance of being correct, it will more likely result in fireworks than in the manager amending his ways. Another problematic situation might occur when the manager will come to the above team leader and inquire why the feature that was a week ago 99% on time still late. All that, assuming the organization really deployed EBS and paid all the additional costs of constantly updating all estimates and drilling down all tasks without which the method will not work at all at best or produce highly inaccurate results at worst.

But even if we leave the practicalities aside, the theoretical aspects of using bootstrapping based statistical methods seems a bit problematic. As Joel himself would tell you, in software you rarely do the same things twice. It is design, and design is similar to an exploration of a highly fluid terrain, with high complexity. Using methods that use linear extrapolation based on precision of your previous estimates to provide precise answers seems at least problematic and might raise similar objections to the ones Nassim Taleb provided for the use of VAR formula in economics.

I am not saying that this tool is not useful, it might for example, reveal the "epistemic arrogance" of the management by showing the ill precision of their previous estimates and thus benefit tired and demotivated programmers. It might provide a good bottom margin for the prediction errors. But not taking into account the complexity and non linearity of the domain at hand it could hardly offset the existing "familinerarity" of the software development planning process.

Software Development and Complexity Theory

Some time ago, I have become interested in Complexity Theory. I do not remember the exact path that lead me to it, but as usual it was probably one thing dragging another until one of the links I clicked brought me to it and I was hooked.

Complexity Theory is a rather broad field of knowledge involving many different scientific areas and disciplines. It is far from coherent and complete, even some definitions of what complexity really means and how it can be measured are still being debated.

Once I have started reading on the subject, I have immediately become interested, among other things, in applications of Complexity Theory to software development, management and creativity. To understand how these fields might be connected, lets start with the following diagram taken from the Cynefin framework, created by Dave Snowden and his company "Cognitive Edge".



This diagram describes a framework for thinking about systems in four different categories: Simple, Complicated, Chaotic and Complex.

An example of a simple system is an old bicycle, the input of force to the pedal is transformed into a wheel rotation in an obvious and predictable way. A complicated system is a watch, which has a much less obvious structure but is still linear and predictable. An example of chaotic system is, say, gas molecules in a closed volume. We can not predict the movement of each individual molecule but we can statistically predict their behavior in general.

Complex systems, however, are something entirely different. They are non linear and dynamic, meaning that the same input may sometimes result in entirely different output. They are, thus, inherently unpredictable and can not be controlled directly requiring an entirely different type thinking and a new set of tools to deal with. (I will talk much more in next posts but for for now if you want more info please read the Wikipedia entry and watch this video of Dave Snowden explaining this concept and other related ideas)

This is all of course a monstrous oversimplification, for a great introduction I would recommend an excellent book "Complexity: A Guided Tour" by Melanie Mitchell.

What I will talk about here however, is specifically the implications of complexity in software development. While most software is complicated, the software development process however is complex. And this has a great impact on how the software development process should be organized and managed.

As obvious as this may sound, far too many managers in software companies have little or no understanding of this fact and keep trying to construct complicated rigid processes that should cover every little detail in the work flow, or use outdated methodologies that were originated in the large factories of the early industrial era. Since measuring complex systems in general, and software development process in particular is extremely difficult if not impossible they usually get away with it and move on to another project by blaming the failure on some external circumstance that is always out there.

I am convinced, however, that proper understanding of complexity issues and their impact on software development can improve the process as well as the resulting software and is important to study and understand. To kick off this track that I am hoping to continue researching in the near future I would really like to recommend this terrific presentation by Jurgen Appelo. It might sound a bit complicated for a beginner, but it is spot on, covering many issues and providing extensive list of references for future research and discussion.

Possibly the best (worst) poker beat I have ever seen

I have always thought that a Poker is a nice simplistic model of our reality, but this one beat is just plain awesome!

Prices of eBooks

The eBook reader market is on the rise. I have just got a brand new Sony Reader 600, and the Nook from Barnes & Noble, due in November, also looks fantastic. The prices of eBooks however, are still way to high, and for no apparent reason. After all, conventional logic shows that if you do not have to print, ship and store the book, it should make it much cheaper for the publishers. You can sell any amount of books and never run out of stock. You also don't take the risk of getting stuck with shelves of unsold books nobody wants. You do not need to rent a store, pay the cleaning lady and the clerk and the driver and the supervisor. Doesn't all this amount to something?

In order to understand why the prices are so high, I had to undertake a little investigation (read few blogs and do few searches). I have found several "logical" explanations why an eBook should cost 9.90 (as they mostly do) and why not printing the book does not really affect the cost too much. Some of the authors go to great length and provide some interesting numbers.

This guy, for example, shows a detailed breakdown of a book cost. He argues that this shows that printing is actually a minor part of expenses. And he is not alone. Here is another guy who is wrong and another. Let's take a look at this and see what is wrong with this picture.

"Based on a list price of $27.95

$3.55 - Pre-production - This amount covers editors, graphic designers, and the like
$2.83 - Printing - Ink, glue, paper, etc
$2.00 - Marketing - Book tour, NYT Book Review ad, printing and shipping galleys to journalists
$2.80 - Wholesaler - The take of the middlemen who handle distribution for publishers
$4.19 - Author Royalties - A bestseller like (John) Grisham will net about 15% in royalties, lesser known authors get less. Also the author will be paying a slice of this pie piece to his agent, publicist, etc.

This leaves $12.58, Money magazine calls this the profit margin for the retailer, however, when was the last time you saw a bestselling novel sold at its cover price."

He then suggests to count the costs from zero up, and suggesting that pre production, marketing and author fees are essentially the same, 9.90 is actually pretty cheap even for an electronic edition.

Everybody got that? Now let's see why this should clearly not be the case. In order to illustrate the point I will take Malcolm Gladwell as an author, and his three books, "The Tipping Point", "Blink" and "Outliers", last one published in 2009 and has a list price on Amazon of 27.99

First let's take a look at marketing. I first saw Malcolm at TED for free. Cause you see, where I live, there are no book tours, no New York Times and no New Yorker. So all that money spent on that advertising, for me, is money wasted. Now let's note, that Malcolm Gladwells first book sold about 2.5 million copies, which according to the above breakdown should amount to 5 million bucks in ads alone.

If this sounds strange, check this out. According to the above logic of breaking the book cost down to it's components, the cover editors and graphic designers of the above book earned little under 9 million dollars. Wow, I must be in the wrong business. Not to mention the author, who (once again according to the above calculation) got 10 million dollars in royalties.

All this leads me to the conclusion that the above breakdown, is a little bit problematic. The costs of producing, storing and selling books online, are close to nothing, taking into consideration the advantages of online distribution such as:

• Instant access to any book (30 seconds from seeing Malcolm on TED to buying his book on Amazon).
• No need to wait for delivery, start reading immediately.
• No forests are hurt in the process
• Availability of recommendation engines save marketing costs.
  
• Vast worldwide markets, not covered by paper book distribution.
  

So to conclude, since there are almost no necessary costs, save for author royalties, a fair price for an average eBook is 1.99$ US. What should be understood that in a modern networked world there should be an author and a reader and everything in between is irrelevant and not interesting. Period.

New Apple Mouse Looks Amazing

If you had not already go to apple.com and marvel the new Magic Mouse. What they basically did is put their superior mouse pad that was used in laptops on a regular mouse thus creating an all touch interface that supports gestures in combination with a regular convenient point and click experience.

The "wow" effect is guaranteed and it looks really slick and amazing.

PS: Just to balance things, they did a much poorer job on the new Apple Remote.

Michael Moore Movies Free Online

I have seen some interviews with Michael Moore on YouTube recently. I have heard about him before but only recently due to his Capitalism movie I have taken the time to study his work more closely.

The very first thing that struck me is why are not all of his movies available for direct download online and for free, I mean if he is such a democrat. After all, he has made a lot of money from those movies already, covering his costs many times over. What's the matter? Maybe he is greedy too? Does he really want to spread his ideas or just make some more bucks on the other part of our fears, the ones that has not been yet exploited by the government.

The only thing I found immediately is his movie "Slacker Uprising" and it was only available for download after you supply and email (what for?) and for citizens of US and Canada.

Well I guess it is much easier to talk about democracy and freedom than really practicing it.

Windows 7 Annoyances: Aero Peek

Are you annoyed by the Windows 7 Alt + Tab hiding your windows and behaving weirdly, want to go back to the old way, go to System Properties -> Advanced -> Performance Options and disable "Aero Peek" option. Downside, this will also disable full window preview from of multiple windows from the taskbar. Make your choice.

Great Presentation on Positive Psychology and Happiness

During my recent searches I have stumbled upon some lectures by Tal Ben Shahar, a psychology professor from Harvard, who represents a movement called "Positive Psychology" and human happiness in general. Besides the general interestingness of the topic itself, I mean, who wouldn't want to be happy, these lectures are good because they are both scientific and very well presented.

As you know, I treasure great presentations and presenters and this is by all means a great example of such. So whether you want to be happier or just improve your presentation skills, enjoy!

http://forum.wgbh.org/lecture/positive-psychology-science-happiness

http://in.truveo.com/positive-psychology-lecture-1-tal-benshahar/id/2472580271

Data Preservation Instinct

If you would sit down to write something meaningful, personal and long lasting, not real memoirs, but something you would definitely want to review some time later, where would you write it. From one side you would definitely want it to be organized, structured and searchable, in other words digital, but on the other hand you would not want it to be on that hard drive that just failed or that nice site that closed recently or that network storage provider who went under and took all your precious data with you.

And what editor would you launch? I use a Mac now, so I have launched the Omni Outliner Pro, the only software I have bought out of sense of gratitude to its creators. However, I will only use it for editing the immediate stuff and initial organizing. I will not be able to store my materials here indefinitely, because it is not portable enough, so I can not use it if I move to another operating system like Linux or Windows. I can not count on it for sustaining over several (or many) years. I will not provide enough organization for large amounts of materials etc. etc.

The same is true for almost any other commercial editor, Word or whatever. Their file formats are very problematic, usually combining both form and content together and having a few years life span. Yes you would be able to open older ones (in most cases) but you will usually have to upgrade and convert your older files, especially if you have used some formatting. And if, god forbid, you would want to change to another environment then you will have real problems on your hands.

You would really want it to be there when you need it in three or five years. Editable, with the structure, the keywords and the formatting. But achieving this, seemingly obvious, goal with current technology is very difficult and required vast technological expertise. You can not really count on anything, these days, not even file dates can be reliably preserved over time. Not to mention any other metadata. Say you invested the time to add some comments to a file in Spotlight, copy it to another machine via a FAT partitioned drive, they are gone, this is just an example.

Even though, it costs absolutely nothing to copy digital data, the fact that saddens some content publishers, the volatility of such data is so high that preserving it usually costs much more in the long run, especially if you really care about it. This is another in the long list of reasons, why most of the existing models of digital data consumption are obsolete. You could have a few thousand dollars worth of music on your iPod only to loose it in a second to a toilet bowl. But wait, the music is still available there on iTiunes and you already paid for it once, so why can't you get it back. At least with software you pay for the license and then can always get the software itself even if the original disks got damaged. What did you really buy when you paid for those thousands of songs?

Fast forward to a perfect world of tomorrow, an international standard for a database backed file system with secure and transparent distributed backup and universal encoding is available and all the problems described above are history. What would it take to get there short of a miracle.

Recently I had a conversation with a friend, about the state of user data backup today. In fact the only OS coming with any decent usable tool for this purpose is the latest OS X, Leopard which has a reasonably easy to use utility called Time Machine. My friend told me that most of the users did not need any such backup, since they did not actually produce any substantial materials on a regular basis, at least not till the development of the personal digital photography. So till fairly recently, there was not real need for sophisticated backup and data structuring systems, at least not at the demand level that would justify the creation of one. Even today, when hard drives are very cheap, I keep seeing people loosing their entire collections of irreplaceable family photos in a second.

Well, I think that now, as our creativity in the digital world rises and more people produce their own digital artifacts we must think of keeping our creations in the digital world at least as safe as they would be in the real one, and even safer.

Spotlight Tip

Ever wondered how to get from the list of results in Spotlight to their actual location in Finder. Turns out holding 'Command' down and clicking it does the trick.

Untitled

Windows 7 is so ugly, that I often go to "Adjust for best appearance" setting to make sure it is still on.

We are the Borg

Just started playing with Python, after C++ and Java it looks... refreshing. Anyway, it's a nice thing to see how most design patterns become obsolete with little use of Python wizardry. Once thing I particularly liked is the "Borg" pattern, which replaces our good old Singleton. Read here for details, very cool.

Fudge

An interesting site I would like to explore is Fudge, a social network for programmers.