Learning from uncertainty

It seems easy to know what caused any failure afterward. At least it is easy to come up with theories of what have caused any given failure. Problem is that there is often no way of verifying the causes. Predicting into the future just seems impossible.

In medicine we have clinical trials where drugs are given under very controlled conditions, some patients even getting a placebo as replacement for the real drug. In IT this would be difficult to arrange.

IT is a field with lots of uncertainty involved. Some things will work sometimes, but not always. A successful setup in one project may lead to a big time failure in the next. A setup failing in one project may work perfectly in the next. There is no way that you can beforehand know if you will have positive or negative surprises and if they will impact your endeavor at all.

Only fool proof way to safeguard against unpredictability seems to be to progress in small scale steps, failing early and not in an expensive way. If we are allowed to try enough times we will eventually succeed.

There is an excellent Tech Nation interview with Nassim Nicholas Taleb dealing with unpredictability and impact on business available at ITConversations, go check it out.

Learning efficiently from failures

There does not seem to be very many lessons to be learned from sucsessful projects according to an article in IEEE Spectrum. From failures on the other hand you can always learn a lot.

Being efficient, you should at least try to learn from your own mistakes. On an organizational level it would make sense if everybody was able to learn from all mistakes done in the whole organisation. On a global level it would make sense to try to learn from all mistakes evere made. Access to information is however limited, and there is probably not enough time to get familiar with all failures in your domain.

If you cover up the mistakes, chanses are that somebody will have to experience the same problems again. Opening up your experiences for other to learn from can help others in your organisation to become more sucessful. Getting comments and feedback from others will further more make you see more clearly what the the problem was and even come up with better solutions to avoid similar problems in the future.

However, nobody likes to admit failure. I think organisations must become more able to accept and tolerate small falures, and start to encourage people to learn from failure in stead of covering up. If failures are detected and admitted at an early stage there is time to prevent the big failure that probably is lying ahead. Big failures are anyway difficult or impossible to cover up.

Efficient frameworks

Using complex frameworks are often motivated by increased efficiency. Some things are however inevitably complex.

Real time thread synchronization across processes and machines is for example hard. You have to avoid dead locks, make sure that your processes yields execute threads in a predictable manner and guarantees execution of all tasks on a given time. Getting all this correct is not easy, not even for experienced developers.

Frameworks are motivated that they make hard things more simple. A common problem is however that they only enable you to shoot yourself in the foot more efficiently, which is most often not what you intended to start with. In order to utilize the framework efficiently you need a thorough understanding of the underlying problem domain and also be able to solve the problem in terms compatible with the tools and possibilities the framework gives you.

Most frameworks are not developed with your problem in mind and general purpose frameworks are most often too generic to be useful. If your need exceed what is intended with the framework of choice, you end up adding more complexity than you would need without the framework.

Complex tings can be made to work, no doubt. For one time fixes, in order to get something working in a quick and dirty fashion the additional complexity does not matter very much.It is just a one time penalty to pay. The trouble starts when modifications are needed, since somebody must either remember or figure out how it works, probably from the source code, which may or may not conform to any pattern or convention.

People in general tend to fall in love with their framework of choice, If the framework suits your every need, there is a benefit if all developers on the project like working with the framework. However, if the framework does not fit the problem you are going to have a tough time ahead.

Crippled by design

Designing optimal solutions is difficult. Settling for a working solution is every day life, but most IT solutions seem to get seriously crippled in one or more aspects at some point in time. The system can be difficult to use, costs of development can sky-rocket, it can be impossible to maintain, it can have limits in scalability. But even crippled software can be useful.

There is probably no single one and only explanation why this crippling is happening. I'm sure that many causes combine. Many of the problems giving the system a crippled appearance can be fixed, given time and resources, but there always seems to be new issues popping up, like peels on an onion. Some times a crippled system delivered today is better than the most optimal system delivered too late.

Generally it is said that it is cheaper to fix a problem that would cripple a system early in the design phase. The problem with this is that early you don't necessarily know every aspect that will appear crippled, even how thoroughly you look at your design. It is just no possibility to know every aspect of how the system will appear before it is actually deployed into production. There is some value to the design up front approach, but on a too detailed level it is doomed to result in analysis paralysis.

When designing software and IT systems there are more degrees of freedom than can be perceived. You can implement a system in many, very many different ways. Most ways of implementation will not differ very much from each other when done, and most will have no influence on the end result, but each possible equal implementation can be given a most important role due to decisions made very much later on. Reversing the latest decision can sometimes only bring negative side effects. Going back and undoing the faulty decisions, made long ago, is not always the most pleasant thing in a normal project setting, but in agile projects doing refactoring this is actually one aspect that in my opinion makes agile approaches better, if only the faulty decision is found.

Real clear failures are just as rare as optimal solutions. Mostly the reason for failure is that projects run out of money. With agile approach there is a natural way of closing down unsuccessful development work. Perhaps we should be ready to scrap and abort more traditional software projects also that are showing signs of crippledness. I think that even the most severely crippled IT system has some value in that it can teach us a lesson perhaps enabling us to doing better next time. The most important thing to remember is that it is your actions as a designer, either the ones that you do or the actions that you omit doing that is going to make or break the system. There is no escaping, you are responsible for your design decisions.

Measurement and feedback

The preoccupation with measuring every aspect of IT efficiency seems to be ever increasing. It seems like there is no limit to what piles of rubbish management and their followers are trying to collect in order to find the one measure that will tell them how their organisation or department is doing.

The problem is that there are lots of things you can measure, and some may even tell you something about the actual performance, but most measurement seems to be a waste of time. Some measures are too abstract to be of any use, like "rate of success". Others are just too detailed to be useful, like lines of code written per day.

In my opinion you can spend your time better by arranging for your teams to have feedback loops in stead of figuring out new and better measures. The shorter the feedback loops the better, since if things change quickly you need to react quickly. With sufficient feedback you can be fairly sure that you are doing the right thing, or else you will know about it quickly. There are numerous examples from nature where feedback loops regulate and interact in order to steer processes and their outcome, examples ranging from metabolism to neural activity.

One of the keys here is that the sensors in nature are integral parts of the process like for example proteins catalysing the conversion of one substance to an otherbeing inhibited by the end product and stimulated by the raw material. The centralized control seems to be kept to a minimum, and only exercised as a means of giving over all direction, so why not mimic the same in IT?

Empowering everyone to make sane decisions based on feedback, or even listening to feedback at all must feel threatening to management accustomed to safely measuring and them self interpreting the result. However, if what counts is the end result and over all efficiency, the score seems clear.

Cost of features

Microsoft seems to be hitting their head in a wall with the new Vista content protection according to a very interesting report by Peter Gutmann.

If the actual implications are as fatal as what is stated remains open to see, but a conclusion we can make at this point is that not all end results can be known beforehand when designing an extremely complex piece of software like for example Vista.

There is no way of knowing how the software/hardware ecosystem will be affected by a gigantic release with lots of new unknown features/traits. Transparency can help, by making the design and implications visible early, but there is naturally no guarantee that anybody will spot the problem before it is too late. Also, afterwards the problem must be acknowledged and corrected, requiring a fair degree of openness.

This time it seems like bending far over in one direction has resulted in a large invasion of problems from an other unwelcome direction. It is very unfortunate and disturbing that the focus does not seem to be set on making real customers happy, but more on satisfying digital rights owners unspecific dreams.

Openness, transparency and efficiency in IT

In Open Source - More Than Code Bill Barr discuss if you really can run IT efficiently while maintaining full transparency to stakeholders. I have also been looking at the effects of transparency.

The spirit of SOX for example seem to require that business have full visibility into everything going on, down to a very technical level. In my opinion business should. First of all, they are paying for the IT systems and their development. Secondly, the more involved they become the bigger is the opportunity that the IT system serves the business need. Following SOX and other rules can then just be seen as another minor reason for driving transparency. Extra paperwork and meetings is often seen as a main drawback, and with the sad state of documentation I have seen on a number of cases this is most certainly true, but not in my opinion valid, since there is usually some value in improving documentation anyway.

The problem with transparency and involvement starts when business stakeholders interpret this as a mandate to lead and manage the IT development. This often result in micromanagement, having everything grind to a halt. The problems may be small or big, but in the end they all depend on the personalities and driving forces of the stakeholders you are working with. Keeping these stakeholders in the dark is often seen as a solution, but that can create even more serious antagonism further down the road. I think that a better solution is to make them aware of other stakeholder views and needs, in terms of costs and business risks, and the fact that they may not necessarily have all the facts or the knowledge required.

I would not say that transparency or openness need to be in conflict with efficiency, but there is a good chance that they will be. In a deep hierarchical "chain of command" type organization where departmental silos are not discussing with each other I think you can have either openness, transparency or efficiency. Achieving any two in any combination would come at an extremely high price, and all three is probably next to impossible achieving.

That said, I still consider involvement of individuals, openness for change and transparency to what is being done as the key to efficiency. The individuals in a lean organization, once set with a task, may well know how to achieve a solution as long as the organization is truly open to this kind of involvement. Transparency guarantees that if some initial condition change or some theory is later found wrong, there are plenty of opportunity spotting and correcting the error. The key i think is that everybody need to agree on what the actual problem that is to be solved is and listen to each other.

Improving efficiency is easy to if you don't need to consider any side effects. In the end, IT department efficiency should not be considered as the ultimate goal, and trying to optimize on a project or code development level can in worst case only reach a local optimum. Optimizing for business efficiency would be far better in many cases.