Thursday, December 31, 2015

Return of test investment

During a recent discussion the question was raised on how to make investment in test automation pay itself back. How do we know its worth the effort? Implementing automated acceptance tests can require quite big effort.

Theoretically there are two alternatives to automated tests. You can either test manually or you can disregard testing altogether. Having no testing today should be considered a breach of ethics in IT. As a presented at USENIX OSDI 14 by Ding Yuan et. al. most catastrophic failures could be avoided by testing error handling without any understanding the underlying software design.

Manual testing is time consuming, subject to testing procedure errors and require lots of attention. As manual tests take time to execute, the manual test procedure can easily slow down the release deployment and cause delay. With a manually executed procedure the results will also not be as reliable as with the automated tests. The variation can naturally be beneficial, but most likely the test result may be compromised.

When calculating how much benefit automated testing can bring, the main consideration is often how much effort and time can be saved by the automation. Less emphasis is typically placed on reliability aspect. Freeing up testers time to do more thorough explorative testing must also be considered. Secondary effects are also of importance. Having automated tests cause code and features to become more easy to test. Having automated tests also enable faster releases. Taking secondary effects into account when calculating benefit of test automation can change the equation.

Additionally there is also the consideration on which side do you want your estimate to side, on the side of promoting improved testing or on the side of limiting quality...

Sunday, January 25, 2009

Null results

Presenting novel ideas is seen as valuable in our scientific culture today, but investigations of what the boundary conditions or side effects are more hard to find. Research programs are started to find and establish new theories, but not too much appear on under what conditions those teories does not apply any more.

Nassim Nicholas Taleb states it quite well: "you cannot do anything with knowledge unless you know where it stops, and the costs of using it".

In IT not much effort is put into understanding the down sides the "improvement" will have. One way of seeing this is that it is not in the interest of the IT staff or managers to know the limits, since that could mean less work since if side effects were well understood, the people making the investment may not be all that willing to back a number of projects any more.

Wednesday, December 31, 2008

Cost control

IT is not the only area where projects are in trouble according to an article in the January 2009 edition of Scientific American that arrived in my mailbox.

NASA also seems to have trouble with the Mars Science Laboratory project. According to the story costs were underestimated and unrealistic, and there seems to have been a lack of will to keep costs under control. In contrast to normal projects where you calculate a return of investment and you can find a limit to how much you should invest, i don't think it matters what the cost is in the case of the Mars mission.

With IT projects there is often no abort button easily at hand for projects going over budget. In NASA however i would guess that the situation is a bit different. When promising the moon, or Mars as in this case you may end up having to deliver at any cost. Wherther it will succeed to deliver on the promise or not we will have to wait and see.

Saturday, November 29, 2008

Understanding management

According to a study management is struggling to get their message across to the people working for them in their organization. Among the around 200 mid level managers interviewed 87 % feels that the strategy is not being understood. Obviously this cause problems for the organizations probably impacting even the profitability.

One other way of interpreting this is that 13 % of the managers use normal language in stead of the normal jargon filled talk that management consultants and managers use. The study however did not present any data from which we could guess if the managers themselves understood what was being communicated. Still another possibility is that the 13 % or more of the managers have misunderstood how their people understand them and just describe the worst.

Seems like this should be a good research subject for further inquiries. For improving efficiency of management this could be useful. However, I don't think that the managers will change the way they communicate, even if this was shown to be the cause.

Friday, October 31, 2008

Efficiency of bureaucracy

Under some conditions efficiency can be impossible to achieve. For example in an article by Dr Anthony Berglas one can find the surprising fact that costs for maintaining the australian tax office hasn't changed in terms of percent of the GDP per year over the last 50 years. Sure the tax office nowadays have more computer systems. This automatization has however just increased the complexity of the tax process, not decreased the amount of human effort. 50 years ago the regulations governing taxation was one order of magnitude shorter. Same seems to have happened with banking.

I find it good to know that even bureaucracy cant grow without bounds. For a tax office budget, anything below 2% of GDP seems to be acceptable to society, and the organization will try to grow until that limit. Growing would mean adding complexity to the bureaucracy, and hence also complexity is limited. Without a limit on the complexity or too much money to spend, there will be real big failures ahead.

Tuesday, September 30, 2008

Weekly dose of IT related problems

According to news here in Finland every employee uses 53 minutes every week on average for solving IT related problems. Taking into account that some of us do this for work, and others even seems to work to create problems with IT i'm surprised the figure is not bigger.

The figure comes from research commissioned by Microsoft and performed by TNS Gallup in small or mid size finnish companies. The original press release can be found here, if you are interested and know finnish.

What the press release brings forward as noteworthy is that of the 22 % responders saying that they experience lots of IT related problems, being among the smallest and newest companies around, e-mail and office programs was causing most of the problems... The research however fails to state how many of those experiencing the problems use Microsoft products already or if there is any correlation.

One hour a week is reported to be a significant number. In Finnish small medium size companies that hour translates into 340 000 man days wasted per month as reported in the press release. That number seems already big and frightening, but in reality it is perhaps not all that much. 53 minutes every 5 days is 636 seconds per day, or between 10 and 11 minutes a day, that does not sound that much, or? Comparing 636 seconds to the 27 000 seconds spent at work every day is not a big deal, its just 2,4 % of the time.

Lets try to verify the calculation... Research stated that IT problems causes people to user 53 minutes every 5 days, fair enough, that would be 636 seconds per day. Assuming that every month has 21.5 working days, in reality it will vary between 19 and 23, depending on month and what kind of days off we have during a month... So, if we take 636 seconds, times 21,5 days per mont and multiplies that with 717 000 people working in the small medium size companies according to the source mentioned in the article, we get 9 804 258 000 seconds wasted per month, which sounds big. if we divide by 60 to get minutes, and further by 60 to get hours and again with 7,5 to get days i arrive at around 363 000 days wasted per month. Well probably the researcher calculated this some other assumption, perhaps taking vacations or holidays into account... Figure is well in the same ballpark.

I guess this could be compared to overlong coffee breaks, lunch breaks and other breaks with the difference that experiencing IT problems is not fun. A motivating work atmosphere was however mentioned as the most important factor making people efficient in their job for 95 %. Perhaps having long coffee breaks and lunch breaks or whatever gets the motivation into the work place is what we should consider. I guess workers simply compensate for the time taken by IT problems to some extent.
The research does not state how much of that time is unpaid overtime. Im however somehow doubting that Microsoft selling more Exchange or Office would decrease the time spent dealing with IT related problems in small or midsize companies. I feel the real issue here is the frustration and demotivation experiencing IT problems is having, not the time spent as such.

Sunday, August 31, 2008

Estimating risk

Humans are not very good at estimating risks. Our life seems to be more focused on estimating medium size risks such that we need to avoid every day. Small risks can be seen as unproportionally big, or big risks can be seen as negligible.

Risk of events occurring more or less frequent is seen out of proportion. A risk occurring with low frequency will however be common if the number of possible situations where the risk can occur is very big.