According to common sense it is simple to state that one thing is better than something else, at least as long as there is some aspect of the thing that can be measured. Stating that one approach to software engineering is better than an other does however not belong to the category of things that can be analysed in a simple way and does not seem to lend itself to any common sense approach. The problem is that there is not only one aspect to measure from which you can draw conclusions, but lots and lots of different aspects playing together, either making an approach good or bad. There are also lots and lots of "parameters" that can be changed and changing one thing will influence the others, even in unpredictable ways. Further more, changing one parameter any way possible may improve efficiency.
Theoretically there must exist a very best way to achieve something, but we don't know how good it is or where in the “parameter space” it is at. With any selected approach you can do better or worse, and finding the best way to archive a goal can be done by trial and error.
For example, by obtaining the latest development tools and making tool usage mandatory to a team of developers used to notepad / vi, delivery is delayed and amount of shipped bugs increase. The task for the team has become learning in stead of creating software. After the price is paid, and the team have done the learning and got used to the new tools the situation is totally different. Mundane tasks are automated by the tools, and tool specific quality assurance measures reduce the number of bugs. In this example you could say that by using notepad /vi the efficiency was trapped in a local maximum value. Getting out of this local maximum could have been difficult, but once done the efficiency increased. There are presumably many other things that could have been done, costing more or less, but before trying them out you can not really know if they would have worked better than the selected approach. There is neither any guarantee that the same approach will work on the next project.
Comparing to when scientists calculate the 3-D conformation of macro molecules like proteins and smaller molecules that bind to proteins, there is a simulation named Monte Carlo method being used, where you by repeatedly and randomly altering energy levels can get a good estimate of what the overall lowest energy conformation of the macro molecule system is. Repeated optimisation cycles and random alteration to the parameters will not guarantee that all possible conformations are evaluated but it does a pretty good job estimating the best conformation.
I don't know of any fully successful approach that would lend itself to study software engineering where you could experimentally or computational control the environment. We only do things once under normal conditions, and afterwards you have learned some more. We can't repeatedly create the same software over and over again altering just one factor and analysing the efficiency. In RAD Races you have different teams competing to create something working and useful, but the teams are constituted by individual members whose skill and motivation can vary by a large extent. Statistical approaches with many parties, contributing results over a long period of time could also be a good possibility, but getting objective and impartial data would be difficult. Further more the vast number of possible values for the vast number of parameters is way too large for any kind of exhaustive analysis.
If productivity is rising exponentially over time, it can be very difficult to pinpoint the most efficient approach. Getting stuck in a local optimum is very easy, and that is in my opinion the biggest problem for many organisations today.
A. A.
