Using complex frameworks are often motivated by increased efficiency. Some things are however inevitably complex.
Real time thread synchronization across processes and machines is for example hard. You have to avoid dead locks, make sure that your processes yields execute threads in a predictable manner and guarantees execution of all tasks on a given time. Getting all this correct is not easy, not even for experienced developers.
Frameworks are motivated that they make hard things more simple. A common problem is however that they only enable you to shoot yourself in the foot more efficiently, which is most often not what you intended to start with. In order to utilize the framework efficiently you need a thorough understanding of the underlying problem domain and also be able to solve the problem in terms compatible with the tools and possibilities the framework gives you.
Most frameworks are not developed with your problem in mind and general purpose frameworks are most often too generic to be useful. If your need exceed what is intended with the framework of choice, you end up adding more complexity than you would need without the framework.
Complex tings can be made to work, no doubt. For one time fixes, in order to get something working in a quick and dirty fashion the additional complexity does not matter very much.It is just a one time penalty to pay. The trouble starts when modifications are needed, since somebody must either remember or figure out how it works, probably from the source code, which may or may not conform to any pattern or convention.
People in general tend to fall in love with their framework of choice, If the framework suits your every need, there is a benefit if all developers on the project like working with the framework. However, if the framework does not fit the problem you are going to have a tough time ahead.
Friday, March 30, 2007
Monday, February 12, 2007
Crippled by design
Designing optimal solutions is difficult. Settling for a working solution is every day life, but most IT solutions seem to get seriously crippled in one or more aspects at some point in time. The system can be difficult to use, costs of development can sky-rocket, it can be impossible to maintain, it can have limits in scalability. But even crippled software can be useful.
There is probably no single one and only explanation why this crippling is happening. I'm sure that many causes combine. Many of the problems giving the system a crippled appearance can be fixed, given time and resources, but there always seems to be new issues popping up, like peels on an onion. Some times a crippled system delivered today is better than the most optimal system delivered too late.
Generally it is said that it is cheaper to fix a problem that would cripple a system early in the design phase. The problem with this is that early you don't necessarily know every aspect that will appear crippled, even how thoroughly you look at your design. It is just no possibility to know every aspect of how the system will appear before it is actually deployed into production. There is some value to the design up front approach, but on a too detailed level it is doomed to result in analysis paralysis.
When designing software and IT systems there are more degrees of freedom than can be perceived. You can implement a system in many, very many different ways. Most ways of implementation will not differ very much from each other when done, and most will have no influence on the end result, but each possible equal implementation can be given a most important role due to decisions made very much later on. Reversing the latest decision can sometimes only bring negative side effects. Going back and undoing the faulty decisions, made long ago, is not always the most pleasant thing in a normal project setting, but in agile projects doing refactoring this is actually one aspect that in my opinion makes agile approaches better, if only the faulty decision is found.
Real clear failures are just as rare as optimal solutions. Mostly the reason for failure is that projects run out of money. With agile approach there is a natural way of closing down unsuccessful development work. Perhaps we should be ready to scrap and abort more traditional software projects also that are showing signs of crippledness. I think that even the most severely crippled IT system has some value in that it can teach us a lesson perhaps enabling us to doing better next time. The most important thing to remember is that it is your actions as a designer, either the ones that you do or the actions that you omit doing that is going to make or break the system. There is no escaping, you are responsible for your design decisions.
There is probably no single one and only explanation why this crippling is happening. I'm sure that many causes combine. Many of the problems giving the system a crippled appearance can be fixed, given time and resources, but there always seems to be new issues popping up, like peels on an onion. Some times a crippled system delivered today is better than the most optimal system delivered too late.
Generally it is said that it is cheaper to fix a problem that would cripple a system early in the design phase. The problem with this is that early you don't necessarily know every aspect that will appear crippled, even how thoroughly you look at your design. It is just no possibility to know every aspect of how the system will appear before it is actually deployed into production. There is some value to the design up front approach, but on a too detailed level it is doomed to result in analysis paralysis.
When designing software and IT systems there are more degrees of freedom than can be perceived. You can implement a system in many, very many different ways. Most ways of implementation will not differ very much from each other when done, and most will have no influence on the end result, but each possible equal implementation can be given a most important role due to decisions made very much later on. Reversing the latest decision can sometimes only bring negative side effects. Going back and undoing the faulty decisions, made long ago, is not always the most pleasant thing in a normal project setting, but in agile projects doing refactoring this is actually one aspect that in my opinion makes agile approaches better, if only the faulty decision is found.
Real clear failures are just as rare as optimal solutions. Mostly the reason for failure is that projects run out of money. With agile approach there is a natural way of closing down unsuccessful development work. Perhaps we should be ready to scrap and abort more traditional software projects also that are showing signs of crippledness. I think that even the most severely crippled IT system has some value in that it can teach us a lesson perhaps enabling us to doing better next time. The most important thing to remember is that it is your actions as a designer, either the ones that you do or the actions that you omit doing that is going to make or break the system. There is no escaping, you are responsible for your design decisions.
Tuesday, January 23, 2007
Measurement and feedback
The preoccupation with measuring every aspect of IT efficiency seems to be ever increasing. It seems like there is no limit to what piles of rubbish management and their followers are trying to collect in order to find the one measure that will tell them how their organisation or department is doing.
The problem is that there are lots of things you can measure, and some may even tell you something about the actual performance, but most measurement seems to be a waste of time. Some measures are too abstract to be of any use, like "rate of success". Others are just too detailed to be useful, like lines of code written per day.
In my opinion you can spend your time better by arranging for your teams to have feedback loops in stead of figuring out new and better measures. The shorter the feedback loops the better, since if things change quickly you need to react quickly. With sufficient feedback you can be fairly sure that you are doing the right thing, or else you will know about it quickly. There are numerous examples from nature where feedback loops regulate and interact in order to steer processes and their outcome, examples ranging from metabolism to neural activity.
One of the keys here is that the sensors in nature are integral parts of the process like for example proteins catalysing the conversion of one substance to an otherbeing inhibited by the end product and stimulated by the raw material. The centralized control seems to be kept to a minimum, and only exercised as a means of giving over all direction, so why not mimic the same in IT?
Empowering everyone to make sane decisions based on feedback, or even listening to feedback at all must feel threatening to management accustomed to safely measuring and them self interpreting the result. However, if what counts is the end result and over all efficiency, the score seems clear.
The problem is that there are lots of things you can measure, and some may even tell you something about the actual performance, but most measurement seems to be a waste of time. Some measures are too abstract to be of any use, like "rate of success". Others are just too detailed to be useful, like lines of code written per day.
In my opinion you can spend your time better by arranging for your teams to have feedback loops in stead of figuring out new and better measures. The shorter the feedback loops the better, since if things change quickly you need to react quickly. With sufficient feedback you can be fairly sure that you are doing the right thing, or else you will know about it quickly. There are numerous examples from nature where feedback loops regulate and interact in order to steer processes and their outcome, examples ranging from metabolism to neural activity.
One of the keys here is that the sensors in nature are integral parts of the process like for example proteins catalysing the conversion of one substance to an otherbeing inhibited by the end product and stimulated by the raw material. The centralized control seems to be kept to a minimum, and only exercised as a means of giving over all direction, so why not mimic the same in IT?
Empowering everyone to make sane decisions based on feedback, or even listening to feedback at all must feel threatening to management accustomed to safely measuring and them self interpreting the result. However, if what counts is the end result and over all efficiency, the score seems clear.
Thursday, December 28, 2006
Cost of features
Microsoft seems to be hitting their head in a wall with the new Vista content protection according to a very interesting report by Peter Gutmann.
If the actual implications are as fatal as what is stated remains open to see, but a conclusion we can make at this point is that not all end results can be known beforehand when designing an extremely complex piece of software like for example Vista.
There is no way of knowing how the software/hardware ecosystem will be affected by a gigantic release with lots of new unknown features/traits. Transparency can help, by making the design and implications visible early, but there is naturally no guarantee that anybody will spot the problem before it is too late. Also, afterwards the problem must be acknowledged and corrected, requiring a fair degree of openness.
This time it seems like bending far over in one direction has resulted in a large invasion of problems from an other unwelcome direction. It is very unfortunate and disturbing that the focus does not seem to be set on making real customers happy, but more on satisfying digital rights owners unspecific dreams.
If the actual implications are as fatal as what is stated remains open to see, but a conclusion we can make at this point is that not all end results can be known beforehand when designing an extremely complex piece of software like for example Vista.
There is no way of knowing how the software/hardware ecosystem will be affected by a gigantic release with lots of new unknown features/traits. Transparency can help, by making the design and implications visible early, but there is naturally no guarantee that anybody will spot the problem before it is too late. Also, afterwards the problem must be acknowledged and corrected, requiring a fair degree of openness.
This time it seems like bending far over in one direction has resulted in a large invasion of problems from an other unwelcome direction. It is very unfortunate and disturbing that the focus does not seem to be set on making real customers happy, but more on satisfying digital rights owners unspecific dreams.
Thursday, December 21, 2006
Openness, transparency and efficiency in IT
In Open Source - More Than Code Bill Barr discuss if you really can run IT efficiently while maintaining full transparency to stakeholders. I have also been looking at the effects of transparency.
The spirit of SOX for example seem to require that business have full visibility into everything going on, down to a very technical level. In my opinion business should. First of all, they are paying for the IT systems and their development. Secondly, the more involved they become the bigger is the opportunity that the IT system serves the business need. Following SOX and other rules can then just be seen as another minor reason for driving transparency. Extra paperwork and meetings is often seen as a main drawback, and with the sad state of documentation I have seen on a number of cases this is most certainly true, but not in my opinion valid, since there is usually some value in improving documentation anyway.
The problem with transparency and involvement starts when business stakeholders interpret this as a mandate to lead and manage the IT development. This often result in micromanagement, having everything grind to a halt. The problems may be small or big, but in the end they all depend on the personalities and driving forces of the stakeholders you are working with. Keeping these stakeholders in the dark is often seen as a solution, but that can create even more serious antagonism further down the road. I think that a better solution is to make them aware of other stakeholder views and needs, in terms of costs and business risks, and the fact that they may not necessarily have all the facts or the knowledge required.
I would not say that transparency or openness need to be in conflict with efficiency, but there is a good chance that they will be. In a deep hierarchical "chain of command" type organization where departmental silos are not discussing with each other I think you can have either openness, transparency or efficiency. Achieving any two in any combination would come at an extremely high price, and all three is probably next to impossible achieving.
That said, I still consider involvement of individuals, openness for change and transparency to what is being done as the key to efficiency. The individuals in a lean organization, once set with a task, may well know how to achieve a solution as long as the organization is truly open to this kind of involvement. Transparency guarantees that if some initial condition change or some theory is later found wrong, there are plenty of opportunity spotting and correcting the error. The key i think is that everybody need to agree on what the actual problem that is to be solved is and listen to each other.
Improving efficiency is easy to if you don't need to consider any side effects. In the end, IT department efficiency should not be considered as the ultimate goal, and trying to optimize on a project or code development level can in worst case only reach a local optimum. Optimizing for business efficiency would be far better in many cases.
The spirit of SOX for example seem to require that business have full visibility into everything going on, down to a very technical level. In my opinion business should. First of all, they are paying for the IT systems and their development. Secondly, the more involved they become the bigger is the opportunity that the IT system serves the business need. Following SOX and other rules can then just be seen as another minor reason for driving transparency. Extra paperwork and meetings is often seen as a main drawback, and with the sad state of documentation I have seen on a number of cases this is most certainly true, but not in my opinion valid, since there is usually some value in improving documentation anyway.
The problem with transparency and involvement starts when business stakeholders interpret this as a mandate to lead and manage the IT development. This often result in micromanagement, having everything grind to a halt. The problems may be small or big, but in the end they all depend on the personalities and driving forces of the stakeholders you are working with. Keeping these stakeholders in the dark is often seen as a solution, but that can create even more serious antagonism further down the road. I think that a better solution is to make them aware of other stakeholder views and needs, in terms of costs and business risks, and the fact that they may not necessarily have all the facts or the knowledge required.
I would not say that transparency or openness need to be in conflict with efficiency, but there is a good chance that they will be. In a deep hierarchical "chain of command" type organization where departmental silos are not discussing with each other I think you can have either openness, transparency or efficiency. Achieving any two in any combination would come at an extremely high price, and all three is probably next to impossible achieving.
That said, I still consider involvement of individuals, openness for change and transparency to what is being done as the key to efficiency. The individuals in a lean organization, once set with a task, may well know how to achieve a solution as long as the organization is truly open to this kind of involvement. Transparency guarantees that if some initial condition change or some theory is later found wrong, there are plenty of opportunity spotting and correcting the error. The key i think is that everybody need to agree on what the actual problem that is to be solved is and listen to each other.
Improving efficiency is easy to if you don't need to consider any side effects. In the end, IT department efficiency should not be considered as the ultimate goal, and trying to optimize on a project or code development level can in worst case only reach a local optimum. Optimizing for business efficiency would be far better in many cases.
Monday, November 06, 2006
Knowledge for public good?
I would agree with Gowers review of IP (Intellectual Property) report that public good should be considered over private rights. I guess somebody eventually will take the time to sit down and plot out a chart of the solutions to the problem, as a function of public good, private profit and the good that can bring over time of a set of different IP types, but that may not be all that useful.
The biggest problem with the IPR legislation as i see is however that the law and the public perception of what is right is diverging. I would say that the damage to coming generations perception of right and wrong simply cant be corrected if that situation is allowed to continue.
Both society and coorporations would in my opinion benefit the most if knowledge and IP works of any kind truly were open for others to mix, match and blend withingiven limits that everybody were ready to aknowledge.
The biggest problem with the IPR legislation as i see is however that the law and the public perception of what is right is diverging. I would say that the damage to coming generations perception of right and wrong simply cant be corrected if that situation is allowed to continue.
Both society and coorporations would in my opinion benefit the most if knowledge and IP works of any kind truly were open for others to mix, match and blend withingiven limits that everybody were ready to aknowledge.
Sunday, October 08, 2006
Openness and efficiency
What is the point in keeping new innovative ways of working secret?
Some state that it is not any single idea, or any kind of secret you hold that is the key to success, but hard work. People would very much like the opposite to be true, since that would mean that they don't need to work their butts off, but can sit back and relax while somebody buys their idea. That is however seldom the case. You realy need to work hard in order to do a better job than your competition. Most of them do a really lousy job anyway so it should not be all that hard, and the rest never understood what it the idea was all about anyway...
However, secrecy, internal competition and the "secrecy cult loving"” corporate climate seems to have got the better of efficiency in today's world. Patents, non disclosure agreements, licenses, information on need to know basis and lawyers all help holding this cult of secrecy in power. Does it really make sense?
Nature teach us that individuals die and the knowledge they have acquired with them, but species survive. Other individuals in the species pick up some of the ideas and transmit them further. Holding secrets is not very smart in the long run with the knowledge that the individual will go under in a short while. Ideas can only be saved by copying them. The more copies the better. If there is a short term benefit to be gained from keeping secrets it used to be profitable only for a short time. In the long run somebody always came up with a better idea. With the prolongation of the patents life and copyright this is perhaps not the case any more.
Organizations and individuals should in my opinion at least be open to new ideas. There is a saying that innovation happens elsewhere. I tend to think that this is very true for major new ideas, since organizations and individuals inside the organizations get used to defend their own position and focus on minor improvements that can be done without disrupting the stagnant state the organization live in. If organizations are closed to new ideas then they will in the long run be doomed, which should however perhaps not be seen as a bad thing. Something else will follow after a major failure.
Openness also implies sharing. You have to give something in order to get something. I think this is also true when sharing ideas. If you only listen you will be able to understand something, but the comments you get from sharing what you do can be very much more valuable.
Some laws however set limits on the openness an organization can display and what can be shared. Health records or other personal information can not be shared without serious risk of violating the law. Openness can generally not be targeted at only one subset of the organizations peers, but must be equal to all. The law however still leaves plenty of room for sharing, openness and learning from others for any organisation to become more efficient.
Some state that it is not any single idea, or any kind of secret you hold that is the key to success, but hard work. People would very much like the opposite to be true, since that would mean that they don't need to work their butts off, but can sit back and relax while somebody buys their idea. That is however seldom the case. You realy need to work hard in order to do a better job than your competition. Most of them do a really lousy job anyway so it should not be all that hard, and the rest never understood what it the idea was all about anyway...
However, secrecy, internal competition and the "secrecy cult loving"” corporate climate seems to have got the better of efficiency in today's world. Patents, non disclosure agreements, licenses, information on need to know basis and lawyers all help holding this cult of secrecy in power. Does it really make sense?
Nature teach us that individuals die and the knowledge they have acquired with them, but species survive. Other individuals in the species pick up some of the ideas and transmit them further. Holding secrets is not very smart in the long run with the knowledge that the individual will go under in a short while. Ideas can only be saved by copying them. The more copies the better. If there is a short term benefit to be gained from keeping secrets it used to be profitable only for a short time. In the long run somebody always came up with a better idea. With the prolongation of the patents life and copyright this is perhaps not the case any more.
Organizations and individuals should in my opinion at least be open to new ideas. There is a saying that innovation happens elsewhere. I tend to think that this is very true for major new ideas, since organizations and individuals inside the organizations get used to defend their own position and focus on minor improvements that can be done without disrupting the stagnant state the organization live in. If organizations are closed to new ideas then they will in the long run be doomed, which should however perhaps not be seen as a bad thing. Something else will follow after a major failure.
Openness also implies sharing. You have to give something in order to get something. I think this is also true when sharing ideas. If you only listen you will be able to understand something, but the comments you get from sharing what you do can be very much more valuable.
Some laws however set limits on the openness an organization can display and what can be shared. Health records or other personal information can not be shared without serious risk of violating the law. Openness can generally not be targeted at only one subset of the organizations peers, but must be equal to all. The law however still leaves plenty of room for sharing, openness and learning from others for any organisation to become more efficient.
Subscribe to:
Posts (Atom)