Models for thinking, evaluation, discussion and affecting people

Models can greatly aid in thinking about systems: A model can be a stake in the ground to disagree with. A model can be something to help evaluate the properties it exhibits. A model can be used for ensuring that the thinking around the model is consistent with reality,  for visualizing of just for general communicating an idea among numerous other uses. Some use cases however takes the potential impact of the models much further.

A model can be formal, in between or just ad hoc, depending on the need. Formal models can be expressed in mathematical notation and even queueing models in some cases, not to mention UML. Based on the model various calculations can be performed to draw new conclusions apart from communicating the idea behind the system depicted by the model. The model can live on the back of a napkin, on paper, on a whiteboard, in mathematics, in computer programs or in an AI implementation generated by machine learning, or just as an abstract concept in ones mind.

With a well equipped mental model you can more easily spot counterfactual misinformation or other issues. Models can however host various kind of bias. When the underlying assumptions are flawed, also the model will be flawed. The model can be too simple to capture what is seen as crucial, like for example when missing feedback loops.

In the obvious domain it is simple to create models of well behaved systems. As a system grows more complex, the difficulties grow. Modelling a chaotic system, or a system in disorder should be seen as virtually impossible. 

Constructing a model used to be a human activity, and the impact somewhat limited, but with the spread of AI / ML the origins of models and use of models will be breaking more new ground. There has been a lot of discussion about the importance of model explainability, fairness and transparency, and i support the idea, although i'm skeptic that this will have the desired result of reducing harm. The outcomes of the use of models need to be tested, evaluated and verified from a very diverse set of points of view, including those affected by the model. The currently proposed regulation and limitations seems prone to circumvention. Perhaps part of the solution is that models with impact to humans need to be evolved to become better over time.

Any model will always be imperfect in some aspect. There is no such thing as a perfect model, except for the system itself in a very limited sense. Some models will even cause harm. Modelling a complex system can newer capture the full detail of the system. Some models are however still useful. 

Language matters

 Using acronyms, internal established slang, management speak and special terms can make writing simpler. This however comes at a cost to the reader having to understand the written content. When your  reader is not initiated to the terms the text may even act as a barrier to exclude people from understanding what you are writing (or talking) about.

There is however a certain trend to exhibit expertise trough obscurity that is especially troubling. The definition of an expert should first and foremostly contain the criteria to communicate clearly and understandably. People may not have the sufficient experience to challenge the expert, don't bother, or are busy doing more important stuff. It requires energy to point out the problem. As an expert the most important thing to accomplish is however to communicate with others, and doing so clearly.

When you write it takes time, but only once. Using an acronym can save a few keystrokes, but is this a significant optimisation? If the text is read by many people you should in stead optimise for reader efficiency. Forcing the reader to look up terms is time consuming and even error prone. Making the reader wonder what something means also costs time.

There are also a lot of terms to avoid that has a problematic past history, even if you do not intend any offence or you intend to give a new meaning. You can however not know what the reader understands and experiences when encountering such a word. It can be hard to know upfront all problematic words, so be ready to apologise and change when someone reacts to what you communicate. 

Extensive use of acronyms, slang, and specialised terms are all ways to set yourself up to fail with what you try to communicate. At least define the terms you are using and assume that the reader does not know the meaning of the special terms. The reader who already know the term can always easily skip the definition at a very low cost.

Knowing what is true and spotting lies

There is lots of research being done on various themes like Artificial Intelligence among other popular fields. We see constant daily stream of publication with new findings. The volume is overwhelming.

All publications are however not that great. It seems that anyone can publish almost anything without putting in the required effort of verifying that the findings are based upon a real effects. There also seems to be a disregard for existing research, either by authors not knowing enough about the subject or authors selecting to reference only the works that fit together with the desired findings. This seems to be especially a problem when the subject of the study is multidisciplinary. Using the language of one field can hide issues with the findings behind difficult terminology. 

Reviewing multidisciplinary research must be quite hard, as none can be expected to be an expert in all areas of many subjects. Recognising bad research publications is also not easy, but there are a few things you can do. First and most importantly everyone needs to practice skepticism when reading. Learn to know what bias looks like. If things look too clean and too good to be true the warning should sound that there may be a problem. When serious money or similar high value stakes are in question we need to be on the lookout for conflicting interests. 

The problem is made worse by the tendency of people to uncritically repeat findings with a catchy point, regardless of the truthiness of the findings. Once a problematic finding is out in this way it takes a lot to refute it.

The book "Calling Bullshit: The Art of Skepticism in a Data-Driven World" by Carl T. Bergstrom, Jevin West gives a very thorough overview of the problem that i highly recommend anyone to read.

Fooling oneself is the easiest thing to do, and by doing so we easily fool others.

The need for speed in agile

Fast feedback, valuing speed of delivery and rapid iteration seems to be the norm today in agile. Is speed over-emphasized in agile?

With complex systems, changing the pace of feedback can cause resonance, that can bring serious disturbance to an otherwise stable system. A faster pace can also drive teams to burn out. Requiring stakeholders to make up their mind and provide feedback at speed about for example a feature implemented in the last sprint may not get the team those well thought out comments that will improve on the final outcome. Forming a well thought out comment require time. Communicating the thought clearly takes time. Considering diverse views take time.

According to modern theory, those that can go trough a OODA loop (OODA comes from Observe, Orient, Decide, Act) most rapidly, will obtain advantage over those progressing slower. In areal combat this is likely to hold true, but in a collaborative environment this mind set seems foreign.

Obtaining reliable high quality feedback is crucial in agile development, but does the need for feedback to be "rapid" actually cause problems?

In my view the need for speed should be carefully considered and not just taken as the absolute undisputed truth.

Thinking, testing, practicing and reevaluating

The traditional view of learning, ie picking up a new skill from someone else and applying it to a problem, is somewhat out of date in our complex or even chaotic world. 

Firstly, if you are part of an organisation delivering training, make sure the learning works for your audience. If unsure, solicit feedback. Show that you care about the people in the audience. Consideration for different learning styles should have no place in any of todays trainings. See https://www.psychologicalscience.org/journals/pspi/PSPI_9_3.pdf for a closer look on why. Everyone can learn, and people have a diverse set of preexisting experiences that can be taken into account but traditional learning styles is not one of them. Bad training will demotivate people and put a break on innovation.

Everyone should have a mental model of the problem domain they experience. If not, take a step back and form an initial model,  acknowledging that the model will have flaws. Training would be most useful when evolving this model. Training could for example help people question the underlying assumptions of their models, reach new conclusions and verifying what is known. Evolve the model step by step by testing and verifying what is known and probing what is unknown. Cause and effect may still not be deterministic, but this can be a lesson and a property of the model too. Ensure that your model stays up to date with what you learn. 

There are also old practices or features of your model that will not make sense anymore as conditions change over time. These will need to be unlearned. Unlearning is also learning.

Learn to question what is taught. Ask why. Stay up to date by reading, listening to podcasts and discussing with colleagues. Don't rely on a single source. Teach others what you learn, and stay curious.

For problems that are simple or for problems where cause and effect can be analysed by an expert to solicit a single highly constrained solution that is likely to work, traditional training still can have its role. For problems where no known solution exists something above this is required.

The efficient and high performing engineer

There was lots of discussion regarding 10x engineers.  You may also recognize the 10x engineer concept as similar to "rock star developers", "ninja coders", and the "lone wolf" saving the day for a system.

Frankly I find the concept unsettlingly stupid.

If there was to exist a real 10x engineer, what characteristics would such person display? I would argue that the most important aspect would be helping others, mentoring and ensuring that fellow humans are as productive as possible. This include communication, teaching, and generally being human, which is very far from what engineering managers consider critical skills. Another critical aspect is that high performers should know when to unlearn aged practices that are not useful anymore.

Having engineers with high impact implies that there is a team around the engineer. Most of leadership practice however revolve around individuals, not teams. I consider it essential for leaders to unlearn traditional management/HR practice and focus more on teams if promoting high performance engineering is something they wish to promote.

Return of test investment

During a recent discussion the question was raised on how to make investment in test automation pay itself back. How do we know its worth the effort? Implementing automated acceptance tests can require quite big effort.

Theoretically there are two alternatives to automated tests. You can either test manually or you can disregard testing altogether. Having no testing today should be considered a breach of ethics in IT. As a presented at USENIX OSDI 14 by Ding Yuan et. al. most catastrophic failures could be avoided by testing error handling without any understanding the underlying software design.

Manual testing is time consuming, subject to testing procedure errors and require lots of attention. As manual tests take time to execute, the manual test procedure can easily slow down the release deployment and cause delay. With a manually executed procedure the results will also not be as reliable as with the automated tests. The variation can naturally be beneficial, but most likely the test result may be compromised.

When calculating how much benefit automated testing can bring, the main consideration is often how much effort and time can be saved by the automation. Less emphasis is typically placed on reliability aspect. Freeing up testers time to do more thorough explorative testing must also be considered. Secondary effects are also of importance. Having automated tests cause code and features to become more easy to test. Having automated tests also enable faster releases. Taking secondary effects into account when calculating benefit of test automation can change the equation.

Additionally there is also the consideration on which side do you want your estimate to side, on the side of promoting improved testing or on the side of limiting quality...