4 types of documentation – towards Pragmatic Knowledge representation

There is a lot of confusion caused by the Agile manifesto simplification regarding “documentation”. 

I will address this and also highlight a concept you just HAVE TO learn and know about (if you did not already), Parsimony!

Parsimony

Parsimony is also called ”Occam's razor”. It means that given two equally accurate answers, science prefers the simpler.

Einstein’s classic quote is a bit false as he was AGAINST simplifying to the point that laypersons might understand -  if it were simple enough to understand, it wouldn’t be a discovery worth lauding!

The full quote is in fact: 

and

It is sad that the use of ”make it simple” has been ”stupefied” to a degree where we can actually choose an apple instead of a full dinner plate, stating that it is simplification without reflecting on parsimony.

The second part of the famous quotes from both Einstein and DaVinci which is so often left out or ”forgotten”, means that we can not compare things regarding ”simplicity” if they do not have the same effect and context.

The act of simplifying is about transformation: to start with a complex statement or representation and to make it more naturally understood and remembered. BUT the transformation result must maintain ALL the needed complexity. Any simplification risk compromising the “truth and the real problem”, and along with the validity of the statement can be completely lost!

In software development, ”simplify” often result in ”stupidify”, as we lose knowledge and/or represent the knowledge in an ambiguous way, even if we use nice colours and happy smiling hand-drawn cartoons.

Remember: The complex does not disappear if we hug each other or hold hands in a ring.

The result of a stupid simplification will often lead to stupid decisions!

Comprehensive documentation

So what does ”comprehensive documentation” really mean. If you do not understand that there are different TYPES of documentation with different needs depending on the situation and the involved stakeholders, then read the quote by William Blake: ”if you generalize, you are an idiot!”

Too much of the WRONG TYPE of documentation is BAD, but not enough documentation of some other type of knowledge representation can be even worse and extremely stupid!

In fact, if a ”map” helps you to understand and get from ”A to B” faster and is more ”economically sound”, then ”draw the f...ng map!”... 

To talk about the weather does not solve the problem. To talk about all the details in a blueprint without having a blueprint is just stupid!  

Talk ABOUT the blueprint (and do the updates in the ”map”). 

So which types of documentation should we consider to become smarter?

The 4 types of Knowledge representation

We need effective and efficient communication in Agile methods because of its focus on short feedback cycles and personal interactivity.  We should not only talk but rather “talk about” something i.e. an important knowledge representation that can be read at any time, can be updated and changed iteratively (=“update the map with input from the reality”).

We should Identify the need for a “keeping knowledge” longer than “the timespan of the mechanical wave that results from the back and forth vibration of the particles of the air” (i.e. talk) and give a Rationale (logical explanation) why it is important to “store” the information, for whom, and what can happen if we do not represent the talk with a more persistent representation (= the possible impact)

It is often so that specific domains, products, project contexts require certain information in a specific format or representation for specific regulators to follow the law. And some information has a lasting value beyond the initial development effort.

We need to be able to predict which knowledge will be needed in the future. In those cases (and they are many!) verbal communication is not enough semantically powerful, and “only talk” often lead to inefficient, misleading communication and different interpretations where the “internal models” do not match at all ... But the participants are not even aware before it is too late, and bad decisions are already made.

Representing knowledge (e.g. writing, drawing/modelling) is also necessary as a means to improve and support the thought processes and the “mental model” for understanding. The improved thinking and understanding will last even if the documents created, is “thrown away” and not stored or updated, as the process of “drawing the map” forces people to ask questions that enhance understanding and often creates a new way of thinking!

If we use the “right type of map” it will make us smarter!  We will also get a better understanding and updated thinking when we communicate using “a map” (= Knowledge Representation) than without!

However, we need to become experts in identifying the important Knowledge that we need to represent in a smart and persistent way.

A good way to start is to NOT only talk about “documentation” as an Agile zealot, but classify all possible documents according to the 4 types of Knowledge representation as shown above.

We should definitely ask ourselves if a “living product/project backlog” is sufficient. For real!

Legally Necessary Documentation

Certain product- and project contexts demand certain information in a specific format for a set of identified regulators. Just to follow the law!

It means that the specific Knowledge Representations MUST be a part of the product, and without it, we will be breaking a set of laws and standards! In other words, we MUST create, read and update these documents to obtain legal approval for the product/project work we are doing.

For example, building a dog house or a simple website is not the same thing as building systems (both processes and automatic systems!) for nuclear plants or in the health care sector. The lack of the correct set of documentation can thus have a huge on the company as a whole...yeas I am thinking about Boeing in the avionics sector.

Preservation Documentation

We need to know which information that has a lasting value beyond the initial development effort.

We should ask ourselves what is the “shared archive for the team, product and organization”?

Can we live with the risk of being dependent on the memory capacity of the individual team members and “the risk of someone forgetting”, which is a reality when dealing with tacit knowledge, only in the head of the people involved and those who happened to be present in the set of meeting in which the concepts were “discussed”...

How you come across situations like: “why did we decide to implement this?”.
In many cases, we gain time and are agile by keeping some sort of representation of discussions about decisions “outside the head”. If we do not all have the same idea on the goals that the system was built to achieve
 or which use cases that are the most important for the automatic system and the manual process, then we will probably have some major problems in the future (or giving someone else an impossible task). We need to save decisions made during development on why we have included or excluded certain functionalities, constraints and why we have focused on certain quality attribute values and not cared about others. Or why we did not involve stakeholders from group X, but we spent a lot of time with stakeholders representing domain A and B...

It is often the team itself that identify what they want to preserve & “formally remember”.

To be agile is to be smart!...do not be agile stupid and “throw away” a “map” that can help you to understand and do your job faster and more effectively as well as enhance communication. To throw away important knowledge because some “manifesto guy tells you to simplify” is like not using a hammer while building with wood and nails. Just stupid.

If you need requirements and test cases for specific parts of the system, just represent that knowledge!

Documentation for Communication

One size does NOT fit all, and direct verbal communication is NOT always the best!

Verbal communication is often not enough semantically powerful... which can lead to inefficient, misleading communication and different interpretations, “internal models” impossible to validate, etc...

There are in fact a lot of communication challenges related to the verbal focus:

When dealing with distributed teams and when you have time restrictions for stakeholders, it is not easy to “just talk”. We often have language barriers and cultural differences that can complicate already complex information to unsurmountable mountains. Talking about a complex “map” is almost impossible!

Do the test: explain a route in a country you do not know only using words. And then try to do the same but also using a map. Talking ABOUT the “map” is a sure way to communicate better!
 

To re-read and iteratively update a “conversation”, you have to have an eidetic memory, for most people it is impossible. Using a diagram to represent a complicated algorithm is not only smart, but it also guarantees iterations and enhancements as the diagram will always be open to enhancements (“let’s continue where we left off”). The conversational situation will NEVER be found again and can not be “re-read” by external parties that were not present in the moment.

We should also be open and agile for the fact that a lot of stakeholders prefer written documents, visual diagrams and info graphs, rather than just meetings, talk or reading the source code

We should never document for the sake of documenting, but IF communication documentation is successful then we should definitely archive the knowledge representation. 

Everything else would be stupid!” ...and we do not want to be agile stupid, do we?

Documentation for Thinking!

One of the most important uses of representation is to improve and support thinking & understanding. We want to represent knowledge (e.g. writing, drawing/modelling to enhance our thought processes and the “mental models” and to be able to do that not only individually but also collaborate on a “collective mental model”

The happy part is that improved thinking and understanding will last even if the knowledge representation created for the specific problem is “thrown away” and not stored or updated!

The process of “drawing the map” forces people to ask questions that enhance the understanding and often creates a new way of thinking! AND it is possible to SHARE!

A common example is to stop talking or textually writing a use case, and instead of creating an activity diagram of the use case flow. This forces the talker/writer to THINK about concrete interactions between the system and the actors including, exceptions and alternative scenarios etc..

It means that the Activity diagram is specific Knowledge representation tool to store the knowledge and understanding of a system. Talking about the system via the activity diagram enhance the communication by the power of the syntax and semantics.
And it has nothing to do with drawing funny cartoons on flip charts.

Be happy. Be smart. Don’t be agile stupid!

To get more inspiration:
Read some of the excellent IREB documents from FL, AL modelling and the RE@Agile courses.
Or apply for one of my seminars!