Author: Daniel Kahneman, Cass Sunstein, Olivier Sibony
Noise: A Flaw in Human Judgment Daniel Kahneman, Cass Sunstein, Olivier Sibony 2021
Judgment and its flaws(Noise Flaws of human judgment)
Judgment must be distinguished from facts and opinions. The fact is clear.
The distance from the Earth to the Sun is 150 million km.Noise Flaws of human judgment
Opinions can be arbitrarily varied and depend on personal taste.
Tarantino is the best director in the world.Noise Flaws of human judgment
Judgments, on the other hand, are always based on some realities of reality and, at the same time, imply a spread of estimates.
“The Brazilian economy is about to default.”Noise Flaws of human judgment
Judges, teachers, doctors, politicians, economists – they are all constantly making judgments. Judgments can be verifiable (the effect of drugs on a patient, the course of a pandemic, election results), or they can be purely evaluative (a student’s grade on an exam or a judge’s verdict does not directly correlate with any objective value since both the law and the criteria for teacher evaluation ultimately account conditional).
All flaws in judgment come down to two types: bias and noise. If some expert group makes the majority of the same type of mistakes, again and again, this is biased.
The directors of the company year after year are guided by an overly optimistic sales forecast. The company year after year continues to invest in unsuccessful projects that should have been written off long ago.Noise Flaws of human judgment
Noise is a spread of judgments that should be the same.
Two judges give different terms for the same crime. One doctor treats the tumor conservatively, while another immediately recommends surgery.Noise Flaws of human judgment
Types of noise
Noise is more insidious than bias. If the doctors of a certain hospital treat the sick again and again badly, it will quickly come to light. But if some doctors in this hospital are good and others are illiterate, this state of affairs can remain unchanged for much longer. At the same time, the work of good doctors does not compensate for the work of bad ones: patients with incorrect diagnoses do not become healthier. The hospital continues to operate, but an inconspicuous noise is holding back its development.
There are three types of noise:
- In the same court, both harsher and more lenient judges can serve. Some economists can predict the development of the same situation optimistically, while others are more critical. The spread of judgments on similar occasions is called level noise.
- Harder judges are not necessarily harsher on all defendants. They can be more lenient towards petty thieves and stricter towards the perpetrators of an accident. This also generates noise – stereotyped: it depends on the specific reaction of a particular person, and the reaction, in turn, is influenced by his experience, mindset, and education.
- A kind of template noise is random noise when the same expert disagrees with himself in different circumstances. Studies show judges are more lenient with defendants after their favorite football team wins, and doctors prescribe more opioids in the afternoon when they are tired and lack the energy to make better decisions.
The level of noise is more noticeable than the stereotyped one: if you are the head physician of the clinic, the difference in the work of the wards will sooner or later catch your eye. But the stereotyped noise sounds inside each of us and therefore remains unattended.
Causes of noise
There are two types of thinking. One is fast, intuitive, and automatic. The second is slow, rational, and logical. Fast thinking simplifies the world around, but it makes it easier to navigate it. Slow thinking is indispensable for solving time-consuming tasks. The problem is that in everyday life, fast thinking often outpaces slow thinking 1. This is fraught with many misconceptions.
- We interpret the facts so that they coincide with our worldview, and discard the rest of the data. This is the confirmation bias and has been called the mother of all mental errors.
The company’s management adopts a new development strategy. All events that hint at the success of the strategy are celebrated with great enthusiasm. What about those that are against the strategy? Yes, these are just exceptional cases.Noise Flaws of human judgment
- We consider probable not what is confirmed by facts and statistics, but what is more impressive and, therefore, comes to mind faster.
News about plane crashes is much more frightening than reports of accidents, while the latter happen much more often, which means they threaten us more.Noise Flaws of human judgment
- When analyzing new information, especially numerical information, we tend to focus on the previously received information, even if it has nothing to do with the current problem (“anchor effect”).
In one experiment, a researcher showed subjects a bottle of unknown wine. And then he asked everyone to write down the last two digits of their insurance number on a piece of paper. Then the subjects named the amount that they would give for wine. Those with the last two digits in the insurance were higher, called the price almost twice as much as the owners of low numbers.Noise Flaws of human judgment
- We are bad at remembering facts, but we are good at remembering stories woven from these facts. The less contradictory these stories are, the better.
Why did World War I start? Because in 1914, in Sarajevo, a Serbian terrorist shot dead the heir to the throne of Austria-Hungary. Of course, a whole chain of factors led to the war, but only a concise story remains in memory, preferably a dramatic one. And the more foldable it looks, the more facts are left “overboard”.Noise Flaws of human judgment
- In any event, it is important for us to see the reason, hence the belief in retribution, and fate. At the same time, we continually confuse the cause with the effect or take mere similarity as the cause.
For the inhabitants of the Hebrides, in the northwest of Scotland, the presence of lice in the hair has long been closely associated with the state of health. If the lice leave the host, expect illness. To rid the patient of a fever, lice were planted on his head. If the insects remain on the head, everything is in order, the patient will live. Doctors are unaware that cause and effect are confused: not lice are curative, but insects do not escape because the fever passes.Noise Flaws of human judgment
- We easily succumb to the illusion of forecasting. Social networks and news sites are full of various predictions: whether China and the United States will quarrel, whether the euro will fall in value, whether the regime in North Korea will fall. An expert who knows what to say about tomorrow is a welcome guest on TV. After all, we all want to know what will happen tomorrow.
University of California professor Philip Tetlock studied 82,361 forecasts from 284 experts over 10 years. Result: the experts guessed no more often than if the scenarios were chosen by a random number generator. Especially bad predictors were those who enjoyed the increased attention of journalists and promised the world a variety of troubles. These people excel at telling cohesive, logical, and most importantly, dramatic stories, which is exactly what the media needs. Few reporters were willing to ask how they came to these conclusions.Noise Flaws of human judgment
- One aspect that impresses us can affect the assessment of the whole situation (“halo effect”).
Dozens of studies have shown that we perceive beautiful people as more honest, smart, reliable. It is easier for them to make a career. This effect also works in a negative way: for some people, gender or race completely overshadows other qualities of a person.Noise Flaws of human judgment
These delusions disorientate us and provoke noise.
Nine Principles of Noise Reduction
All these causes of noise come down to three theses:
1) we think inaccurately, preferring intuition to logic;
2) we do not want (and do not know how) to notice and recognize this;
3) we are easily influenced.
Bypassing these traps means reducing the amount of noise. Simple principles will help. Daniel Kahneman and colleagues call these principles mental hygiene. Following these principles is like washing your hands before eating: it may not completely avoid the disease, but it will significantly reduce the risk of it.
Principle 1. If you need experts, choose the best
In any business, there are those who do it much better than the rest. Choose just such, and the noise level in the work will decrease. Key criteria: good experts are experienced, open to new information, and always ready to learn.
The professional level of an expert is easier to assess, while general cognitive abilities are more difficult. There are many intelligence tests, but here is one of the most revealing. Psychologist Shane Frederick developed what is known as the Cognitive Reflection Test (CRT). The most famous question from there is: “A tennis racquet and a tennis ball cost $1.10 together. A tennis racket is $1 more expensive than a ball. How much does the ball cost? The professor tested a thousand people. It turned out that people who successfully passed the CRT differed from those who did not pass the test in a number of important ways. They are less impulsive and better able to assess the value of delayed gains (Frederick asked subjects, “Would you like $3,400 now or $3,800 in a month?” People with low CRT scores mostly chose $3,400 now). Other people who have successfully passed the CRT, more skeptical and generally non-religious. In a word, the test helps to identify the owners of slow, rational thinking.Noise Flaws of human judgment
Principle 2: Appoint a Decision Monitor
An outside perspective is often more honest and always gives us food for thought. It is also easier to recognize prejudices in others than in oneself.
Of course, not everyone wants to go against the collective. Such a “devil’s advocate” is a skeptic. He is not fascinated by the corporate spirit of the team and is by no means sure that luck will always be on the side of his native company. In a word, a rather unpleasant type. He is what the team needs!
It is better if the observer will be guided by a list of control questions that reflect in detail the characteristics of the company or project. Sample questions for this list:
- Have all alternatives been considered? Has the group actively sought evidence to support these alternatives?
- Was uncomfortable data ignored?
- Do the participants in the discussion exaggerate the importance of this or that event because of its excessive drama, anecdotal?
- When it came to forecasts, were their sources and validity questioned?
Principle 3. Give the floor not only to people but also to algorithms
Machine algorithms today not only help doctors make diagnoses but also, for example, predict the decisions of the US Supreme Court, determining which defendants are more likely to be released on bail.
Algorithms do not know the disadvantages of fast thinking. They can be biased, but only if they were trained on biased data, in which case their work can be corrected. Algorithms are receptive not to stories, but to facts. People’s judgments are very often unreliable – the conclusions of algorithms are very often reliable, and they can always be double-checked. Seek their help if possible in your business.
Principle 4: Give the Expert As Much Information as You Need
Extra information can affect the judgment of an expert and make him biased. Therefore, the expert must receive exactly as much and exactly the information that is necessary for making a judgment – but no more.
By supplying fingerprint examiners with material for analysis, US forensic labs keep them in the dark about the entire case (say, whether the suspect has previously been the victim of a miscarriage of justice). Otherwise, as observations have shown, extra information negatively affects the expert’s judgment: he becomes biased. In addition, examiners must document their fingerprint analysis before looking at a fingerprint sample to decide if they match. This sequence of steps helps the expert avoid the risk of seeing only what he is looking for.Noise Flaws of human judgment
Principle 5. Remember the “wisdom of the crowd” (but make sure that she is not too smart)
Collectively, we are just as wrong as we are individuals. But there is an interesting feature that psychologists called “the wisdom of the crowd.” What percentage of all airports in the world do you think are in the US? If one person answers this question, their answer is likely to be inaccurate (accurate – 32%). But the average answer of a large number of people is likely to be close to the truth. Such is the paradox of statistics: averaging many independent judgments yields a new, less “noisy” judgment.
With the same answerer, this also works – but only if he gives a second answer after some time, ideally after a few days. His average answer will also turn out to be less “noisy”.
However, a crowd is wise only when the people who make it up to do not discuss the issue with each other. Once they know other people’s results, wisdom disappears. People are suggestible beings. This is the danger of collective discussions: the tone in them is too often set by the one who speaks first.
Experiments with juries have shown that if at the beginning of the discussion the group is lenient, then the result of the discussion (the verdict) will be even more lenient, and vice versa. Participants not only infect each other with the same opinion, but also emotionally wind each other up.Noise Flaws of human judgment
How to be? Get independent judgments from several experts without informing them about each other, and only then consider whether these judgments can be combined. If you are gathering a team of experts for a face-to-face discussion, make sure that they are as diverse as possible in terms of life and professional experience: in this case, their average answer will be more balanced.
Principle 6. Turn the “anchor effect” to your advantage
- “Rate the performance of our company’s strategy department this year on a scale of 1 to 5.”
- “Evaluate the performance of our company’s strategy department this year compared to last year.”
Which question is easier to answer? Of course, on the second: our brain does not manage absolute values well, but loves to compare everything. Therefore, we often become victims of the “anchor effect” (think of the experiment with a bottle of wine). Still, it can be used to your advantage, if the choice of “anchor” is not accidental. In the above question, such an “anchor” is the phrase “last year”. Replace judgments with rankings, scales with absolute values with scales with relative values, and a separate assessment of each employee with their comparative assessment. Let the basis for the assessment be not abstract figures or turnovers (“on a scale from 1 to 5”, “rate it as satisfactory, good, excellent”), but specific formulations that correspond to the work of your company.
Principle 7. When dealing with a forecast, especially an improbable one, remember to regress to the average
Extreme, incredible performance, whether it’s exam grades, the temperature outside the window, or the stock market, does not stay that way for long – in the end (and sooner rather than later) everything returns to the average numbers. Knowing about this principle allows you to evaluate any forecasts more soberly.
The most successful business plan in five years is unlikely to be the most successful in the next five years – the life of the company will return to the average.
Patients hospitalized with depression leave the hospital in a better condition, but there is no guarantee that the stay in the hospital has had a beneficial effect: it is possible that the course of the disease has returned to the mainstream of the average indicators.
The VP of Sales hires a new salesperson who performs brilliantly in an interview. Impressed by this conversation, the vice president estimates that the newcomer will bring in at least $1 million in the first year of work for the company – twice the average amount earned by previous employees in the same period. However, it is much more realistic to count on $600-700, since everything is returning to the average.Noise Flaws of human judgment
Principle 8: Avoid stimuli
Random noise is generated by fatigue, hunger, time pressure, and bad mood – they make us impatient (remember how fatigue makes doctors avoid looking for more flexible treatment). Each of us knows exactly when during the day he experiences a breakdown when the headworks worse. Do not make difficult decisions on an empty stomach or in a bad mood, and problems in life will become less.
But a good mood can also do a bad job. Therefore, it is best to make decisions in a state of even calmness – not in sadness, not in discomfort, but not in euphoria either.
Experiments have shown that people in a good mood (for example, after watching a comedy) become much more trusting and lose their vigilance. A dose of skepticism never hurts.Noise Flaws of human judgment
Principle 9. Consider different aspects of the problem separately, making an independent judgment on each, and only then move on to a holistic assessment
Daniel Kahneman recalls how, early in his teaching career, he graded exam papers. Out of habit, he read all the papers in turn: first one student, then another, a third … As a result, students who gave an excellent answers to the first questions aroused sympathy, and this was reflected in the assessment of subsequent answers. Realizing this, Kahneman changed tactics: first, he gave marks for the answer to the first question for all students, then for the second, and so on. Each aspect of the problem was evaluated separately. This tactic is relevant in job interviews – it allows you to avoid becoming a victim of the “halo effect”.
- First, clearly define what business qualities are required (focus on results? ability to work independently? or something else?).
- Evaluate the candidate independently on each of these criteria.
- Structured, model interviews create far less noise than unstructured ones. Ask predefined questions, and rate them on a predetermined rating scale.
- Of course, when communicating with a person, we cannot distance ourselves from the situation, appearance, and other factors that give so much food for our intuition. But intuition must be the last to enter the business when all the data has been collected and ordered.
At Google, which follows exactly this methodology, the final hiring recommendation is made collegially: a special group examines the full list of recorded grades for each predetermined criteria. The final hiring decision is based on the average score assigned by the four interviewers.Noise Flaws of human judgment
The same tactic (a list of assessments of various aspects of the transaction – their independent consideration – delayed holistic judgment) works for other important decisions.
Company noise audit
Select the department in which to conduct a noise audit. The leaders of this unit should be informed at the outset in general terms that their unit has been selected for a special study. In the explanation, you can not use the word “noise”, replacing it with the wording “optimization of decision making”.
- Assign a project team – it will be responsible for all stages of the audit.
- Appoint judges who will make audit decisions. All members of the project team and judges must have high professional authority.
- Appoint a project leader – a senior manager. This position does not require special professional knowledge, however, a high position will facilitate administrative support for audit participants.
The purpose of the audit is to determine in which areas undesirable differences in judgment are possible. This may be a new project, a business plan, etc. For this, a detailed questionnaire is drawn up for the employees of the unit: what are their forecasts about each aspect of the project, and what factors influence their judgments.
The information is collected on condition of anonymity, employees should know that no punitive measures will follow for these answers.
Having prepared an audit project, the working group submits it to the company’s management. At this stage, it is worth discussing both the possible results of the review and the possible objections of management. The main purpose of this meeting is to get a commitment from management to accept the results of the audit, no matter how unexpected they may be. It is worth discussing with management questions like “What difference do you expect between a randomly selected pair of answers for each case?” and “What is the maximum acceptable level of disagreement for our business on this issue?”. The answers to these questions should be documented and will be compared with the actual results of the audit.
The design team then gathers information and looks for patterns in the data. Each of the judges works independently, in accordance with the principle of “independent review – pending holistic judgment”. At this stage, shortcomings in the company’s business strategy, employee training, and the amount of information provided to them may be found. Based on their analysis, a report is prepared for management. Next, the team moves on to developing tools that reduce noise in the company. They are based on the principles listed above.
Is noise always bad?
Noise is inevitable: where judgments are made, it is always there. The question is whether measures to combat noise will not cost more than its harm.
- The level of noise in evaluating school responses is very high, but if five teachers checked each student’s work, the education system would not be able to withstand such a burden.
- When Facebook introduced strict guidelines for removing (in the company’s opinion) inappropriate content, it reduced the noise but resulted in many acceptable posts being removed as well.
- Measures to combat noise in some organizations turn into an increase in bureaucracy (new positions for auditors, endless revisions), this slows down work.
To regulate the level of noise, society has come up with two tools – rules and regulations. The rules do not appeal to judgments, but to facts, they are rigid and unambiguous, and therefore effectively reduce noise. Regulations leave room for action. “Do not sell alcohol to persons under 18” is a rule. Respect for elders is a social norm. Both rules and norms are imperfect and therefore are adjusted over time (careful handling of animals is the norm, but its regular violation can contribute to the tightening of laws in this area). This is how society evens out the level of noise itself: when is it better, when is it worse.
It’s all about how big the cost of the error generated by the noise is and the cost of its solution. But they should not be underestimated: noise constantly and imperceptibly reduces the quality of our judgments, and therefore the quality of life. In order to draw attention to this enemy and explain how to deal with it, the book was written.
Top 10 Thoughts
1. Where there are judgments, there is always noise – an undesirable discrepancy between the decisions of different experts on the same issue. And this noise is more than it seems.
2. The most dangerous noise is the template: it depends on the specific reaction of a particular person, his experience, and delusions.
3. Never trust first impressions.
4. We interpret the facts so that they coincide with our worldview, and discard the rest of the data. This is the main mental trap! When solving a problem, constantly think about what you might be missing.
5. Get a “devil’s advocate” in the company or for yourself.
6. If you want to know the qualitative judgment of several or many people on one issue, do not allow them to consult.
7. A list of assessments of various aspects of the transaction – their independent consideration – a deferred holistic judgment. This algorithm is suitable for interviews and for making any important decisions.
8. When dealing with a forecast, especially an improbable one, remember to regress to the average.
9. Replace judgments with rankings, scales with absolute values with scales with relative values, and a separate assessment of each employee with their comparative assessment.
10. Noise is inevitable, but you should not make excessive efforts to combat it: these measures can be more expensive than the harm from noise.
1. Read Daniel Kahneman’s book Think Slow… Decide Fast.
For more go to Home