Noise: A Flaw in Human Judgment

Miles: An insurance company gives forty eight of its own underwriters the same realistic risk summaries and asks each of them to set a premium, as on any working day.

Miles: Before anyone sees the results, the executives get a question. Pick two of those underwriters at random. How far apart will their prices be? The usual guess is about ten percent.

Miles: The typical gap comes back at fifty five percent of the average premium. More than five times what the people running the place believed about their own firm.

Miles: Today: Noise, from Daniel Kahneman, Olivier Sibony and Cass Sunstein. In two sentences: everybody hunts bias, the error that leans one way, while the scatter, the same case drawing different answers depending on who looks at it and when, does as much damage and almost nobody counts it. Its remedy is structure, which makes it the best argument in print for an interview that doesn't reward whoever performs best in a room.

Miles: Sixty Books. Presented by Stoar Consulting and Coaching.

Miles: Book forty, and you've found Sixty Books, with me, Miles. Same terms as ever: a single management title per episode, sixty picked from the best of the past hundred years, each one read against two questions. What does it offer a leader or a team whose minds run on a different setting. And what still works once AI is part of the working week. Nora has the research, and this book hands her an equation.

Miles: Nora, we left Tetlock on a split. Two forecasters given one question land in different places, and so does a single forecaster asked on two different days.

Nora: And this whole book is built on the second half of that. Disagreement between people, everyone expects. A person disagreeing with an earlier version of himself is the part nobody plans for, and here it gets a name. Occasion noise.

Miles: And you'd already handed us the number that joins the two books.

Nora: The reanalysis of his tournaments, last time: on that model, roughly half of what made the trained, teamed and tracked forecasters better was less scatter, with bias and information sharing the rest. This book is the general theory behind that result, carried out of forecasting and into courts, insurers and hiring panels.

Miles: So the superforecasters weren't mainly smarter.

Nora: Steadier, on that model. They lurched about less. And this book has three authors, so by its own advice I should've read it three times, separately, and averaged myself.

Miles: Without conferring.

Nora: Yeah. Without conferring. That one's on the book's list of rules, and we'll reach it.

Miles: Start where the book starts.

Nora: In a courtroom. Marvin Frankel was a federal judge in New York, and in nineteen seventy three he published Criminal Sentences: Law Without Order, arguing for sentencing commissions and binding guidelines to rein in how much a sentence depended on the judge.

Miles: And somebody went and measured that.

Nora: Several times. The study the authors lean on hardest is a survey from nineteen seventy nine. Federal district judges were each handed the same sixteen hypothetical cases and asked to sentence them, and two hundred and eight gave complete answers. In the authors' summary, two judges picked at random differed by more than three and a half years on average, in cases where the mean sentence was about seven.

Miles: Half the sentence again, depending on whose courtroom.

Nora: On paper cases, and that matters, so I'll come back to it. Frankel's argument fed into federal reform: a commission, guidelines, the Sentencing Reform Act of nineteen eighty four. Then in two thousand five a Supreme Court ruling made those guidelines advisory rather than binding.

Miles: Then the insurance company.

Nora: Which is the authors' own work rather than a study they're reporting. Forty eight underwriters, the same realistic summaries, the job they did every day. And the part I'd underline isn't the fifty five percent. It's the executives' guess. Nobody inside knew, because nobody had ever handed two people the same file and compared.

Miles: And that comparison has a name.

Nora: A noise audit, and it's the one tool in the book you could run this month. Same cases, several people, working separately, and then you look at the spread before anyone explains themselves.

Miles: The authors.

Nora: Daniel Kahneman first. Born in Tel Aviv in nineteen thirty four, a psychologist who worked for close to thirty years with Amos Tversky on how people judge under uncertainty, and who received the Nobel in economics for that work in two thousand two. He was emeritus at Princeton, and he died in March of twenty twenty four, at ninety.

Miles: Second.

Nora: Olivier Sibony. Professor of strategy at HEC Paris, and before that twenty five years at a global consulting firm, finishing as a senior partner.

Miles: And the third.

Nora: Cass Sunstein, professor at Harvard Law School. From two thousand nine to twenty twelve he ran the White House office that reviews federal regulations, and in two thousand eight he wrote Nudge with Richard Thaler. So a psychologist, a strategist and a lawyer, which is roughly where this book goes looking for noise.

Miles: And the book.

Nora: May of twenty twenty one. Six parts and twenty eight chapters, plus appendices. It goes from finding noise, to measuring it, to where it comes from, to what reduces it, and it closes on when reducing it isn't worth the price.

Miles: In the book's order, please.

Nora: The opening move treats a judgment as a measurement. Your mind is the instrument, and an instrument can be off in two ways. Bias is a consistent lean, every reading too high. Noise is scatter, readings spread around the true value with no lean at all.

Miles: And most instruments manage both.

Nora: Most do. And then comes the equation that gives the book its argument. Overall error, measured as the average squared distance from the truth, splits into bias squared plus noise squared. So a unit of noise costs exactly what a unit of bias costs. Organizations pour effort into the first and, the authors argue, almost never measure the second.

Miles: And noise comes in kinds.

Nora: Three. Level noise is the harsh judge beside the lenient one: some people sit higher or lower across the board. Pattern noise is personal, where one person is tough on one kind of case and easy on another and a colleague has it the other way round. Occasion noise is one person, one case, a different moment.

Miles: Which is where last episode's forecaster lives.

Nora: It is, and here's the study in the book I like most. Vul and Pashler, Psychological Science, two thousand eight. Four hundred and twenty eight people answer eight general knowledge questions, then guess again without warning, either straight away or three weeks later. The average of your two guesses beats a single guess. Asked straight away, your second guess is worth about a tenth of asking someone else. Wait three weeks and it's worth about a third.

Miles: So the cheapest second opinion is you, a few weeks from now.

Nora: A slightly different you, which is the point. Then the book takes occasion noise into the courts. Eren and Mocan, American Economic Journal, twenty eighteen: every juvenile court decision in Louisiana from nineteen ninety six to twenty twelve. After an unexpected loss by the state's big college football team, judges handed down longer sentences the following week.

Miles: So a sentence can hang partly on a missed field goal.

Nora: Partly, in one study, in one state.

Miles: And a group? Five people in a room should cancel each other out.

Nora: The book says they often don't. Whoever speaks first, or with the most confidence, pulls everyone after, so a group can finish noisier than its members were alone. Which is why the fix is to collect each view separately, before anybody talks.

Miles: Then the models.

Nora: The oldest finding here and the least comfortable. Paul Meehl, nineteen fifty four, set expert clinical judgment against simple statistical rules across roughly twenty studies, and in all but one the rules did as well or better. Then Lewis Goldberg, in nineteen seventy, did something stranger. He built a simple model of each of twenty nine clinical psychologists out of their own past judgments, and for most of them, the model of the psychologist beat the psychologist.

Miles: How does a copy beat the original?

Nora: The copy never has a bad afternoon. It keeps the person's own policy and drops the scatter. That's the book in one experiment: a lot of what expert judgment loses, it loses to inconsistency, not to ignorance.

Miles: Huh.

Nora: Then the remedies, which the book calls decision hygiene, because it prevents errors you'll never get to see. Sequence the information, so the evidence gets judged before the story arrives. That chapter rests on work by Itiel Dror and colleagues, who gave five experienced fingerprint examiners prints they'd already matched in earlier casework, now framed with context suggesting no match. Four of the five reversed their own earlier call.

Miles: Same print, same expert, new story.

Nora: Then aggregate: several independent judgments, combined. Use relative scales, comparing against fixed examples rather than marks out of five that mean something different to every rater. And for big decisions, the Mediating Assessments Protocol. Break the decision into a handful of separate assessments, score each on its own, by different people where you can, and form the overall view last, once every piece is in.

Miles: And hiring gets its own chapter.

Nora: Structure in Hiring. The same questions for every candidate, scored against answers agreed in advance, and interviewers who score independently before anyone compares notes. That chapter is our segment, so I'll leave it there.

Miles: And the final part of the book argues with all of this?

Nora: It argues with itself, which I like. Cutting noise costs money, rules get gamed, and people want to feel that a person heard them, which the book discusses under dignity. Sometimes a looser standard is worth the scatter. It's the part I'd hand a manager first.

Miles: Now the evidence, and first, a name on the cover.

Nora: One line, since episode twenty six covered that book. The social priming studies in Kahneman's earlier bestseller mostly failed to replicate, and he said so himself. In an open letter to priming researchers in twenty twelve he warned of a train wreck looming, and in twenty seventeen he wrote that he had placed too much faith in underpowered studies. That's his own account of that book. This one gets judged on its own.

Miles: Then what's this one standing on?

Nora: Three layers, with three different strengths. The headline examples, the old research on models against experts, and the evidence behind the remedies.

Miles: Headlines first.

Nora: The insurance audit comes from the authors' own consulting work. It's reported in the book and in their articles, not in a journal where anybody outside could examine the data. The sentencing figures come from judges sentencing paper cases in a survey, not real defendants in real courtrooms, and that survey's own authors limited their conclusion to the hypothetical cases. And the occasion noise examples are, for the most part, single studies of single systems.

Miles: And the critics?

Nora: The most careful one I found was Michael Blastland, reviewing it in Prospect in June of twenty twenty one. He recommended the book and still pressed on two things the authors concede in their own small print. The error equation needs a true value, so it doesn't apply to evaluative judgments, where there isn't one. And the costs of error are rarely symmetrical, so a measure that counts every miss the same way can mislead you about which misses matter.

Miles: Is that damaging?

Nora: It's a limit, and the book states it. His sharper point is that in plenty of real judgments nobody knows the true value, and a spread of opinion may be telling you exactly that. Push everyone toward one figure and you might agree, very consistently, on the wrong one. That's fair, and the book's final part concedes that some noise is worth keeping.

Miles: And the old research?

Nora: That's where it's strongest. Grove and colleagues, two thousand, Psychological Assessment, a meta-analysis of studies predicting human health and behavior. Mechanical prediction was about ten percent more accurate on average, clearly better in a third to nearly half of the studies, and clearly worse in only six to sixteen percent. Aegisdottir and colleagues did it again in two thousand six with sixty seven studies, and in their strictest set the statistical methods came out about thirteen percent more accurate.

Miles: And the remedies?

Nora: Mostly sturdy. The crowd within study was repeated in twenty fourteen by a team in Leuven, registered in advance and run with a large sample, and the effect held at both delays. The rating noise holds too. Scullen, Mount and Goff, Journal of Applied Psychology, two thousand: about four and a half thousand managers across two data sets, each rated by seven people. The raters' own quirks accounted for more than half of the variance in the ratings. The managers' performance accounted for about a fifth to a quarter.

Miles: So a performance review mostly measures the reviewer.

Nora: In that data, more than it measures the person being reviewed. And the interview finding has moved, though not the way a skeptic would bet. Sackett and colleagues reworked the standard validity estimates in twenty twenty two and concluded that many selection methods predict job performance less well than people had believed. Structured interviews came out at the top of the revised list, with a pooled correlation of about point four two with job performance. Another group has disputed some of their corrections and they've replied in print, so treat the exact figure as open and the ranking as firmer.

Miles: And the tier statement.

Nora: Here it is. Noise is practitioner literature, written for general readers, not peer reviewed research, and its best known examples are the authors' own audits and single studies. Underneath sits seventy years of research on models, ratings and interviews that has held up better than most of what this shelf stands on. So trust the old science beneath it more than the vivid stories on top of it, and neither half of that is a verdict on anyone.

Miles: Our half.

Nora: And here's the claim I'd put on the jacket. Structure is the best documented way to make a hiring decision, and it's also the strongest accommodation on offer for anyone who reads badly in a room. Nobody designed it for that. It falls out of the method.

Miles: Start with the room.

Nora: Sasson and colleagues, Scientific Reports, twenty seventeen. Three studies. In the first, two hundred and fourteen students judged the opening ten seconds of twenty autistic adults and twenty non autistic adults in conversation. In every format but one, the autistic adults were rated less favorably, and the observers were less willing to talk with them. The exception was the transcript. Same words, no face, no voice.

Miles: So the delivery is being judged, not the content.

Nora: On those clips, in those samples, yes. Whelpley and May took it into hiring, Journal of Autism and Developmental Disorders, twenty twenty two. Mock job interviews with autistic and non autistic candidates, rated by people who weren't told who was who, from either the video or the transcript. On video the non autistic candidates came out ahead. On transcript, the autistic candidates did better.

Miles: That's a full reversal.

Nora: In one study, and I'd want it repeated before anyone builds policy on it. But set it beside this book. Grove's meta-analysis noticed something in two thousand that sits right next to it: clinical judgment did relatively worse when the information it worked from included an interview. My reading is that the face to face impression is where human judgment adds the most noise, and it's the same channel where these candidates lose marks.

Miles: And does structure fix that?

Nora: The closest test I found is Maras and colleagues, in the journal Autism, twenty twenty one. Twenty five autistic and twenty five non autistic adults did two mock interviews, about six months apart, scored by employment specialists who weren't told anyone's diagnosis. The first used ordinary interview questions, and the autistic candidates were rated lower. The second used questions rewritten from the first round's feedback to be clearer and more specific. Everyone's ratings went up, the autistic candidates' most, and the gap narrowed.

Miles: So the wording of the question was part of the problem.

Nora: Part of it, and that's the caution tucked inside the good news. Structure isn't fair automatically. A vague question asked identically of every candidate is consistent, and it's still vague. The book asks you to decide in advance what you're measuring and how you'll score it, and Maras is evidence that the words of the question carry weight as well.

Miles: And reviews.

Nora: Same logic, and Scullen's numbers are the reason. If more than half of a rating belongs to the rater, then a person who comes across badly in meetings is partly being scored on how those meetings felt to the manager. The book's fixes are anchored scales, several independent raters, and scores tied to things that happened rather than to an overall impression. Every one of those moves the score further from the room.

Miles: Cautions.

Nora: Two. The first is that the join is mine. Nobody has tested a full structured hiring or review process with neurodivergent employees as the outcome. Maras is fifty people in mock interviews, and Sasson and Whelpley are about first impressions, not careers. I've set those beside a book about noise and drawn a line between them, and that's an argument, not a finding.

Miles: And the second.

Nora: Comes from the book's own chapter on dignity. Some people want a conversation, not a scoring sheet, and structure done coldly can feel like being processed. So keep the conversation warm and score it against the rubric afterwards, on your own. The structure belongs in the scoring, not in being curt with the person in front of you.

Miles: Fair enough.

Miles: One minute, on this show's terms. Noise says that when the same case gets different answers depending on who decides and when, that scatter costs as much as bias, and organizations almost never measure it. For anybody leading a mind that runs on a different setting, or working with one, here's Monday. Run a small noise audit: give the same three candidate files to several people who score them separately, and look at the spread before anyone talks. Write the interview questions, and what a strong answer contains, before you meet a single candidate, ask everyone the same ones, and word each one so it says plainly what you want to hear about. Score each part on its own and form the overall view last. And for reviews, use more than one rater and tie every score to something that happened, because otherwise the review mostly measures the reviewer, and the person who reads badly in a room pays for it.

Miles: Now the tools.

Nora: The general caution about these tools waits at the end of the episode, word for word as always, so I'll point there and spend this time on what this book adds.

Miles: Go on.

Nora: The book's case for formulas was never that they're clever. It's that they give the same answer twice, which is how Goldberg's dull little models beat the psychologists. These tools work differently: they generate, and generating has scatter built in. One research group ran five language models on eight benchmark tasks, ten runs each, on settings meant to make the output repeat exactly. None of them delivered repeatable accuracy across every task, and accuracy shifted by as much as fifteen percent between runs. That's a preprint, not yet through peer review, and the models have changed since.

Miles: So the machine has occasion noise.

Nora: Yeah. The machine has occasion noise, and you can run the book's audit on it. Give it the same candidate summary three times, in three separate sessions, and compare the scores. If they wander, it isn't ready to be a judge, however sure it sounds.

Miles: And the jobs it can do?

Nora: Three, all on the structure side rather than the verdict side. Drafting: give it the role and have it propose questions, plus what a strong, middling and weak answer would contain, then edit hard, because choosing the anchors is your call. Sequencing: have it strip a file of the details that shouldn't sway a first read, the way the forensic chapter keeps the story away from the examiner. And a separate first read for a review, written before you've formed your own view and set beside it afterwards. That last one is the crowd within, with a second instrument.

Miles: Leave us with one move.

Nora: The book's own move. Before you hand any judgment to a machine or a panel, find out how much it disagrees with itself. Those insurance executives guessed ten percent, and the real gap was more than five times that. Ask the question twice before you trust the answer once.

Miles: That's Noise.

Miles: Next time: How Big Things Get Done, by Bent Flyvbjerg and Dan Gardner, the same Dan Gardner who co-wrote last episode's book. Noise wrecks a judgment, and when that judgment is a cost estimate it wrecks the budget too, so next we follow it into the projects that run over.

Nora: Which is where the scatter finally gets a price tag.

Miles: That was a careful one, Nora. Thanks for it.

Nora: Happy to, Miles.

Miles: See you at the next book.

Miles: This was Sixty Books, made by Stoar Consulting and Coaching. Logan Williamson is a Licensed Professional Counselor in Texas and Louisiana. What you heard was a single book examined for people who lead teams and people who work in them. None of it was coaching or counseling, and none of it was a judgment about you.

Nora: Sources for every claim sit on the episode page, alongside the transcript. Book forty one is two days away.

Miles: A disclosure in two parts. Neither of these voices is a person: Miles and Nora are synthetic, licensed in nothing, and nothing in this episode was said about whoever's listening. AI helps write each script, and then Logan Williamson, a Licensed Professional Counselor in Texas and Louisiana, at Stoar Consulting and Coaching, reviews it, revises it and approves it.

Nora: And one note about the AI segment. The evidence that these tools help neurodivergent people at work is early. There are controlled trials now, but they measure things like conversation skill rather than how a job actually goes, and for work itself most of what exists is individual providers reporting success. So take it as promising and not proven. The tools also get rebuilt every few months, so whatever you set up today is built on something that moves.