Black Box Thinking Read online

  Many of the interviewees talked about their new lives. Almost all credited the three-hour visit to Rahway two decades earlier as having turned their lives around. Terence, the young black kid who had once stolen cars and broken into stores, was now a part-time preacher at his local Baptist church, with a wife and two sons. “Chances are, if I wouldn’t have gone to Rahway, I would probably be locked up and could be in my grave,” he said.

  Lori, the sixteen-year-old with the wide smile and big earrings, who had been dealing drugs, was now a thirty-six-year-old bookkeeper and mother. “I just thought it was a day away from school,” she said. “I don’t think I have ever been as afraid in my whole life . . . It made me not want to be an idiot anymore . . . I started going to school more after that.”

  Angelo, the kid with the unkempt hair and wispy mustache, was now thirty-seven years old, tiled floors for a living and had three kids. He said “If I didn’t go to Rahway, I think I would have done hard time,” he said. “If that one day didn’t happen, I might not have my family. And my family to me right now is everything; it is the most beautiful experience in the world.”

  This, then, is how the phenomenon of Scared Straight looked to millions of TV viewers. The statistics look good, too. This was a scheme, unlike most social programs, that actually bothered to collect data. According to the evidence, around 80 to 90 percent of people who attended the program went straight. As stated in the documentary: “That is an amazing success story. And it is unequalled by traditional rehabilitation methods.”

  But if we rewind to the late spring of 1977, a rather different picture was starting to emerge. In April of that year, James Finckenauer, a professor at the Rutgers School of Criminal Justice, decided to test Scared Straight. He wasn’t just interested in the observational statistics. As a scientist he knew that these could be misleading. He was not interested in hype or slickly presented documentaries either. He wanted to know if the scheme really worked. In short, he wanted to run an RCT.

  Finckenauer has silver-white hair and inquiring eyes. He has published dozens of papers and won multiple awards for his research, but his most striking quality is his conversational style. He is cautious, considered, and attentive. He also has a laserlike quality, as if he is trying to cut through the surface to find the truths lying beneath. These qualities would serve him well as he forensically unpicked the Scared Straight phenomenon.

  Before starting the RCT, Finckenauer probed the existing evidence for Scared Straight. Where did the 80 to 90 percent figure for kids going straight come from? He found that it was based on a questionnaire sent to the parents or guardians of children who had visited Rahway. (Another source of the data was letters of commendation sent in by the sponsoring agencies which brought kids to Rahway. These were not terribly reliable. These agencies may have had all sorts of hidden incentives to believe in the program.)

  There were four yes-or-no questions:

  Have you noticed a marked change in your child’s conduct since their visit to the prison?

  Has there been a slight change in their behavior since their visit to prison?

  Do you think another visit is necessary for your son/daughter?

  Are there any specific areas you think we might be of some assistance to you, or your son or daughter?

  There was also space to write comments.13

  But what did a “marked” change actually mean? What did a “slight” change mean? The questions were open to all kinds of interpretation. Finckenauer also discovered that many of the kids who visited Rahway had not been delinquent or even pre-delinquent in the first place. It hardly counts as a success that they didn’t commit crime afterward if they were already on the straight and narrow. Furthermore, the letters to parents were often sent within weeks of the prison visit. That was scarcely enough time to judge a change in behavior.

  And yet these were only minor quibbles. The deeper flaws go to the heart of what constitutes valid evidence. The first is that only those who responded to the questionnaire were included in the statistics. Those who didn’t respond were entirely absent from the data. Consider how that might have distorted the result. It is possible that only the parents of children whose behavior improved bothered to respond. Parents whose kids continued to behave badly might have thrown the questionnaire in the bin, or at least responded in fewer numbers. This could have skewed the stats beyond recognition.

  This is a type of so-called “selection bias” and it should sound familiar. It is pretty much the same problem that bedeviled medieval medicine when only those who recovered from bloodletting were able to testify to its effectiveness. The evidence sounded terrific but that is because it was dangerously incomplete. Those who did not recover from bloodletting were never given a chance to express an opinion. Why? Because they were already dead.

  The deepest problem with the Scared Straight statistics, however, related to the counterfactual. Even if everyone had responded to the questionnaire (which they hadn’t), we still wouldn’t know whether the outcomes had been caused by the intervention or by something else. Perhaps behavior would have improved without the intervention. Perhaps it improved because the local economy was improving, or because of a new scheme at school, or some other factor. Perhaps the outcome would have been even better without the intervention.

  In August 1978, Finckenauer divided a set of delinquent youths into two random groups.* One group attended the Scared Straight program. The other group (the control group) did not. He then sat and waited to measure the results. Despite the hype, the stellar-looking stats, the slick PR, the Oscar-winning documentary, the commendations from politicians, the tributes from corrections officers, and the widespread adoption of the scheme around the world, this was the first time the project had been subjected to the most rigorous kind of failure test.

  And the results, when they finally arrived, were dramatic. Scared Straight didn’t work. The children who attended Rahway were more likely to commit crimes than those who did not. “The evidence showed that the kids who went on the program were at greater risk of offending than those who didn’t,” Finckenauer said. “The data when you compared the treatment and control group was clear.”

  This was, to many people, a surprise. The program looked good. The logic seemed compelling. It had parents lining up to say that it had “cured” their kids. The questionnaire data seemed solid, too. But all of these things were true of bloodletting. Only with an RCT could we cut through the ambiguity and see the real effect of the program.

  Finckenauer says:

  People were convinced of the success of Scared Straight because it seemed so intuitive. People loved the idea that kids could be turned around through a tough session with a group of lifers. But crime turns out to be more complex than that. Children commit offenses for many different, often subtle reasons. With hindsight, a three-hour visit to prison was unlikely to solve the problem.

  The intentions of the inmates were genuine: they really wanted the kids to go straight. But the program was having unintended consequences. The experience of being shouted at seemed to be brutalizing the youngsters. Many seemed to be going out and committing crime just to prove to themselves and their peers that they weren’t really scared.14

  Defenders of the scheme reacted angrily to Finckenauer’s report. Judge Nicola, who had lavishly praised the program in the documentary, said: “. . . the [Scared Straight] program doesn’t need defending.” Robert J. McAlesher, the staff adviser to Scared Straight, was even more blistering. “We question the motives of dilettantes [i.e., Finckenauer] who compromise their intellectual integrity by thrusting themselves into the national limelight with meaningless statistics deceptively presented as the result of scientific study.”15

  These responses were, in a sense, predictable. When we are presented with evidence that challenges our deeply held beliefs, we tend to reject the evidence or shoot the messenger rather than amend our beliefs. Indeed
, many of the defenders of Scared Straight responded to the results of Finckenauer’s RCT by saying that they had become more convinced of the efficacy of the program, not less. This is precisely what the theory of cognitive dissonance would predict.

  But even those with no prior commitment to Scared Straight continued to be attracted to the program, like moths to a flame. The hard data showed that it was counterproductive, but the narrative of kids being deterred from crime by mean-talking inmates was too seductive to ignore. By the 1980s, Scared Straight–style programs were in operation in Georgia, South Carolina, and Wisconsin. Further programs were set up in New York, Virginia, Alaska, Ohio, and Michigan.16

  It was as if the research conducted by Finckenauer had never happened.

  By the 1990s similar programs were burgeoning. The Los Angeles Police Department ran a scheme where one of the components was kids visiting the city prison to be “shouted and screamed at” by convicts. At a program in Carson City, Nevada, a youngster was reported as saying that the part of the tour that made the greatest impact was “all the inmates calling us for sex and fighting for our belongings.” The idea was soon exported to the UK, Australia, and Norway.

  Meanwhile, the hard evidence against the scheme was multiplying. RCTs were conducted on Scared Straight–style programs from the West to the East Coast of America. They found the same thing: Scared Straight doesn’t work. It often damages kids. One of the trials showed a 25 percent increase in delinquency in the treatment group compared with the control group.

  But none of this seemed to matter. The glitzy narrative was far more seductive than the boring old data.17

  Even government officials eulogized the program. In 1994, a Scared Straight–style scheme in Ohio was commended in the official publication of the U.S. Office of Juvenile Justice and Delinquency Prevention. The experts had been bewitched by the narrative fallacy. In 1996, almost twenty years after Finckenauer’s RCT, the New York Times reported that the original program at Rahway was at the height of its popularity, hosting around ten groups per week or 12,500 kids per year.

  But then in 2002 the Campbell Collaboration arrived on the scene. This is a global, nonprofit organization devoted to evidence-based policy. They conducted what is called a “systematic review.” This is where the data from all the randomized trials are collated into a single spreadsheet. By pooling the results from all the individual trials (seven were used in the so-called meta-analysis), a systematic review represents the gold standard when it comes to scientific evidence. It is the ultimate failure test.18

  Forgive me if you know what’s coming, but the results were emphatic. Scared Straight doesn’t work. It increases crime. Some research indicates that this increase can be as high as 28 percent.19 In exquisitely understated language, the authors effectively damned its entire rationale: “We conclude that programs like Scared Straight are likely to have a harmful effect and increase delinquency . . . Doing nothing would have been better than exposing juveniles to the program.”20

  Scared Straight was, in many ways, ahead of its time. Unlike most social programs, which collate no data whatsoever, it actually sent out questionnaires and gathered statistics. But, as with medieval bloodletting, observational stats do not always provide reliable data. Often, you need to test the counterfactual. Otherwise you may be harming people without even realizing it.

  And this is really the point. It doesn’t require people to be actively deceitful or negligent for mistakes to be perpetuated. Sometimes it can happen in plain view of the evidence, because people either don’t know how to, or are subconsciously unwilling to, interrogate the data.

  But how often do we actually test our policies and strategies? How often do we probe our assumptions, in life or at work? In medicine, as we have seen, there have been almost one million randomized trials. In criminal justice, they scarcely exist. Policy, almost across the board, is run on narrative, hunch, untested ideology, and observational data skewed to fit predetermined conclusions.

  Closed loops are not just an intellectual curiosity, they accurately (and sometimes terrifyingly) describe the world in which we live.

  • • •

  On January 1, 1982, an intruder broke into the home of a nineteen-year-old called Michele Mika. After rummaging through several rooms, he took a knife from the kitchen, entered Ms. Mika’s bedroom, and murdered her. Michele’s mother later found her facedown in bed with an eight-inch carving knife in her back. After she was killed, Ms. Mika was sexually assaulted for several hours. The motive was pure sexual gratification.21

  More than twenty-five years later, on March 17, 2007, police arrested Angelo Speziale, a forty-five-year-old living in Hackensack, New Jersey. Speziale was one of the original seventeen youngsters profiled in Scared Straight! He was the kid with the unkempt hair and wispy mustache who had robbed shops in the neighborhood. He had also been interviewed in the follow-up feature twenty years later, by which time he had three kids and a job tiling floors.

  Like most of the people interviewed for the follow-up program, Speziale claimed that the visit to Rahway had transformed his life. It sounded almost inspirational. “If I didn’t go to Rahway, I think I would have done hard time,” he said. Danny Glover, the narrator, said: “Angelo, thirty-seven, is now a law-abiding family man.”

  But the reality was rather different. In 2005, Speziale was arrested for shoplifting and police obtained a DNA sample. During routine testing they discovered that it matched the DNA of the sperm found in the corpse of Michele Mika. Mika and Speziale, it turned out, had lived on opposite sides of the same duplex on Teaneck Avenue at the time the murder had taken place.

  The makers of the documentary did not deliberately mislead audiences about Speziale. They couldn’t have known that he was deceiving them when he said he had “gone straight.” They couldn’t have realized that just three years after he had visited Rahway, he had raped and murdered an innocent nineteen-year-old. Only the test provided by DNA revealed the truth.

  But the documentary makers did know by the early 1980s that Scared Straight was increasing crime. And yet they continued to make celebratory programs on the project. A&E, an American cable and satellite channel, introduced Beyond Scared Straight, a new series, in 2011. By 2014 it was in its eighth season. Arnold Shapiro, the producer (who also made the original 1978 documentary), continues to defend the scheme, despite the overwhelming evidence against it. He argues that Scared Straight today involves more counseling and less shouting. But the logic of conducting the interventions in prisons has always relied on a confrontational component. As the Daily Beast put it:

  The episodes themselves do emphasize the horrors of prison life more than discussion. At the beginning of one filmed at Maryland’s Jessup prison, a 50-year-old man convicted of first-degree murder barks into a 17-year-old dropout’s face, “Don’t smile at another man in prison, ’cause if you smile at another man in prison, that makes them think that you like them, and for you to like another man in prison, something seriously is wrong with you.”

  In his three-hour visit to Rahway in 1978, Speziale endured a number of degradations, but one event is particularly chilling in hindsight. The youngster was forced to stand in front of the group and read out a newspaper report of a knife attack that had taken place in prison. “Rahway inmate stabbed to death in cell block,” the sixteen-year-old read, voice trembling. “He was stabbed about a dozen times in the neck, chest, head and back. Robinson was pronounced dead on arrival at Rahway General Hospital.”

  There is no evidence of any connection between the fact that Speziale was humiliated into reading out loud the details of a savage knife attack on his visit to Rahway in 1978 and the fact that he perpetrated a similar crime a few years later. This is almost certainly a coincidence. But what we do know is that these visits, on average, damage the kids who are taken on them. We have known that for more than three decades.

  In 2010, Speziale pleaded gui
lty to sexual assault and stabbing and was sentenced to twenty-five years.22 He is now back in Rahway prison, where this story began. It is an endlessly disturbing and cautionary tale. But the deepest irony of all, and the one that takes us to the heart of the closed-loop phenomenon, is that Speziale might soon be delivering Scared Straight–style confrontations to the next generation of delinquents.*

  Part IV

  SMALL STEPS AND GIANT LEAPS

  Chapter 9

  Marginal Gains

  I

  At around 9 a.m. the riders of Team Sky, the British professional cycling team, made their way out of a small hotel in Carcassonne, a beautiful town in the Languedoc-Roussillon region of southern France. It was a warm morning and the riders walked to the team bus in silence, contemplating the day to come.

  They were about to start Stage 16 of the 2014 Tour de France, one of the sternest tests of endurance in the sporting world. They had already ridden 3,000 kilometers over the preceding fifteen stages and now faced a 237.5-kilometer ride culminating at the feared Port de Balès, a 19-kilometer climb into the Pyrenees. “Here we go again,” Bernhard Eisel, one of the team members, said with a grim smile.

  On the Team Sky bus there was a sense of anticipation. The riders were getting into their sports gear. The coaches were reviewing race plans. With thirty minutes to go, Nicolas Portal, one of Team Sky’s sporting directors, began his pre-race briefing. He talked about the importance of the stage and alerted the riders to difficult sections along the route. As he did so photographs of tough corners and steep climbs were flashed onto a screen at the front of the bus.

  As he finished his talk, a man toward the back, silent until that moment, started to speak. He had a shaved head, dark-rimmed glasses and an intense manner. He is the man who always has the final word before the race: the general manager of Team Sky, Sir David Brailsford.