Math is Always a Weapon

The authors of “Justices Flunk Math” worry that math—or more narrowly, statistics and probabilities—is being misused in the courtroom. After looking at a few examples, they conclude:

The challenge is to make sure that the math behind the legal reasoning is fundamentally sound. Good math can help reveal the truth. But in inexperienced hands, math can become a weapon that impedes justice and destroys innocent lives.

Inexperienced and experienced hands use math as a weapon. Let’s take, for example, the analogy they use in the article.

The appellate judge in the Amanda Knox case refused to retest the murder weapon for traces of the victim’s DNA. According to the article, the judge claimed that if there was too little material to provide a reliable result the first time it was tested, in 2007, then tests on less material in 2011 would be no better.

Although the authors claim that the judge “demostrated a clear mathematical fallacy,” they describe a simple flaw in reasoning and then use a misleading mathematical example to make their case. They are right. Repeated tests of the same material in comparable conditions can confirm or disconfirm previous results. And the greater number of confirming or disconfirming results should influence the conclusions we draw from those results. But their analogy, while superficially persuasive, doesn’t necessarily apply to the judge’s decision not to perform additional DNA tests.

They say:

Imagine, for example, that you toss a coin and it lands on heads 8 or 9 times out of 10. You might suspect that the coin is biased. Now, suppose you then toss it another 10 times and again get 8 or 9 heads. Wouldn’t that add a lot to your conviction that something’s wrong with the coin? It should.

According to the article, the judge’s decision was based on there being less material to test. How much less? We are not told. But less. So a better analogy might be (my changes are in bold):

Imagine, for example, you toss a coin and it lands on heads 8 or 9 times out of 10. You might suspect that the coin is biased. Now, suppose you then toss it another 3 times and get 2 heads. Would that add a lot to your conviction that something’s wrong with the coin? Maybe.

We can play with the numbers for that second coin-toss series, but a useful analogy here is not 10 tosses followed by another 10 but followed by some smaller number. How much the second coin-toss series will add to our conviction that the coin is biased will depend on how many times we toss it and how many times it comes up heads.

At first glance their math looks persuasive. The information they present in the article, however, does not allow us to assess the relevance of their analogy or evaluate the judge’s decision. We are being asked, on the strength of their mathematical analogy, to accept their criticism of the judge. In this case, their math distracts us from asking about why the judge made his decision and whether or not that was a reasonable decision. But this is nothing new. People hoping to achieve certain ends apply the math they think will be most persuasive. Math is always a weapon.

4 comments

  1. Michael Weiss says:

    Interesting. As you point out, the authors of “Justices Flunk Math” leave out the smaller amount of material. But then you leave out one of their points, namely the improvements in DNA analysis from 2007 to 2011. While their op-ed is not completely clear on this issue, their phrasing:

    By the time Ms. Knox’s appeal was decided in 2011, however, techniques had advanced sufficiently to make a retest of the knife possible, and the prosecution asked the judge to have one done.

    suggests that a smaller amount of material doesn’t result in a less reliable test, but rather determines whether the test can be done at all.

    Any experts on DNA analysis out there?

  2. Darin says:

    Michael,

    Thanks for the comments. You are right, the authors do say it was possible to test the knife again in 2011 even though there was less material to test. But possible and reliable are two separate issues.

    Testing was possible but, apparently, inconclusive in 2007. Just because it is possible again in 2011 doesn’t mean it will be conclusive this time. The article says nothing—and allows us to infer nothing—about the reliability of the test in 2011 or the likelihood that the results will be conclusive. Their analogy diverts our attention from asking about reliability by implying that one test is equivalent to the other—10 tosses are 10 tosses. The authors don’t give us the information needed to determine if the two tests are equivalent or if the results are as seemingly unambiguous “heads 8 or 9 times out of 10.”

    And that’s my point: in this case the math distracts us from asking the important questions about reliability in 2007, about the judge’s reasons for deciding that a second test would not be conclusive, about possible reliability in 2011, etc. In other words, why did the judge make his decision and do we find it reasonable?

    • Michael Weiss says:

      You’re right, it would be nice to know what reason the judge gave for his decision. Their description is not too clear:

      If the scientific community recognizes that a test on so small a sample cannot establish identity beyond a reasonable doubt, he explained, then neither could a second test on an even smaller sample.

      This suggests that the judge believed a second test could not change the probabilities at all. If that’s true, then that’s bad, and it sure seems relevant. If it’s not true, then the op-ed writers have unfairly maligned the judge.

      Admittedly, we don’t know enough about either the judge’s reasons nor the DNA details. Some of the other cases in the op-ed make the writers’ point more clearly.

      I have to disagree on your final point:

      in this case the math distracts us from asking the important questions about reliability in 2007, about the judge’s reasons for deciding that a second test would not be conclusive, about possible reliability in 2011, etc. In other words, why did the judge make his decision and do we find it reasonable?

      The math doesn’t distract us, it directs our attention to a critical issue: what is best way to model statistically the DNA test and proposed retest? Perhaps the authors have given us a misleading analogy — which matters a lot — but it matters because the issue is, in fact, quite relevant. Given the space constraints of an op-ed, I’m not ready to convict the authors of misdirection.

      The broader point, as I see it, is that probability and statistics often play a critical role in judicial decisions. Further, egregious examples of errors in the understanding of basic statistics occur in the judicial system. Finally, these really are errors, not mere matters of opinion.

      What bothers me most are your last two sentences:

      People hoping to achieve certain ends apply the math they think will be most persuasive. Math is always a weapon.

      This suggests a relativistic perspective. If I said:

      So what if educated people in the middle ages knew the earth was round. People hoping to achieve certain ends apply the history they think will be most persuasive. History is always a weapon,

      then I think you’d have a legitimate beef.

Comments are closed.